How do I multithread BufferedImage rendering in ChartPanel?

[ptd26]

I am trying to use JFreeChart to display several charts on one window. I would like to use threading to speed up the performance of my application. My first attempt was to pull out the buffered image rendering from paint component and put it in a separate thread. But it's not working out well. It's glitchy on rendering (for a second or two, all the charts show up as one of the charts, then flash back to the updated chart...wierd). And this all takes just as long if not render as the default unthreaded application (using ChartPanel without modification). However, with the extra threads created, I expected to see some increase in CPU utilization, but not so. Can someone please share their multithreading approach or tell me where I am going wrong with this code (paintComponent is mostly unchanged. Only part I changed is directly under the comment: "// do we need to redraw the buffer?" refreshBuffer() uses the first part of paintComponent):

Code: Select all

package com.rapidminer.operator.display;

import java.awt.Dimension;
import java.awt.Graphics;
import java.awt.Graphics2D;
import java.awt.GraphicsConfiguration;
import java.awt.Insets;
import java.awt.Rectangle;
import java.awt.Transparency;
import java.awt.geom.AffineTransform;
import java.awt.geom.Rectangle2D;
import java.util.Iterator;

import org.jfree.chart.ChartPanel;
import org.jfree.chart.JFreeChart;
import org.jfree.chart.panel.Overlay;

public class MultiChartPanel extends ChartPanel {

	public MultiChartPanel(JFreeChart chart) {
		super(chart);
	}

	/**
	 * Paints the component by drawing the chart to fill the entire component,
	 * but allowing for the insets (which will be non-zero if a border has been
	 * set for this component).  To increase performance (at the expense of
	 * memory), an off-screen buffer image can be used.
	 *
	 * @param g  the graphics device for drawing on.
	 */
	public void paintComponent(Graphics g) {
		if (this.chart == null) {
			return;
		}
		Graphics2D g2 = (Graphics2D) g.create();

		// first determine the size of the chart rendering area...
		Dimension size = getSize();
		Insets insets = getInsets();
		Rectangle2D available = new Rectangle2D.Double(insets.left, insets.top,
				size.getWidth() - insets.left - insets.right,
				size.getHeight() - insets.top - insets.bottom);

		// work out if scaling is required...
		boolean scale = false;
		double drawWidth = available.getWidth();
		double drawHeight = available.getHeight();
		this.scaleX = 1.0;
		this.scaleY = 1.0;

		if (drawWidth < this.minimumDrawWidth) {
			this.scaleX = drawWidth / this.minimumDrawWidth;
			drawWidth = this.minimumDrawWidth;
			scale = true;
		}
		else if (drawWidth > this.maximumDrawWidth) {
			this.scaleX = drawWidth / this.maximumDrawWidth;
			drawWidth = this.maximumDrawWidth;
			scale = true;
		}

		if (drawHeight < this.minimumDrawHeight) {
			this.scaleY = drawHeight / this.minimumDrawHeight;
			drawHeight = this.minimumDrawHeight;
			scale = true;
		}
		else if (drawHeight > this.maximumDrawHeight) {
			this.scaleY = drawHeight / this.maximumDrawHeight;
			drawHeight = this.maximumDrawHeight;
			scale = true;
		}

		Rectangle2D chartArea = new Rectangle2D.Double(0.0, 0.0, drawWidth,
				drawHeight);

		// are we using the chart buffer?
		if (this.useBuffer) {

			// do we need to resize the buffer?
			if ((this.chartBuffer == null)
					|| (this.chartBufferWidth != available.getWidth())
					|| (this.chartBufferHeight != available.getHeight())) {
				this.chartBufferWidth = (int) available.getWidth();
				this.chartBufferHeight = (int) available.getHeight();
				GraphicsConfiguration gc = g2.getDeviceConfiguration();
				this.chartBuffer = gc.createCompatibleImage(
						this.chartBufferWidth, this.chartBufferHeight,
						Transparency.TRANSLUCENT);
				this.refreshBuffer = true;
			}

			// do we need to redraw the buffer?
			if (this.refreshBuffer) {
				new Thread(new Runnable() {
					public void run() {
						refreshBuffer();
					}
				}).start(); 
				g2.dispose();
				return;
			}

			// zap the buffer onto the panel...
			refreshParentJPanel(g2);
			g2.drawImage(this.chartBuffer, insets.left, insets.top, this);

		}

		// or redrawing the chart every time...
		else {

			AffineTransform saved = g2.getTransform();
			g2.translate(insets.left, insets.top);
			if (scale) {
				AffineTransform st = AffineTransform.getScaleInstance(
						this.scaleX, this.scaleY);
				g2.transform(st);
			}
			refreshParentJPanel(g2);
			this.chart.draw(g2, chartArea, this.anchor, this.info);
			g2.setTransform(saved);

		}

		Iterator iterator = this.overlays.iterator();
		while (iterator.hasNext()) {
			Overlay overlay = (Overlay) iterator.next();
			overlay.paintOverlay(g2, this);
		}

		// redraw the zoom rectangle (if present) - if useBuffer is false,
		// we use XOR so we can XOR the rectangle away again without redrawing
		// the chart
		drawZoomRectangle(g2, !this.useBuffer);

		g2.dispose();

		this.anchor = null;
		this.verticalTraceLine = null;
		this.horizontalTraceLine = null;

	}

	public void refreshBuffer()
	{
		long startime = System.currentTimeMillis();
		// first determine the size of the chart rendering area...
		Dimension size = getSize();
		Insets insets = getInsets();
		Rectangle2D available = new Rectangle2D.Double(insets.left, insets.top,
				size.getWidth() - insets.left - insets.right,
				size.getHeight() - insets.top - insets.bottom);

		// work out if scaling is required...
		boolean scale = false;
		double drawWidth = available.getWidth();
		double drawHeight = available.getHeight();
		this.scaleX = 1.0;
		this.scaleY = 1.0;

		if (drawWidth < this.minimumDrawWidth) {
			this.scaleX = drawWidth / this.minimumDrawWidth;
			drawWidth = this.minimumDrawWidth;
			scale = true;
		}
		else if (drawWidth > this.maximumDrawWidth) {
			this.scaleX = drawWidth / this.maximumDrawWidth;
			drawWidth = this.maximumDrawWidth;
			scale = true;
		}

		if (drawHeight < this.minimumDrawHeight) {
			this.scaleY = drawHeight / this.minimumDrawHeight;
			drawHeight = this.minimumDrawHeight;
			scale = true;
		}
		else if (drawHeight > this.maximumDrawHeight) {
			this.scaleY = drawHeight / this.maximumDrawHeight;
			drawHeight = this.maximumDrawHeight;
			scale = true;
		}

		Rectangle2D chartArea = new Rectangle2D.Double(0.0, 0.0, drawWidth,
				drawHeight);

		this.refreshBuffer = false; // clear the flag

		Rectangle2D bufferArea = new Rectangle2D.Double(
				0, 0, this.chartBufferWidth, this.chartBufferHeight);

		Graphics2D bufferG2 = (Graphics2D)
		this.chartBuffer.getGraphics();
		Rectangle r = new Rectangle(0, 0, this.chartBufferWidth,
				this.chartBufferHeight);
		bufferG2.setPaint(getBackground());
		bufferG2.fill(r);
		if (scale) {
			AffineTransform saved = bufferG2.getTransform();
			AffineTransform st = AffineTransform.getScaleInstance(
					this.scaleX, this.scaleY);
			bufferG2.transform(st);
			this.chart.draw(bufferG2, chartArea, this.anchor,
					this.info);
			bufferG2.setTransform(saved);
		}
		else {
			this.chart.draw(bufferG2, bufferArea, this.anchor,
					this.info);
		}
		System.out.println(System.currentTimeMillis() - startime);
		repaint();
	}

}

[remiohead]

Your time is better spent profiling the single threaded approach and finding out why it's slow. My app creates and displays dozens of charts on screen without any performance issues. See also important notes on using threads with Swing components: http://java.sun.com/products/jfc/tsc/ar ... eads3.html.

[skunk]

I think the reason that you are not seeing any performance improvement is that you are not introducing any parrallelism by spinning off another thread.

The way updating the screen works in Swing is essentially:

1) As soon as the component decides that it should be repainted on the screen, JComponent.repaint() is called. This results in an asynchronous repaint request being sent to the RepaintManager, which uses invokeLater() to queue a Runnable on the EDT.

2) When the Runnable executes, it invokes the RepaintManager, which invokes paintImmediately() on the Component. The component then sets the clip rectangle and calls paint() which ends up calling paintComponent() which you have overridden. Remember that the screen is locked and will remain locked until the component has entirely repainted the dirty rectangle.

There is no point in spinning off a thread to generate the image buffer, because the RepaintManager HAS TO block until the buffer is ready so it can finish updating the dirty rectangle before releasing the lock on the screen.

All the toolkits that swing supports (windows, linux, mac) are single threaded by design. It is not possible to concurrently update more than one region of the screen.

If you are willing to modify the library source, there is low hanging fruit in XYPlot.draw() / XYPlot.render()
Pseudo code would look something like this

Code: Select all

Create/clear a buffer
Draw titles legends etc
Draw the axes
For each dataseries
    For each visible datapoint
        draw the datapoint (may require multiple passes)
        generate the tooltip
   End
End

Depending on the characteristics of the data you are rendering, there are modifications that can be made to the rendering code that can vastly improve the rendering performance / update frequency. If the majority of the data is unchanged between each repaint of the chart (like financial time series), you can improve performance by reducing the number of datapoints that are repainted each frame. If you are really ambitious, you can modify ChartPanel and try to save the image buffer across calls and only redraw data points that have changed since the last repaint.

In my experience, the most important thing is a design that never allows repaint requests to be queued. If 10 data updates arrive before the system has a chance to repaint the chart, you dont need to display the 9 stale states as long as the chart displays all the accumulated changes when it finally is repainted.

If you really wanted to generate the chart on another thread you would have to start the thread in response to a data event and then store the generated image somewhere. Then, when paintComponent() is called you could simply draw the saved image. I imagine that the complexity required to synchronize access to the shared image buffer would incur so much overhead that it would not actually improve the frame rate and you would end up regenerating the buffer far more times than required.

Good luck

[ptd26]

Skunk, thanks for your response. I have already modified the library source code slightly. I do everything in one pass. Indeed, a better idea would be to save the image buffer and then only add on most recent updated points. This would save having to redraw the buffer everytime. However, normally JFreeChart scales the data so that it fills up the entire horizontal axis. In order to be able to add only recent updates, some space would need to be saved for appending the more recent updates. Is there a way to initially instruct JFreeChart to plot data using only 90% (or some adjustable percentage) of the axis? Then I could simply append points on the end until the full axis is covered, then rescale / repaint to fill up 90% again. I think that would allow the quick real-time plotting that I am looking for.

For the general case (moving cursor around, rescaling chart, etc.), I think I still have one additional multi-threading solution that I would like to propose for feedback. Override the repaint() call in ChartPanel. Inside the repaint call, check to see if the bufferedImage needs to be repainted (paintComponent() in JFreeChart already does this by default). If the BufferedImage needs to be re-rendered, fire off a SwingWorker thread to do the re-rendering. When it finishes re-rendering, call super.repaint() which will cause the Swing Event Dispatch thread to queue up a call to paintComponent. Modify paintComponent to only display the buffered image. Would this speed things up? I would have to think this approach would be faster at least on a quad-core system then doing all the work on the swing thread.

UPDATE:
Here's my latest attempt at multi-threading ChartPanel. Leave rest of ChartPanel as it is, but add this code:

Code: Select all

	public void repaint()
	{
		new Thread(new Runnable() {
			public void run()
			{
				refreshBuffer();
			}			
		}).start();
	}

	public void refreshBuffer()
	{
		if (this.chart == null) {
			return;
		}

		if(!this.refreshBuffer || chartBuffer == null)
		{
			super.repaint();
			return;
		}

		// first determine the size of the chart rendering area...
		Dimension size = getSize();
		Insets insets = getInsets();
		Rectangle2D available = new Rectangle2D.Double(insets.left, insets.top,
				size.getWidth() - insets.left - insets.right,
				size.getHeight() - insets.top - insets.bottom);

		// work out if scaling is required...
		boolean scale = false;
		double drawWidth = available.getWidth();
		double drawHeight = available.getHeight();
		this.scaleX = 1.0;
		this.scaleY = 1.0;

		if (drawWidth < this.minimumDrawWidth) {
			this.scaleX = drawWidth / this.minimumDrawWidth;
			drawWidth = this.minimumDrawWidth;
			scale = true;
		}
		else if (drawWidth > this.maximumDrawWidth) {
			this.scaleX = drawWidth / this.maximumDrawWidth;
			drawWidth = this.maximumDrawWidth;
			scale = true;
		}

		if (drawHeight < this.minimumDrawHeight) {
			this.scaleY = drawHeight / this.minimumDrawHeight;
			drawHeight = this.minimumDrawHeight;
			scale = true;
		}
		else if (drawHeight > this.maximumDrawHeight) {
			this.scaleY = drawHeight / this.maximumDrawHeight;
			drawHeight = this.maximumDrawHeight;
			scale = true;
		}

		Rectangle2D chartArea = new Rectangle2D.Double(0.0, 0.0, drawWidth,
				drawHeight);


		this.refreshBuffer = false; // clear the flag

		Rectangle2D bufferArea = new Rectangle2D.Double(
				0, 0, this.chartBufferWidth, this.chartBufferHeight);

		Graphics2D bufferG2 = (Graphics2D)
		this.chartBuffer.getGraphics();
		Rectangle r = new Rectangle(0, 0, this.chartBufferWidth,
				this.chartBufferHeight);
		bufferG2.setPaint(getBackground());
		bufferG2.fill(r);
		if (scale) {
			AffineTransform saved = bufferG2.getTransform();
			AffineTransform st = AffineTransform.getScaleInstance(
					this.scaleX, this.scaleY);
			bufferG2.transform(st);
			this.chart.draw(bufferG2, chartArea, this.anchor,
					this.info);
			bufferG2.setTransform(saved);
		}
		else {
			this.chart.draw(bufferG2, bufferArea, this.anchor,
					this.info);
		}
		super.repaint();
	}

Note that this code is only useful for moving the cross-hairs around quickly. It defaults to the slow single-threaded case for resizing the chart. But it's still not working as I had hoped. My idea was as discussed above, to intercept the repaint() call and render the buffered image in a separate thread, then call super.repaint() to draw the new buffered image to screen. I know that it's creating new threads because if I add in some very process-intensive code at the end of the repaint() function above, I see CPU usage go to 100%. However, for updating the charts, it doesn't seem to actually be threading. Or at least the threads are so short, they have not noticeable impact on CPU usage when monitored through Linux Top program. But still, drawing the buffered image takes 1-2 ms yet it takes 2-4 seconds for all the charts to update. Why aren't the updates instantaneous?

[skunk]

All super.repaint() does is update the dirty rectangle(s)
paintComponent() is what actually updates the screen
The code as posted looks like it performs a lot of unneccesary work in the thread created by your repaint() override since nothing on the screen will change until paintComponent() is called by the RepaintManager.

If drawing the image only takes 1-2ms then the performance bottleneck must be somewhere else.

[ptd26]

I thought calling super.repaint() would cause a repaint request to be enqueued in Swing Event Dispatch. I still think the code above should speed things up, but don't know why it doesn't. I understand that paintComponent() is the function that actually does the painting, but the reason it only takes 1-2 ms is because it only has to paint the BufferedImage that was generated by the off-swing thread. You're right that my repaint() override will not change anything on the screen. The purpose of it is to regenerate the BufferedImage which will later be displayed by the second call to paintComponent (since the spawned thread calls super.repaint() after it's done repainting). Basically, here's what I am trying to do:

1. Something calls repaint()
2. My overloaded repaint() function starts a new thread which calls refreshBuffer().
3. refreshBuffer() checks to see if the buffered image needs to be regenerated. If so, it regenerates the buffered image. It then calls super.repaint() which will eventually cause paintComponent() to get called.
4. paintComponent() gets called and unless the chart was resized, the bufferedImage won't need to be refreshed, only painted.

So step 3 should be taking the most time, approx. 300-500 ms on my system. Step 4 only takes 1-2 ms because usually all it has to do is repaint the bufferedImage. So I would expect that it would take less than 1 second to paint all ~10 of my charts. Step 3 should now be happening in parallel. Step 4 happens in serial (on event dispatch thread), but it now runs much faster since all it has to do is repaint bufferedImage. But what I see happening is that it still takes about the same amount of time as the unmodified / single-threaded JFreeChart. I don't understand why.

[skunk]

Looking through the source code, it seems like useBuffer is incorrect? Depending on the version of source you are working on, the default value for DEFAULT_BUFFER_USED may be false.

Also, you will have to add some synchronization to prevent 1) multiple worker threads writing to the buffer at the same time and 2)one of your worker threads writing to the buffer at the same time as the code in paintComponent() is reading the buffer.

[ptd26]

It is using bufferedImage. I was able to tell by placing print statements / debugging. I'm using the latest release.

I think you're right that the best bang for the buck will be in optimizing the single-thread case so that's where I'll focus my attention for now. The more I think about it, I really only need two things:

1. Fast crosshair updates. I'll try to come up with a better method for placing the crosshairs than relying on a O(n) linear pass over the array.
2. Fast last-item add charting. Where 99% of the chart stays the same, I just want to tack on another bar or two at the end.

If I can get that, I probably won't need multi-threading. Just out of curiosity, on a modern computer running Java. How many fillRectangles could it plot per second? Just a rough ballpark number:

a) 100 - 1000
b) 1000 - 10000
c) 10000 - 100000
d) more

Trying to determine how much of the bottleneck is Java's graphics system. I know there are 3D FSPs implemented in Java so I tend to think the answer to the above question is either c or d. Would the answer actually depend on the video card? In Windows, I would assume so due to Direct3D rendering, but in Unix / Linux, I'm not sure.

[KurtP]

In my experience, the most important thing is a design that never allows repaint requests to be queued. If 10 data updates arrive before the system has a chance to repaint the chart, you dont need to display the 9 stale states as long as the chart displays all the accumulated changes when it finally is repainted.

Do you happen to have a suggestion of how to prevent queueing of repaint requests? Is there an (easy) way to determine if repainting is in progress respectively when it is done?

I thought of recording all event ID's between two repaints, and remove them if they are still in the queue before repainting with chart.setNotify(true). But that sounds rather adventurous. User paradoxoff recommended this: http://www.jfree.org/phpBB2/viewtopic.php?f=3&t=37991, which sounds not less adventurous.

Thanks,
Kurt