JFreeChart with Huge Datasets - Optimization

[SamiLakka]

Hi

We have to deal with huge datasets in our application (usually over 2 million points). We have developed a custom XYPlot that optimizes the rendering by testing that points that translate to same pixels are drawn only once.

I was thinking that could XYPlot be further optimized by first "drawing" the pixels to an two-dimensional double array where the dimensions contain the xy position and the value is the pixel color. After that this double array would be drawn once instead of pixel at a time as we do now. The drawing could be done in several passes by using some shape, say a box 10 x 10 pixels wide.

We would go through the array using this shape and when all the pixels that could be drawn with this shape are drawn, the size of the shape is decresed. In the end only individual pixels would plotted.

The idea behind this optimization is that when you have a large dataset, there usually are large area of pixels with same color. I think that drawing an x-by-x box is also faster than drawing the individual pixels at a time (though I'm not sure).

Does anyone have an idea if this would work.

[cube]

Hi,

Will you share your XYPlot implementation ?
I am also dealing with huge datasets and it could help me a lot.

[SamiLakka]

Sure thing, here it is.

Code: Select all

/**
 * Fast XY Plot implementation from
 * http://www.jfree.org/phpBB2/viewtopic.php?t=18592&postdays=0&postorder=asc&highlight=efficiency&start=15
 * 
 * Added some optimization.
 * 
 * @author WA Technologies Oy (Sami Lakka)
 * 
 */
public class FastXYPlot extends XYPlot {	
	
	
	private static final long serialVersionUID = 1424417952643263459L;

	public FastXYPlot() {
		this(null, null, null, null);
	}

	public FastXYPlot(XYDataset dataset, ValueAxis domainAxis,
			ValueAxis rangeAxis, XYItemRenderer renderer) {
		super(dataset, domainAxis, rangeAxis, renderer);
	}

	
	public boolean render(Graphics2D g2, Rectangle2D dataArea, int index,
			PlotRenderingInfo info, CrosshairState crosshairState) {

		boolean foundData = false;
		XYDataset dataset = getDataset(index);
		if (!DatasetUtilities.isEmptyOrNull(dataset)) {
			foundData = true;
			ValueAxis xAxis = getDomainAxisForDataset(index);
			ValueAxis yAxis = getRangeAxisForDataset(index);
			XYItemRenderer renderer = getRenderer(index);
			if (renderer == null) {
				renderer = getRenderer();
			}

			XYItemRendererState state = renderer.initialise(g2, dataArea, this,
					dataset, info);
			int passCount = renderer.getPassCount();

			SeriesRenderingOrder seriesOrder = getSeriesRenderingOrder();

			// Render series in reverse order
			if (seriesOrder == SeriesRenderingOrder.REVERSE) {
				
				for (int pass = 0; pass < passCount; pass++) {
					int seriesCount = dataset.getSeriesCount();
					for (int series = seriesCount - 1; series >= 0; series--) {
						renderFast(xAxis, yAxis, dataset, series, renderer,
								state, pass, g2, dataArea, info, crosshairState);
					}
				}											
			} 
			// Render series in forward order
			else {				
				for (int pass = 0; pass < passCount; pass++) {
					int seriesCount = dataset.getSeriesCount();
					for (int series = 0; series < seriesCount; series++) {
						renderFast(xAxis, yAxis, dataset, series, renderer,
								state, pass, g2, dataArea, info, crosshairState);
					}
				}
			}
		}
		return foundData;
	}

	
	/**
	 * Performs fast rendering. Skips overlapping pixels
	 */
	private void renderFast(ValueAxis xAxis, ValueAxis yAxis, XYDataset dataset,
			int series, XYItemRenderer renderer, XYItemRendererState state,
			int pass, Graphics2D g2, Rectangle2D dataArea,
			PlotRenderingInfo info, CrosshairState crosshairState) {

		// Contains the x-values for the current series
		int count = dataset.getItemCount(series);
		double[] xValues = new double[count];		
				
		for (int item = 0; item < count; item++) 
			xValues[item] = dataset.getXValue(series, item);
		
		int firstItemIndex = findItemIndex(xValues, xAxis.getLowerBound(), count);
		int lastItemIndex = findItemIndex(xValues, xAxis.getUpperBound(), count);

		RectangleEdge domainEdge = getDomainAxisEdge();
		RectangleEdge rangeEdge = getDomainAxisEdge();
		
		double xAxisUpperBound = xAxis.getUpperBound();
		double yAxisUpperBound = yAxis.getUpperBound();
		int maxX = (int) Math.ceil(xAxis.valueToJava2D(xAxisUpperBound,dataArea,domainEdge)) + 1;
		int maxY = (int) Math.ceil(yAxis.valueToJava2D(yAxisUpperBound,dataArea,rangeEdge))  + 1;
		
		boolean drawnArray[][] = new boolean[maxX][maxY];
				
		for (int item = firstItemIndex; item <= lastItemIndex; item++) {

			double xValue = dataset.getXValue(series, item);
			double yValue = dataset.getYValue(series,item);
			double xCoordinate = xAxis.valueToJava2D(xValue, dataArea, domainEdge);
			double yCoordinate = yAxis.valueToJava2D(yValue,dataArea,rangeEdge);			
			int roundedX = (int) Math.round(xCoordinate);
			int roundedY = (int) Math.round(yCoordinate);
			
			// This happens if we are doing double zooming, i.e. zooming
			// before the previous zoom result has been drawn.			
			if (roundedX >= 0 && roundedY >= 0 && roundedX < maxX && roundedY < maxY) {
				
				if (drawnArray[roundedX][roundedY] == true) 
					continue;
			
				drawnArray[roundedX][roundedY] = true;
			}
			
			renderer.drawItem(g2, state, dataArea, info, this, xAxis,
						yAxis, dataset, series, item, crosshairState, pass);
								
		}
	}

	
	private int findItemIndex(double[] values, double value, int maxCount) {		
		int itemIndex = Arrays.binarySearch(values, value);
		
		if (itemIndex < 0) 
			itemIndex = -itemIndex - 1;
		
		itemIndex = Math.max(0, Math.min(itemIndex, maxCount - 1));
		return itemIndex;
	}
		
}

[david.gilbert]

Does anyone have an idea if this would work.

It's a valid approach for pixel based output. But Java2D's Graphics2D class (which JFreeChart does all its rendering through) supports vector formats too (for example, PDF) and for those formats you could lose output fidelity by assuming some "pixel" size.

This is where I think some form of "rendering hints" mechanism needs to be added to JFreeChart so that specific optimisations can be employed where it makes sense.

[SamiLakka]

Could you specify what you mean with rendering hints. Should there be some higher level renderer that would use Java2D to do the rendering but also provide support for pluggable optimizers (e.g. pixel optimizer that I described)?

Then it would be up to user to choose whether to use optimizers or not. This way the optimizers could lose some of the generality of Java2D.

If you have any ideas I could try to help with this. I have noticed that people are using JFreeChart with large datasets so any kind of optimization would probably help lot of users.

[david.gilbert]

I'm thinking of a mechanism very similar to that used by the Graphics2D class so that we can pass some "hints" to the XYItemRenderer (or CategoryItemRenderer), for instance that our maximum output resolution is 72 dpi. So the renderer can take different paths based on the hint(s). If there is no resolution hint, the renderer can continue doing what it does now, drawing everything to a high-level of precision (and taking more time), or if the hint says that the max resolution is 72dpi then renderer can drop some drawing operations that won't be visible at that resolution (which can be common for large datasets).