large dataset problem

[jiny]

I should say JFreeChart is very flexible, however, for large data sets, the performance problem arises. I have a scatter plot that could display multiple series and could add points or delete points by user. It's working fine for smaller data sets, but when the data set is large, for e.g., 400, 000
points, it will take about 40secs to display the chart, adding point or deleting point is also slow. Even with anti-aliasing off.
I have searched the forum for sloving the performance issue. But the FastScatterPlot class using array is not suitable to my case, since I need to use XYSeries to add points or delete points. Is it possible to improve the performance for this case?
It seems that updating dataset will cause the chart repaint itself. Is there any better way?

Your help is greatly appreciated.

[dbasten]

I'm not sure what you've tried so far, but I would try to measure where you are spending all of your time.

Do multiple tests with different data volumes to establish a solid baseline of how much time is spent adding data to the datasets, generating images, doing UI refreshes, etc.

Different actions could be considered based on where most of the time is spent. For example, I've seen cases where performance was 'slow' but testing showed that I was trying to plot so much data that it ultimately resulted in a 1 MB image being sent over the network to a web browser. My wait time was mostly spent in waiting for the image to be sent/received.

I hope this helps.

Regards,
David

[Guest]

Thanks for the suggestion. But I really want to know if there are any ways to improve performance in my case. I've found out creating dataset like following is a killer, but I couldn't find a better one since I have to use XYSeries class to add or delete points.

Code: Select all

 try{
      for(int k=0; k<length; k++){
        series.add(dataMatrix[xIndex][k], dataMatrix[yIndex][k]);
      }
    }
    catch(Exception e){
      e.printStackTrace();
    }

Another thing is I try to use SwingWorker class to create a thread to executing the scatter plot code instead of the event dispatching thread, since when the user select "scatter plot" menu item from "view" menu, it will take much time to display the chart and the GUI is freezen if the dataset is large. I tried but it doesn't work. I'd like to know is it possible to do that in this way. Any suggestion is greatly appreciated.

[BigWillyStyle42]

You should definitely be calling it with false as a 3rd argument. That will prevent it from redrawing the graph. Then once you've added all the data points you can readd the last point without the 3rd argument and it'll draw all the points.

Code: Select all

for(int k = 0; k < length; k++)
{
  series.add(dataMatrix[xIndex][k], dataMatrix[yIndex][k], false);
}

[jiny]

Thanks for the suggestion. I tried that method, but it doesn't improve much. So can I say ploting a scatter plot with about 400, 000 points is not much practical and the performance will be definitely bad?

[BigWillyStyle42]

I don't know, I've never come close to having a dataset that large. For my datasets of 10K ish points, I noticed a significant improvement when using the method I showed above. I'd almost say it was a 50% decrease in time needed to add and draw all the points.

[Guest]

Hi BigWillyStyle42,

Thanks for the follow up. I used the code as following to create the data set and tested it on 50k points, however, I can only get a 2% decrease in time needed to draw all the points. (CPU 2.2GHz, windows 2000)

Code: Select all

 int temp = totalPoints - 1;
      for(int k=0; k<temp; k++){
        series.add(dataMatrix[xIndex][k], dataMatrix[yIndex][k], false);
      }
     series.add(dataMatrix[xIndex][temp], dataMatrix[yIndex][temp]);

I'm wondering is it ok to post your code for solving this problem. Thanks.

[BigWillyStyle42]

That is essentially the exact same code I used. The only difference seems to be our machines. I've got a p3 700MHz which is ridiculously slow, so maybe I'm exagerrating a bit with the improvement, but it sure seemed like 50% to me.

[havfrue]

I think the rendering would be faster if whole arrays are rendered instead of one by one item. This means that you have to create your own dataset that implements these methods:
public double[] getXValues(int serie);
public double[] getYValues(int serie);
and your own renderer that renders the whole array at once.
You could also change the XYPlot render method so that it only renders the points that are visible (by default JFreechart renders all points each time and don't care if they are inside the visible rectangle)

[havfrue]

you might also want to take a look at this post:
XYItemRenderer too slow - why not draw all points at once?
(Mon Jun 14, 2004 4:05 pm)

[havfrue]

and this post: Why do renderers still use getXValue method??
(Mon Jun 07, 2004 12:40 pm)

[jiny]

Hi havfrue,

Thanks for your suggestion. I'll try that later.
Currently I'd like to know whether or not you get much performance gain after modifying these code? Did you submit any patches for specific renderers?

many thanks,
jin

[havfrue]

I have actually started looking into if there are other software that is better suited when rendering large datasets so I haven't implemented this yet. Part of why I haven't fixed this is that the jfreechart API changes quite frequently and this means I would have to change my code quite frequently too...
Take a look at pathch ID 969389, 969393

I would suggest to use datasets that contains
double[] getXArray()
double[] getYArray() or just submit the arrays directly to a renderer containing a method like this one..

Code: Select all

public void drawItems(
		Graphics2D g2,
		XYItemRendererState state,
		Rectangle2D dataArea,
		PlotRenderingInfo info,
		XYPlot plot,
		ValueAxis domainAxis,
		ValueAxis rangeAxis,
		FastXYDataset dataset,
		int series,
		CrosshairState crosshairState,
		int pass) {
		Shape entityArea = null;
		EntityCollection entities = null;
		if (info != null)
			entities = info.getOwner().getEntityCollection();
		PlotOrientation orientation = plot.getOrientation();
		java.awt.Paint seriesPaint = getSeriesPaint(series);
		java.awt.Stroke seriesStroke = getSeriesStroke(series);
		g2.setPaint(seriesPaint);
		g2.setStroke(seriesStroke);

		int itemCount = dataset.getItemCount(series);
		double[] xArray = dataset.getXValues(series);
		double[] yArray = dataset.getYValues(series);

		//		 double x1n = dataset.getX(series, item);
		//		 double y1n = dataset.getY(series, item);
		//		 if (Double.isNaN(y1n) || Double.isNaN(x1n))
		//			 return;
		//		 double x1 = x1n;
		//		 double y1 = y1n;

		org.jfree.ui.RectangleEdge xAxisLocation = plot.getDomainAxisEdge();
		org.jfree.ui.RectangleEdge yAxisLocation = plot.getRangeAxisEdge();

		// double transX1 = domainAxis.valueToJava2D(x1, dataArea, xAxisLocation);
		// double transY1 = rangeAxis.valueToJava2D(y1, dataArea, yAxisLocation);

		if (getPlotLines() && (xArray != null) && (xArray.length > 0)) {
			Rectangle2D boundingBox = dataArea;
			double[] transXArray = new double[itemCount];
			double[] transYArray = new double[itemCount];

			XYToolTipGenerator tipGenerator = getSeriesToolTipGenerator(series);
			XYURLGenerator urlGenerator = getURLGenerator();
            
			for (int index = 0; index < itemCount; index++) {
				transXArray[index] = domainAxis.valueToJava2D(xArray[index], dataArea, xAxisLocation);
				transYArray[index] = rangeAxis.valueToJava2D(yArray[index], dataArea, yAxisLocation);

				entityArea =
					new java.awt.geom.Rectangle2D.Double(transXArray[index] - 2D, transYArray[index] - 2D, 4D, 4D);

				String tip = null;
				if (tipGenerator != null) {
					tip = tipGenerator.generateToolTip(dataset, series, index);
				}
				String url = null;
				if (urlGenerator != null) {
					url = urlGenerator.generateURL(dataset, series, index);
				}
				XYItemEntity entity = new XYItemEntity(entityArea, dataset, series, index, tip, url);
				entities.addEntity(entity);
			}

			if (orientation == PlotOrientation.HORIZONTAL) {
				_polyShape.setCoordinates(boundingBox, itemCount, false, transYArray, transXArray);
			} else {
				_polyShape.setCoordinates(boundingBox, itemCount, false, transXArray, transYArray);
			}

			g2.draw(_polyShape);

			//            double x0n = dataset.getX(series, item - 1);
			//            double y0n = dataset.getY(series, item - 1);
			//            if (!Double.isNaN(y0n) && !Double.isNaN(x0n)) {
			//                double x0 = x0n;
			//                double y0 = y0n;
			//                boolean drawLine = true;
			//                if (getPlotDiscontinuous()) {
			//                    int numX = dataset.getItemCount(series);
			//                    double minX = dataset.getX(series, 0);
			//                    double maxX = dataset.getX(series, numX - 1);
			//                    drawLine = x1 - x0 <= ((maxX - minX) / (double) numX) * getGapThreshold();
			//                }
			//                if (drawLine) {
			//                    double transX0 = domainAxis.valueToJava2D(x0, dataArea, xAxisLocation);
			//                    double transY0 = rangeAxis.valueToJava2D(y0, dataArea, yAxisLocation);
			//                    if (Double.isNaN(transX0)
			//                        || Double.isNaN(transY0)
			//                        || Double.isNaN(transX1)
			//                        || Double.isNaN(transY1))
			//                        return;
			//                    if (orientation == PlotOrientation.HORIZONTAL)
			//                        state.workingLine.setLine(transY0, transX0, transY1, transX1);
			//                    else if (orientation == PlotOrientation.VERTICAL)
			//                        state.workingLine.setLine(transX0, transY0, transX1, transY1);
			//                    if (state.workingLine.intersects(dataArea))
			//                        g2.draw(state.workingLine);
			//
			//                }
			//            }
			//        }
			//        if (getPlotShapes()) {
			//            Shape shape = getSeriesShape(series);
			//            if (orientation == PlotOrientation.HORIZONTAL)
			//                shape = createTransformedShape(shape, transY1, transX1);
			//            else if (orientation == PlotOrientation.VERTICAL)
			//                shape = createTransformedShape(shape, transX1, transY1);
			//            if (shape.intersects(dataArea))
			//                if (getItemShapeFilled(series, item))
			//                    g2.fill(shape);
			//                else
			//                    g2.draw(shape);
			//            entityArea = shape;
			//        }
			//        if (getPlotImages()) {
			//            Image image = getImage(plot, series, item, transX1, transY1);
			//            if (image != null) {
			//                Point hotspot = getImageHotspot(plot, series, item, transX1, transY1, image);
			//                g2.drawImage(image, (int) (transX1 - hotspot.getX()), (int) (transY1 - hotspot.getY()), null);
			//                entityArea =
			//                    new java.awt.geom.Rectangle2D.Double(
			//                        transX1 - hotspot.getX(),
			//                        transY1 - hotspot.getY(),
			//                        image.getWidth(null),
			//                        image.getHeight(null));
			//            }
			//        }
			//        if (isItemLabelVisible(series, item))
			//            drawItemLabel(g2, orientation, dataset, series, item, transX1, transY1, y1 < 0.0D);
			//        updateCrosshairValues(crosshairState, x1, y1, transX1, transY1, orientation);

			//            if (entities != null) {
			//				for (int index = 0; index < itemCount; index++) {
			//                 if (entityArea == null)
			//                        entityArea =
			//                            new java.awt.geom.Rectangle2D.Double(
			//                                transXArray[index] - 2D,
			//                                transYArray[index] - 2D,
			//                                4D,
			//                                4D);
			//				              
			//                    String tip = null;
			//                    XYToolTipGenerator generator = getSeriesToolTipGenerator(series);
			//                    if (generator != null) {
			//
			//                        tip = generator.generateToolTip(dataset, series, index);
			//                    }
			//                    String url = null;
			//                    if (getURLGenerator() != null) {
			//                        url = getURLGenerator().generateURL(dataset, series, index);
			//                    }
			//                    XYItemEntity entity = new XYItemEntity(entityArea, dataset, series, index, tip, url);
			//                    entities.addEntity(entity);
			//                }
			//            }
		}
	}