To Those Who Use JFreeChart for Dynamic Plotting/Large Sets

A discussion forum for JFreeChart (a 2D chart library for the Java platform).
Locked
RoyW
Posts: 93
Joined: Wed Apr 23, 2008 7:42 pm
Contact:

Post by RoyW » Sat May 10, 2008 7:48 am

Here is a demo Candlestick chart.
Use Drag-Zoom and zoom in to March 28 2003. Hover over the CandleStick at that point and you will get a tooltip.
If you read the comments in the code and use a normal ChartPanel you will find the tooltip will not display.

Code: Select all

import org.jfree.chart.axis.DateAxis;
import org.jfree.chart.axis.NumberAxis;
import org.jfree.chart.axis.SegmentedTimeline;
import org.jfree.chart.renderer.xy.CandlestickRenderer;
import org.jfree.chart.plot.XYPlot;
import org.jfree.chart.JFreeChart;
import org.jfree.chart.ChartPanel;
import org.jfree.chart.TooltipChartPanel;
import org.jfree.data.xy.XYDataset;
import org.jfree.data.xy.AbstractXYDataset;
import org.jfree.data.xy.DefaultOHLCDataset;
import org.jfree.data.xy.OHLCDataItem;

import javax.swing.*;
import java.awt.*;
import java.util.ArrayList;
import java.util.StringTokenizer;
import java.util.Date;
import java.util.Collections;
import java.net.URL;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.text.DateFormat;
import java.text.SimpleDateFormat;

public class TooltipDemoCandle extends JFrame {
    public TooltipDemoCandle(String symbol) {
        super("TooltipDemoCandle");
        this.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);

        DateAxis domainAxis       = new DateAxis("Date");
        NumberAxis rangeAxis        = new NumberAxis("Price");
        CandlestickRenderer renderer = new CandlestickRenderer();
        XYDataset dataset          = getDataSet(symbol);

        XYPlot mainPlot = new XYPlot(dataset, domainAxis, rangeAxis, renderer);
        //Uncomment this to test horizontal charts (Horizontal candlestick charts ?!?)
//        mainPlot.setOrientation(PlotOrientation.HORIZONTAL);

        //Do some setting up, see the API Doc
        renderer.setSeriesPaint(0, Color.BLACK);
        renderer.setDrawVolume(false);
        renderer.setBaseCreateEntities(false);  //**** BUG : Candlestick renderer ignores this (in vers 1.0.9)!!!
        rangeAxis.setAutoRangeIncludesZero(false);
        domainAxis.setTimeline( SegmentedTimeline.newMondayThroughFridayTimeline() );


        JFreeChart chart = new JFreeChart(symbol, null, mainPlot, false);
        //****** uncomment and use ChartPanel instead of TooltipChartPanel then zoom in to March 28 2003; the tooltip will not display (in vers 1.0.9)!!!
        //ChartPanel chartPanel = new ChartPanel(chart);
        ChartPanel chartPanel = new TooltipChartPanel(chart);
        chartPanel.setPreferredSize(new Dimension(600, 300));

        this.add(chartPanel);
        this.pack();
    }
    protected AbstractXYDataset getDataSet(String symbol) {
        //This is the dataset we are going to create
        DefaultOHLCDataset result = null;
        //This is the data needed for the dataset
        OHLCDataItem[] data;

        //This is where we go get the data, replace with your own data source
        data = getData(symbol);

        //Create a dataset, an Open, High, Low, Close dataset
        result = new DefaultOHLCDataset(symbol, data);

        return result;
    }
    //This method uses yahoo finance to get the OHLC data
    protected OHLCDataItem[] getData(String symbol) {
        java.util.List<OHLCDataItem> dataItems = new ArrayList<OHLCDataItem>();
        try {
            String strUrl= "http://ichart.finance.yahoo.com/table.csv?s="+symbol+"&a=0&b=1&c=2000&d=3&e=30&f=2008&ignore=.csv";
            URL url = new URL(strUrl);
            BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
            DateFormat df = new SimpleDateFormat("y-M-d");

            String inputLine;
            in.readLine();
            while ((inputLine = in.readLine()) != null) {
                StringTokenizer st = new StringTokenizer(inputLine, ",");

                Date date       = df.parse( st.nextToken() );
                double open     = Double.parseDouble( st.nextToken() );
                double high     = Double.parseDouble( st.nextToken() );
                double low      = Double.parseDouble( st.nextToken() );
                double close    = Double.parseDouble( st.nextToken() );
                double volume   = Double.parseDouble( st.nextToken() );
                double adjClose = Double.parseDouble( st.nextToken() );

                OHLCDataItem item = new OHLCDataItem(date, open, high, low, close, volume);
                dataItems.add(item);
            }
            in.close();
        }
        catch (Exception e) {
            e.printStackTrace();
        }
        //Data from Yahoo is from newest to oldest. Reverse so it is oldest to newest
        Collections.reverse(dataItems);

        //Convert the list into an array
        OHLCDataItem[] data = dataItems.toArray(new OHLCDataItem[dataItems.size()]);

        return data;
    }

    public static void main(String[] args) {
        new TooltipDemoCandle("MSFT").setVisible(true);
    }
}


mkivinie
Posts: 51
Joined: Wed Jul 06, 2005 8:35 am

Post by mkivinie » Tue May 13, 2008 9:08 am

Hi Roy!

Your code might help me as well, so I actually studied it :)
You are assuming that there exists only one Domain and Range axis and all datasets are mapped against them.
RoyW wrote:

Code: Select all

            ValueAxis domainAxis = xyPlot.getDomainAxis();
            ValueAxis rangeAxis  = xyPlot.getRangeAxis();
I propose an improvement to get the correct axes for the dataset where you are looping the datasets.
Also it is possible that a returned dataset is null.

Code: Select all

...
            for(int datasetIndex=0 ; datasetIndex<datasetCount ; datasetIndex++){
                XYDataset dataset = xyPlot.getDataset(datasetIndex);
                if ( dataset == null ) 
                   continue;
 
                ValueAxis domainAxis = xyPlot.getDomainAxisForDataset(datasetIndex);
                ValueAxis rangeAxis  = xyPlot.getRangeAxisForDataset(datasetIndex);

...
Edit: forgot to mention that I am referring to the TooltipChartPanel code

RoyW
Posts: 93
Joined: Wed Apr 23, 2008 7:42 pm
Contact:

Post by RoyW » Tue May 13, 2008 8:17 pm

mkivinie, good catch.
I have been experimenting with the code as I read in another thread somewhere about wanting to return all entities where the mouse was clicked.

I was thinking a general purpose method is required that takes a rectangle and returns a list of objects (similar to entities). Each object would contain the dataset, series-index and item-index of all datapoints in the rectangle. That method could then be used for tooltips, mouse moved, mouse clicked. It could also be used to draw a rectangle on the chart and select all point in the rectangle.

So far the code I need this for is only XYPlots and for tooltips and mouse clicked so it is kindof specialized for my app.
The answer does not come from thinking outside the box, rather the answer comes from realizing the truth; There is no Box. my js site

tharter
Posts: 3
Joined: Mon Jun 09, 2008 7:42 pm

For us RCP guys, TooltipChartComposite!

Post by tharter » Mon Jun 09, 2008 7:53 pm

This is just a Frankensteining of the relevant parts of TooltipChartPanel into the ChartComposite to create, wallah!, TooltipChartComposite. I got FastXYPlot working, and that definitely helped rendering speed (20k point Candlestick, still pretty slow, but now down to under 10 seconds on my E6600 1.8ghz). That of course lost the tooltips.

If you replace ChartComposite with this class it appears your FastXYPlot will now draw tooltip text once again, which it won't do on the standard Chart(Composite|Panel).

Great work guys. This is really saving me a bunch of work. Now for double buffering...

Here's the relevant section you need to insert into ChartComposite in place of its existing getToolTipText() method. It is identical to the one in TooltipChartPanel except for 2 or 3 lines where I had to deal with the SWT vs AWT versions of a couple objects.

NOTE: I had to comment out a couple event handlers to get this to compile, which I haven't tracked down the reason for as of yet. Maybe SWT version skew. So you MAY have to fiddle a bit to get this to work. So far it seems fine with the CandlestickRenderer, but I have not yet played with much else.

/**
* Returns a string for the tooltip.
*
* @param e the mouse event.
*
* @return A tool tip or <code>null</code> if no tooltip is available.
*/
public String getToolTipText(org.eclipse.swt.events.MouseEvent e) {
// OK, we're using an SWT MouseEvent here, not much difference...
String result = getTooltipAtPoint(new Point(e.x,e.y));
if( result != null)
return result;
else
return super.getToolTipText();
}

private static int HOTSPOT_SIZE = 5;
//TODO Add code to set and get these values
private HighLowItemLabelGenerator hiLoTips = new HighLowItemLabelGenerator();
private StandardXYToolTipGenerator xyTips = new StandardXYToolTipGenerator();

/**
* This method attempts to get a tooltip by converting the screen X,Y into Chart Area X,Y
* and then looking for a data point in a data set that lies inside a hotspot around that value.
* @param point The Java 2D point
* @return A string for the data at the point or null if no data is found.
*/
protected String getTooltipAtPoint(Point point) {
String result = null;

Point2D translatedPoint = this.translateScreenToJava2D(point);
Plot plot = this.getChart().getPlot();
PlotRenderingInfo info = this.getChartRenderingInfo().getPlotInfo();
if (plot instanceof CombinedDomainXYPlot) {
int index = info.getSubplotIndex(translatedPoint);
if (index < 0)
index = 0;
plot = (Plot) ((CombinedDomainXYPlot) plot).getSubplots().get(index);
info = this.getChartRenderingInfo().getPlotInfo().getSubplotInfo(index);
}
if (plot != null && plot instanceof XYPlot) {
XYPlot xyPlot = (XYPlot) plot;
ValueAxis domainAxis = xyPlot.getDomainAxis();
ValueAxis rangeAxis = xyPlot.getRangeAxis();
// had to switch to SWT's rectangle here.
Rectangle screenArea = this.scale(info.getDataArea());

double hotspotSizeX = HOTSPOT_SIZE * this.getScaleX();
double hotspotSizeY = HOTSPOT_SIZE * this.getScaleY();
double x0 = point.getX();
double y0 = point.getY();
double x1 = x0 - hotspotSizeX;
double y1 = y0 + hotspotSizeY;
double x2 = x0 + hotspotSizeX;
double y2 = y0 - hotspotSizeY;
RectangleEdge xEdge = RectangleEdge.BOTTOM;
RectangleEdge yEdge = RectangleEdge.LEFT;
//Switch everything for horizontal charts
if (xyPlot.getOrientation() == PlotOrientation.HORIZONTAL) {
hotspotSizeX = HOTSPOT_SIZE * this.getScaleY();
hotspotSizeY = HOTSPOT_SIZE * this.getScaleX();
x0 = point.getY();
y0 = point.getX();
x1 = x0 + hotspotSizeX;
y1 = y0 - hotspotSizeY;
x2 = x0 - hotspotSizeX;
y2 = y0 + hotspotSizeY;
xEdge = RectangleEdge.LEFT;
yEdge = RectangleEdge.BOTTOM;
}

// OK, here we have to get ourselves back into AWT land...
Rectangle2D r2d = new Rectangle2D.Double();
r2d.setRect(screenArea.x, screenArea.y, screenArea.width, screenArea.height);

double tx0 = domainAxis.java2DToValue(x0, r2d, xEdge);
double ty0 = rangeAxis.java2DToValue(y0, r2d, yEdge);
double tx1 = domainAxis.java2DToValue(x1, r2d, xEdge);
double ty1 = rangeAxis.java2DToValue(y1, r2d, yEdge);
double tx2 = domainAxis.java2DToValue(x2, r2d, xEdge);
double ty2 = rangeAxis.java2DToValue(y2, r2d, yEdge);

int datasetCount= xyPlot.getDatasetCount();
for(int datasetIndex=0 ; datasetIndex<datasetCount ; datasetIndex++){
XYDataset dataset = xyPlot.getDataset(datasetIndex);
int seriesCount = dataset.getSeriesCount();
for(int series=0 ; series<seriesCount ; series++){
int itemCount = dataset.getItemCount(series);
if(dataset instanceof OHLCDataset) {
//This could be optimized to use a binary search for x first
for(int item = 0 ; item < itemCount ; item++){
double xValue = dataset.getXValue(series, item);
double yValueHi = ((OHLCDataset)dataset).getHighValue(series, item);
double yValueLo = ((OHLCDataset)dataset).getLowValue(series, item);
//Check hi lo and swap if needed
if(yValueHi < yValueLo)
{
double temp = yValueHi;
yValueHi = yValueLo;
yValueLo = temp;
}
//Check if the dataset 'X' value lies between the hotspot (tx1 < xValue < tx2)
if(tx1 < xValue && xValue < tx2)
//Check if the cursor 'y' value lies between the high and low (low < ty0 < high)
if(yValueLo<ty0 && ty0 < yValueHi){
return hiLoTips.generateToolTip(dataset, series, item);
}
}
} else {
//This could be optimized to use a binary search for x first
for(int item = 0 ; item < itemCount ; item++){
double xValue = dataset.getXValue(series, item);
double yValue = dataset.getYValue(series, item);
//Check if the dataset 'X' value lies between the hotspot (tx1< xValue < tx2)
if(tx1 < xValue && xValue < tx2)
//Check if the dataset 'Y' value lies between the hotspot (ty1 < yValue < ty2)
if(ty1 <yValue && yValue <ty2){
return xyTips.generateToolTip(dataset, series, item);
}
}
}
}
}
}

return result;
}

Chh
Posts: 1
Joined: Wed Jul 02, 2008 8:08 am

Post by Chh » Wed Jul 02, 2008 8:18 am

Found a bug in StandardXYItemRendererFast.java

Drawing line between first and second data point doesn't occur.

Original code:

Code: Select all

    public void drawItem(Graphics2D g2, XYItemRendererState state,
                         Rectangle2D dataArea, PlotRenderingInfo info,
                         XYPlot plot, ValueAxis domainAxis, ValueAxis rangeAxis,
                         XYDataset dataset, int series, int item,
                         CrosshairState crosshairState, int pass,
                         PlotIndexes currentIndex) {

        [...]

            } else if (item != 0) {
                // get the previous data point...
                double x0 = dataset.getXValue(series, item - currentIndex.getPreviousDrawnItem());
                double y0 = dataset.getYValue(series, item - currentIndex.getPreviousDrawnItem());

        [...]
To fix, I changed the calculation of the previous data point to the following:

Code: Select all

    public void drawItem(Graphics2D g2, XYItemRendererState state,
                         Rectangle2D dataArea, PlotRenderingInfo info,
                         XYPlot plot, ValueAxis domainAxis, ValueAxis rangeAxis,
                         XYDataset dataset, int series, int item,
                         CrosshairState crosshairState, int pass,
                         PlotIndexes currentIndex) {

        [...]

            } else if (item != 0) {
                // get the previous data point...
            	int prev = currentIndex.getPreviousDrawnItem();
            	int prevIndex = prev == 0 ? 0 : item - prev;
                double x0 = dataset.getXValue(series, prevIndex);
                double y0 = dataset.getYValue(series, prevIndex);

        [...]

mahfoud
Posts: 4
Joined: Tue Jul 08, 2008 12:18 pm

problem avec jfreechart

Post by mahfoud » Tue Jul 08, 2008 12:40 pm

bonjour,
je suis entrain généré des graphe grâce a jfreechart mais pour le moment j'ai un problème au niveau d'affichage la courbe de tendance " régression" : mon problème j'arrive pas affiche la courbe de tendance sachant que j'ai les dates en abscisse
merci pour votre aide.

d_rambabu
Posts: 8
Joined: Mon Apr 14, 2008 8:36 am
Location: Hyderabad, India
Contact:

Post by d_rambabu » Tue Jul 29, 2008 10:58 am

Hi,
Are these enhancements and optimizations including FastXYSeries and FastXYPlot available in the latest release of JFreeChart or do we have to download the source separately and use them?

-Rambabu

skunk
Posts: 1087
Joined: Thu Jun 02, 2005 10:14 pm
Location: Brisbane, Australia

Post by skunk » Tue Jul 29, 2008 3:45 pm

d_rambabu wrote: Are these enhancements and optimizations including FastXYSeries and FastXYPlot available in the latest release of JFreeChart or do we have to download the source separately and use them?
Did you look at the supplied source code yet?

jsnel
Posts: 19
Joined: Thu Apr 19, 2007 12:00 pm

Post by jsnel » Sat Sep 06, 2008 4:25 pm

skunk wrote:
d_rambabu wrote: Are these enhancements and optimizations including FastXYSeries and FastXYPlot available in the latest release of JFreeChart or do we have to download the source separately and use them?
Did you look at the supplied source code yet?
What supplied source code are you referring too? The JFreeChart on sourceforge or the supplied source code in this thread?

paradoxoff
Posts: 1634
Joined: Sat Feb 17, 2007 1:51 pm

Re: To Those Who Use JFreeChart for Dynamic Plotting/Large Sets

Post by paradoxoff » Mon Apr 27, 2009 9:14 pm

Time to bring this thread up.
I have uploaded a FastXYShapeRenderer that uses some ideas discussed in this thread. Basically, if multiple shapes occupy the same (physical) position (same pixel index), only one shape is drawn.

The question is which shape!

If one sticks to the JFreeChart convention of drawing one item at a time, it is possible to avoid the rendering of a shape on top of an already existing shape. However, in a normal XYPlot, the shape on top is visible, and the one in the background is hidden. This inversion in rendering order can be visible and can also be irritating.

In order to keep the original rendering order of a normal XYPlot, the uploaded renderer first goes through the items in reverse order (from the highest to the lowest item index) and sorts the items out that would not be visible. The visible items of the series are then rendered. All this is happening in a single pass of drawItem(). Once a series is rendered, drawItem() immediately returns.
This approach is not really clean, as several loops in the render() method of XYPlot are unneccesary. I do believe that the "render a complete series at a time" approach could also be useful for other XYItemRenderers.

What about an additional interface XYSeriesRenderer that can draw entire series in a single method call or even XYDatasetRenderer that can do the same with entire XYDatasets?
Something like

Code: Select all

public interface XYSeriesRenderer extends XYItemRenderer{
public void drawSeries(java.awt.Graphics2D g2, XYItemRendererState state, java.awt.geom.Rectangle2D dataArea, PlotRenderingInfo info, XYPlot plot, ValueAxis domainAxis, ValueAxis rangeAxis, XYDataset dataset, int series, CrosshairState crosshairState, int pass)
public boolean getRenderEntireSeries(); 
}
public interface XYDatasetRenderer extends XYItemRenderer{
public void drawDataset(java.awt.Graphics2D g2, XYItemRendererState state, java.awt.geom.Rectangle2D dataArea, PlotRenderingInfo info, XYPlot plot, ValueAxis domainAxis, ValueAxis rangeAxis, XYDataset dataset, SeriesRenderingOrder order, CrosshairState crosshairState, int pass)
public boolean getRenderEntireDataset(); 
}
Then make AbstractXYItemRenderer implement these interfaces, return false for both the getRenderEntireXXX methods, and provide a setter for this flag.
Derived renderers could set this flag to true and implement their own logic to render entire series/datasets.
Finally, change XYPlot, in particular its render() method. If the renderer for the given index implements XYDatasetRenderer and if its getRenderEntireDataset() method returns true, call its drawDataset() method and let the renderer go through all the series, go through all the items and do all kinds of checks to make sure that only necessary items are rendered (same if the renderer implements the XYSeriesRenderer interface). Only if the renderer is an ordinary XYItemRenderer, the looping over the series and the items should be performed in the XYPlot .
This approach would make it much easier to plug high performance renderers into an XYPlot.

I have benchmarked the new renderer against an XYShapeRenderer by drawing charts onto the Graphics2D of a BufferedImage and found it 8-10 times faster for 10 reps of rendering a chart with a dataset with 100k items. With -Xmx256M, I was able to render a dataset with 1M items in a ChartPanel.

grimlock
Posts: 22
Joined: Wed Jul 14, 2004 7:50 am

Re: To Those Who Use JFreeChart for Dynamic Plotting/Large Sets

Post by grimlock » Fri Jun 05, 2009 9:07 am

Hi,

I've just stumbled across this excellent thread. I tried substituting FastXYPlot in place of XYPlot in my time series charts. They are definitely much faster but they do not look exactly the same. The difference is some points are not rendered (which is correct according to the class's description) however, if this point isn't rendered, the line between the points will be missing too and hence the charts do not look the same.

My question is is FastXYPlot meant to be a drop-in replacement for XYPlot or do I need to make further changes to my charts? Was the FastXYPlot version posted here meant for XY charts other than a Time Series (e.g. a scatter plot)?

paradoxoff
Posts: 1634
Joined: Sat Feb 17, 2007 1:51 pm

Re: To Those Who Use JFreeChart for Dynamic Plotting/Large Sets

Post by paradoxoff » Fri Jun 05, 2009 7:22 pm

Hi,
I assume that most of the optimizations indeed aim at improving the rendering speed of scatter plots.
For scatter plots, you can safely ignore points that would be rendered on an already occupied pixel, but for a line charts you might get artifacts.
One scenario is the following five x/ypoints: 0/0; 500,000/0; 500,001/100; 500,002/0; 1,000,000/0.
In a scatter chart you will see only four points, and if you ignore the 500,000/0 point the chart will still look the same.
In a line chart, you will see a spike around an x value of 500,000. This spike will be gone if you ignore the 500,000/0 or the 500,002/0 data point, and you will get a chute either between x = 0 and 500,00 or between x = 500,000 and 1,000,000.
If you need improved rendering speed for line charts, I´d recommend the SamplingXYLineRenderer.
I think that the fact that line charts and scatter charts have different criteria for data items that can be ignored w/o changing the appearance will make it quite difficult to implement a truly general solution into the XYPlot class or its subclasses. One of my next goals is to add the rendering algorithm of the SamplingXYLineRenderer to the FastXYhapeRenderer class to get a renderer that can draw both line and scatter plots (and mizxed plots) with high speed.

paradoxoff
Posts: 1634
Joined: Sat Feb 17, 2007 1:51 pm

Re: To Those Who Use JFreeChart for Dynamic Plotting/Large Sets

Post by paradoxoff » Sun Jun 07, 2009 1:16 pm

I have uploaded a FastXYLineAndShapeRenderer that combines the optimizations of the SamplingXYLineRenderer and of the fast XYShapeRenderer into a single class.
The link.
Feedback is welcome.

grimlock
Posts: 22
Joined: Wed Jul 14, 2004 7:50 am

Re: To Those Who Use JFreeChart for Dynamic Plotting/Large Sets

Post by grimlock » Tue Jun 09, 2009 2:00 am

Hi paradoxoff,

Thanks for the replies. In response to your first post, I have tried SamplingXYLineRenderer and while it is impressive in its performance, there are a few teething problems with it as can be seen in this thread:
http://www.jfree.org/phpBB2/viewtopic.p ... 309#p78309

I'll have to wait until those kinks are ironed out.

As for your second post, I am excited by it and will be putting it through its paces. Just reading through your comments on the Tracker, these two seem contradictory to me:
...and should thus produce an output that is
indistinguishable from that of an XYLineAndShapeRenderer.
- the new renderer does not only care about the "goodness" of a point but
also about the visibility, and will not draw a line that connects two
points that are outside the axes ranges.
The XYLineAndShapeRenderer will draw the line that you have omitted in the 2nd point. I have a scroll bar in my time series charts that the user can scroll through a subset of the time period (X-axis). Having the line that connects the first visible point to the previous one which is out of range is a neccessity. I'll post some more feedback once I get an opportunity to run it.

grimlock
Posts: 22
Joined: Wed Jul 14, 2004 7:50 am

Re: To Those Who Use JFreeChart for Dynamic Plotting/Large Sets

Post by grimlock » Tue Jun 09, 2009 3:10 am

Initial observations:

Running the XYLineAndShapeRenderer and FastXYLineAndShapeRenderer side by side, the FastXYLineAndShapeRenderer actually renders slower than the XYLineAndShapeRenderer (plotting 5000 points). (1800 ms (Normal) vs 2300 ms (Fast)). And for comparison's sake, the SamplingXYLineRenderer does it in 90ms.

The good news is the two charts almost look exactly the same. The one small discrepancy is that the Fast renderer renders lines over the top of the shapes of other series whereas the Normal renderer renders it below. I suspect this is an artifact of having doing only one pass in the rendering instead of the two that the Normal renderer does. It's such a small difference that I don't think it will matter if the speed improvements are there.

I will run it under a profiler and see what's the major bottleneck and post some more feedback for you.

Locked