Crude scaling benchmarks of JFreeChart - YMMV

[Developer Dude]

I am assessing a number of reporting and charting frameworks. Since JFreeChart and JFreeReports are used in at least one of these (Jasper) I was interested to see how they scaled. The application would be generating tens of thousands of reports each night, each with at least one graphic (pie chart or maybe a line graph) included in either a web page or an emailed PDF file.

Either the image files will be generated in a batch, or depending on the number of concurrent users the image file may be created on the fly - or both.

Here are the results of my crude tests:

Each test was run 3 times and the average taken. The JVM (1.5.15) was warmed up with ten iterations to invoke any compiler optimizations before timing started. The same data (created once) was used for every iteration.

Evironment: The tests were run inside an Eclipse environment on a 2.4 gHz dual CPU Dell T3400 with 2GB of RAM running Debian. Other processes (such as an email client) were running concurrently with the test, but probably did not heavily influence the results.

* JFreeChart
o Piechart with 6 slices. 250x250 pixels
+ Chart creation - 100,000 charts - 41 seconds
+ Chart creation then create buffered image - 10,000 charts - 43 seconds
+ Chart creation then PNG encoded from image - 1,000 charts - 34 seconds
+ Chart creation then saved as a PNG file - 1,000 charts - 32 seconds

o Piechart with 6 slices. 500x500 pixels
+ Chart creation then saved as a PNG file - 1,000 charts - 76 seconds

o XYChart with 6 data points. 250x250 pixels
+ Chart creation - 100,000 charts - 27 seconds
+ Chart creation then create buffered image - 10,000 charts - 40 seconds
+ Chart creation then PNG encoded from image - 1,000 charts - 36 seconds
+ Chart creation then saved as a PNG file - 1,000 charts - 38 seconds

o XYChart with 6 data points. 500x500 pixels
+ Chart creation - 100,000 charts - 28 seconds
+ Chart creation then create buffered image - 10,000 charts - 130 seconds
+ Chart creation then PNG encoded from image - 1,000 charts - 82 seconds
+ Chart creation then saved as a PNG file - 1,000 charts - 82 seconds

* Notes on JFreeChart:
o Times (one run to another) varied by a few seconds at most.
o Creation of the charts is obviously faster than the creating of a buffered image, encoding, or writing to file - by orders of magnitude.
o Encoding is about as expensive as file creation.
o Chart size does affect performance, but not as much as the increase in size would suggest (500x500 is four times the size of 250x250).
o Experimentally increasing the dataset size of the XYChart by five times increased the file creation time by 50%, but not the creation time of the chart in memory.
o Using TOP I noted that the benchmarks used approximately 100% of one CPU when running.
o With each benchmark I scaled down the number of iterations to keep the time consumed manageable.

I used the simplest example I could manage. A call to the static method in the chart factory with the simplest options that seemed reasonable. The VM settings were the defaults.

[Taqua]

You should also notice:

Within JFreeReport when exporting PDFs, we do not generate buffered images (as this is ugly for printing); we generate Vector-images instead. AFAIK, JasperReports does the same.

Chart-Generating in reporting servers is triggered by the reporting engine, and so if 1000 users request the same report with the same chart, usually they generate each chart independently. In such cases, instead of complex caching, most organizations simply create the report once and publish the generated document somewhere central.

Also: The report-generation and therefore chart-generation always runs in parallel. So if you have 1000 users and 1000 CPUs and all hit the server at the same time, then each request consumes one CPU. So instead of having to wait 40000 seconds (40 secs chart generation executed serially for 1000 users) all users get the report/chart in 40 seconds. (Yeah, that example is simplified, but I guess thats ok here ).

[Developer Dude]

You should also notice:

Within JFreeReport when exporting PDFs, we do not generate buffered images (as this is ugly for printing); we generate Vector-images instead. AFAIK, JasperReports does the same.

That's good to know. I had come across an example of the difference between the two and it is quite apparent in the quality of the PDF report.

Chart-Generating in reporting servers is triggered by the reporting engine, and so if 1000 users request the same report with the same chart, usually they generate each chart independently. In such cases, instead of complex caching, most organizations simply create the report once and publish the generated document somewhere central.

Also: The report-generation and therefore chart-generation always runs in parallel. So if you have 1000 users and 1000 CPUs and all hit the server at the same time, then each request consumes one CPU. So instead of having to wait 40000 seconds (40 secs chart generation executed serially for 1000 users) all users get the report/chart in 40 seconds. (Yeah, that example is simplified, but I guess thats ok here ).

There are a number of different approaches a person could take if they had full control over how the report was served. We are currently discussing what will work best for our needs.

The question I was trying to answer was a small but important one; how did the rendering of a chart to an image (displayable in a browser) scale? After looking at a number of frameworks (Cognos, Crystal, Jasper, BIRT) and hearing how some of them support only 25 to 60 concurrent users (per CPU - with an expensive per CPU license) I was wondering about the bottlenecks - one of which is rendering.

Of course, there is a lot more to the BI/reporting frameworks than just the rendering, but if we decide to roll our own or use Jasper, or use JFreeChart with our own framework, then this is a bit of info that we need to know.

I had heard about Jasper charts not rendering quickly, but it was just a rumor and the best way to know something is to measure it. From the results, I would *guess* that JFreeChart does an okay job of rendering simple charts with small datasets and that there doesn't seem to be an systemic problem in this regard.

Also, as I mentioned in the notes, it does seem that the encoding/writing to file is what takes the most time - which isn't surprising to me.