XYPlot not releasing resources when setting dataset to null

[tomkieffer]

I have a memory problem in my JFreeChart Application (a data viewer with variable data sources). I wrote a custom XYPlot that has an addDataset() and removeDataset() method, among others. removeDataset() sets the dataset in question to null.
Now the app does two things:
You can add and remove datasets by means of a JCheckBox. Here I remark that removing a dataset (setting it to null) has absolutely no effect on memory use. No memory is freed. This is not that bad because the null datasets are reused and the amount of memory doesn't get higher than the maximum number of datasets ever needed.
The application also has a refreh Button to view data for a different time period. And here I have a real problem. The application needs about 45m at rest. When I add 3 relatively simple series (3000 items) to the plot, it still needs about the same amount of memory. When I refresh the plot by removing datasets and adding them again, memory use raises to 75m, the second time to 95m and then stays at 95 where it stays, even if I refresh more often. If I add more or bigger datasets to the plot, it runs into OutOfMemory at the first refresh.
Now can anyone explain this behaviour or parts of it? Disposing the JFrame containing the application is not releasing resources either. Is there anything to be done? I included the refreshing code, which is not very complicated (notice that removeDataset() is, as described above, a custom method that sets the dataset to null and addDataset() is a wrapper for setDataset that reuses null Datasets).

Code: Select all

private void repaintPlot(String begin, String end, String intervalString) {
        
//        this.setCursor(new Cursor(Cursor.WAIT_CURSOR));
                
//        try { //OutOfMemoryError

            int interval = Integer.valueOf(intervalString);

            if (checkDates(begin, end, intervalString)) {

                for (int i = 1; i < plot.getDatasetCount(); i++) {
                    if (plot.getDataset(i) != null) {
                        plot.removeDataset(i, true, (String)plot.getDataset(i).getSeriesKey(0)+"\n");

                    }
                }

                System.gc();

                GregorianCalendar greg;

                XYSeriesCollection dataset;
                XYSeries series;
                Object[] currentData;
                
                for (int i = 0; i < tableModel.getRowCount(); i++ ) {

                    if (tableModel.getValueAt(i, 1).equals(true)) {
                        
                        dataset = new XYSeriesCollection();
                        String seriesLabel = (String)(tableModel.getValueAt(i, 3))+" in "+(String)(tableModel.getValueAt(i, 5));
                        series = new XYSeries(seriesLabel);
                        greg = DateUtil.parseDateTime("-", ":", begin, DateUtil.YYMMDD);

                        currentData = dataModel.getData(DateUtil.parseDateTime(DateUtil.YYMMDD, "-", ":", begin),
                                DateUtil.parseDateTime(DateUtil.YYMMDD, "-", ":", end), interval, i, false);

                        int anzahlDatensaetze = ((long[]) currentData[0]).length;

                        for (int j = 0; j < anzahlDatensaetze; j++) {
                            series.add(((long[])currentData[0])[j], ((double[])currentData[1])[j], false);
                        }

                        dataset.addSeries(series);
                        XYItemRenderer rend = getRenderer((String)tableModel.getValueAt(i, 0));
                        plot.addDataset(dataset, rend, 1.0, 0, seriesLabel+"\n");

                        hashtable.put((String)(tableModel.getValueAt(i, 2)), plot.getIndexOfDataset(dataset));
                        indexArray[i] = true;

                    }
                }
            }
        /*
        } catch (java.lang.OutOfMemoryError error) {
            JOptionPane.showMessageDialog(this, "The Java Virtual Machine is out of memory.\n Try " +
                    "to use a smaller time period or a bigger interval or restart the application. You" +
                    "can also try to allocate more memory.\n" +
                    "Class Auswertung in Package dscaem.", null, JOptionPane.ERROR_MESSAGE);
            System.out.println(error.toString());

        } finally {
            this.setCursor(new Cursor(Cursor.DEFAULT_CURSOR));
            //hier sollten die Häkchen wieder rausgenommen werden, aber das würde den Listener
            //aktivieren und removeDataset() würde einen Fehler verursachen
        }*/
    }

[david.gilbert]

I suspect that somewhere in your code you are retaining references to all these datasets so that the garbage collector can never reclaim the memory they are using. I'd be surprised if the bug is in the JFreeChart code (something like that would have been noticed by now), but you can never be sure. The only efficient way we can trace such a bug, though, is if you can supply a self-contained test program that reproduces the problem.

Have a go at producing the test app - I often find that the attempt to reduce bugs to a minimal test case reveals the source of the problem.

[Carl Manaster]

Hi, Tom,

I think this bit of code:

Code: Select all

for (int i = 1; i < plot.getDatasetCount(); i++) 
	if (plot.getDataset(i) != null) 
		plot.removeDataset(i, true, (String) plot.getDataset(i).getSeriesKey(0) + "\n");

is problematic, although I'm not sure it's the source of your memory problems.

Say there are 3 datasets: d1, d2, d3. I understand starting i at 1 because getDataset is 1-based, but seeing that always catches my eye.

Since you're removing the ith dataset, iterating over them forward, you'll be skipping. Assuming none of the datasets is null, the first time through the loop you'll remove d1, leaving {d2, d3}
the second time through the loop, you'll remove the second dataset, which is now d3, leaving d2.

If there were more - like {d1, d2, d3, d4}, you would delete d1, then d3, then the 3rd element, which is now beyond the limit. Basically you wind up removing every other element.

Work on this bit of code and see if things don't get better. The classic way to handle deleting is to loop backwards, although there are other perfectly legitimate approaches.

Peace,
--Carl

[RichardWest]

Code: Select all

for (int i = 1; i < plot.getDatasetCount(); i++) {
    if (plot.getDataset(i) != null) {
        plot.removeDataset(i, true, (String) plot.getDataset(i).getSeriesKey(0) + "\n");
    }
}

That would certainly be the problem. Each time a dataset is removed, the indexes get shifted. Even if one dataset is removed, this code would be unsafe unless removeDataset checks the index condition. removeDataset must be defined in a custom class since it is not part of the API for XYPlot. While I dislike how this code looks, you could do the following:

Code: Select all

for (int i = 1; i <= plot.getDatasetCount(); i++) {
    if (plot.getDataset(i) != null) {
        plot.removeDataset(i, true, (String) plot.getDataset(i).getSeriesKey(0) + "\n");
        --i;
    }
}

Another option would be to reference an unmodifiable array to check for a null dataset and remove any null datasets from the modifiable array. The unmodifiable array could then be discarded since it would only be used for this purpose. If done intelligently, the extra overhead is minimal. Situations like this are where iterators in both Java and C++ come in handy since most ensure correct behavior even as the underlying datastructure is being modified.

[Carl Manaster]

I forgot to add:

and since the count is 1-based rather than zero-based, you should probably be using

i <= (instead of <) plot.getDatasetCount

as your loop exit condition.

[skunk]

1-based?

[Carl Manaster]

Hi, Skunk,

1-based?

Since the loop starts at 1, I'm assuming that plot.getDataset() addresses its first element by the value 1, rather than the value 0. So in a 3-dataset list, you would want to cover the values 1, 2, and 3.

for (int i = 1; i < 3; i++)

covers only 1 & 2,

for (int i = 1; i <=3; i++)

covers all 3.

Peace,
--Carl

[RichardWest]

1-based?

Good spot. getDataset(int) is zero-based as one would expect. Another bug with the for loop then since the loop is one-based. It is funny how everyone focuses on a different aspect of the problem.

[jpmaia]

I had a similar problem with deleting things from lists....

The solution is to loop through the list in REVERSE order, otherwise you will always miss deleting items later in the list.

[Carl Manaster]

Hi, jpmaia,

I had a similar problem with deleting things from lists....

The solution is to loop through the list in REVERSE order, otherwise you will always miss deleting items later in the list.

That's a solution, and a good one. Others include:

as Richard suggested, decrementing the pointer inside the loop;

also as Richard suggested, copying the list and iterating over the copy;

collecting a set of items-to-be-deleted in one pass through the loop, then iterating over the set and deleting its elements from the array;

inserting the elements-to-be-kept into a new array, then replacing the array of interest with the new array.

I'm sure there are more.

Peace,
--Carl

[RichardWest]

also as Richard suggested, copying the list and iterating over the copy;

collecting a set of items-to-be-deleted in one pass through the loop, then iterating over the set and deleting its elements from the array;

The latter is an efficient implementation of the former.

The solution is to loop through the list in REVERSE order, otherwise you will always miss deleting items later in the list.

This solution also has problems in some circumstances. The problem is you are modifying the list you are iterating over and the iterator (an int) is not a safe iterator. It does not matter if you are doing this forward or in reverse. Athough reverse would probably work in this case assuming removeDataset does nothing fancy.

Note: I am using 'iterator' in the generic sense and not in the formal context of the Iterator class (or ::iterator in C++).

[tomkieffer]

I thank all of you for your prompt relies, because I already passed quite some time on this problem.

But there is no problem with the loop.
- The plot is created with an empty dataset (it needs one in the constructor) at first that I will never touch again. This makes for the 1 in the loop.
- No indexes are skipped. XYPlot counts null datasets just like it counts non-null datasets. So setting the dataset to null has no effects on getDatasetCount(). This is also in the API docs for YPlot getDataset(int i): returns the ith dataset, (possibly null). I added a getNotNullDatasetCount() method to my custom XYPlot to get this information. Besides, if you try to really remove elements (e.g. from a Vector) and go the wrong way round, it will trow an ArrayIndexOutOfBoundsException or something similar. In my code, I extend XYPlot and use only the methods already in the original XYPlot. I don't have control of any collection of datasets as it is a private field in XYPlot (why, actually?).
- I use a method pretty much similar (at least I think so) to the repaintPlot() method when adding or removing single datasets through the checkboxes, and this code isn't making any problems. Memory use increases proportionally to the amount of data added to the chart. I include the code at the end of this post.

By now I checked the application also with a watching agent. When adding datasets with the checkboxes I have a few 10.000 or 100.000 active double 'objects' running and the corresponding quantity of XYItems. When setting the datasets to null (removing them) all of the doubles are gone. There only remains a quantity of strings. I don't know yet why they are held in memory but they don't have a dangerous effect, cause on removing the datasets one by one by disabling the checkboxes, the memory use lowers to approximately the beginning value.
Now if I use the repaintPlot() method above with data that originally used up some 20m (4 datasets x 60.000 itms), memory use will grow to 200m or more and the OutOfMemory error is trown either at the series.add(x, y) statement or at the Object[] currentData = dataModel.getData() statement.

In short: while I can add and remove 10 or more datasets on a one by one basis wthout encountering problems, I can't refresh 3 of them in a loop without an OutOfMemory Error. This I can't understand.

[RichardWest]

The plot is created with an empty dataset (it needs one in the constructor) at first that I will never touch again. This makes for the 1 in the loop.

While it is true the constructor needs an XYDataset, you do not have to keep it indefinitely. You can always override the empty dataset with another populated dataset. If you add and remove populated datasets starting wtih dataset zero, you will save yourself some memory and allow you to make everything zero-based.

No indexes are skipped. XYPlot counts null datasets just like it counts non-null datasets. So setting the dataset to null has no effects on getDatasetCount(). This is also in the API docs for YPlot getDataset(int i): returns the ith dataset, (possibly null). I added a getNotNullDatasetCount() method to my custom XYPlot to get this information. Besides, if you try to really remove elements (e.g. from a Vector) and go the wrong way round, it will trow an ArrayIndexOutOfBoundsException or something similar.

Yes, getDatasetCount() returns the number of datasets including the null datasets. However, if you remove a dataset inside the loop, the number of datasets and the indexes for the datasets change. The number of datasets decreases by one, and the datasets that succeed the removed dataset will have their indexes reduced by one. For example, I have five datasets a, b, c, d, and e. I want to remove dataset c. Prior to removing dataset c, the following is true:

Code: Select all

getDatasetCount() returns 5
[a, b, c, d, e] are the five datasets with indexes [0, 1, 2, 3, 4]

Once I remove dataset c, the following is now true:

Code: Select all

getDatasetCount() returns 4
[a, b, d, e] are the five datasets with indexes [0, 1, 2, 3]

As you can see, datasets d and e are now indexes 2 and 3 respectively instead of 3 and 4. Your for-loop will skip dataset d since dataset d is now index 3 which you already checked to determine you should delete dataset c. You will not overrun the end of the array since you recompute the loop-ending condition each iteration.

In my code, I extend XYPlot and use only the methods already in the original XYPlot. I don't have control of any collection of datasets as it is a private field in XYPlot (why, actually?).

It is private because that forces you to use the accessor methods. These methods do checks to ensure there are no errors etc. I consider it good form to use accessor methods when extending another class since it allows the base class to be modified without breaking the derived class. As long as the API does not change, your derived class will still work as intended. This would not be the case if the dataset collection was protected or, for some unholy reason, public and you referenced it directly in your derived class.

I use a method pretty much similar (at least I think so) to the repaintPlot() method when adding or removing single datasets through the checkboxes, and this code isn't making any problems. Memory use increases proportionally to the amount of data added to the chart. I include the code at the end of this post.

Adding and removing a single dataset is easy and should not require logic anywhere near as complex as that you provided.

In short: while I can add and remove 10 or more datasets on a one by one basis wthout encountering problems, I can't refresh 3 of them in a loop without an OutOfMemory Error. This I can't understand.

It is hard to tell without seeing the complete code, but I suspect that adding and removing 10 datasets one-by-one is simpler logic than that of repaintPlot(). Complex logic tends to lead to bugs, and it seems that everyone who has responded to this post thinks the problem is in the for loop. However, none of us can run the code you provided, so we cannot know for sure and/or tell you what the problem is exactly.

[tomkieffer]

Here is a running code sketch of what I'm doing. Just check it out for the remove thing. In the original libraries add and remove are class methods of the plot and have some more sophisticated code, but this is the essence. And this is why I think there can't be a problem with the removing.
For the whole application, I'm not shure how to provide a running demo, because it's connected to a MySQL server and has lots of classes.

Code: Select all

/*
 * RemoverProblem.java
 *
 * Created on 10. April 2008, 11:25
 *
 * To change this template, choose Tools | Template Manager
 * and open the template in the editor.
 */

package bugfinder;

import org.jfree.chart.JFreeChart;
import org.jfree.chart.axis.NumberAxis;
import org.jfree.chart.plot.XYPlot;
import org.jfree.chart.renderer.xy.XYLineAndShapeRenderer;
import org.jfree.data.xy.AbstractXYDataset;
import org.jfree.data.xy.XYDataset;
import org.jfree.data.xy.XYSeriesCollection;

/**
 *
 * @author Kieffer
 */
public class RemoverProblem {
    
    JFreeChart chart;
    XYPlot plot;
    
    
    /** Creates a new instance of RemoverProblem */
    public RemoverProblem() {
        
        XYSeriesCollection ds1 = new XYSeriesCollection();
        XYSeriesCollection ds2 = new XYSeriesCollection();
        XYSeriesCollection ds3 = new XYSeriesCollection();
        XYSeriesCollection ds4 = new XYSeriesCollection();
        XYSeriesCollection ds5 = new XYSeriesCollection();
        XYSeriesCollection ds6 = new XYSeriesCollection();
        XYSeriesCollection ds7 = new XYSeriesCollection();
        XYSeriesCollection ds8 = new XYSeriesCollection();
        XYSeriesCollection ds9 = new XYSeriesCollection();
        XYSeriesCollection ds10 = new XYSeriesCollection();
        
        plot = new XYPlot(ds1, new NumberAxis(), new NumberAxis(), new XYLineAndShapeRenderer());
        
        addDataset(ds2);
        addDataset(ds3);
        addDataset(ds4);
        addDataset(ds5);
        
        System.out.println("\ndatasets after adding: ");
        
        for (int i = 0; i < plot.getDatasetCount(); i++) {
            System.out.println(plot.getDataset(i)!=null?plot.getDataset(i).toString():"null");
        }
        
        //the first dataset shall not be removed to avoid exceptions
        for (int i = 1; i < plot.getDatasetCount(); i++) {
            removeDataset(i);
        }
        
        System.out.println("datasetCount after removing: "+plot.getDatasetCount());
        System.out.println("\ndatasets after removing: ");
        
        for (int i = 0; i < plot.getDatasetCount(); i++) {
            System.out.println(plot.getDataset(i)!=null?plot.getDataset(i).toString():"null");
        }
        
        addDataset(ds6);
        addDataset(ds7);
        addDataset(ds8);
        addDataset(ds9);
        addDataset(ds10);
        
        System.out.println("\ndatasets after 2. adding: ");
        
        for (int i = 0; i < plot.getDatasetCount(); i++) {
            System.out.println(plot.getDataset(i)!=null?plot.getDataset(i).toString():"null");
        }
        
        for (int i = 1; i < plot.getDatasetCount(); i++) {
            removeDataset(i);
        }
        
        System.out.println("datasetCount after second removing: "+plot.getDatasetCount());
        System.out.println("\ndatasets after 2. removing: ");
        
        for (int i = 0; i < plot.getDatasetCount(); i++) {
            System.out.println(plot.getDataset(i)!=null?plot.getDataset(i).toString():"null");
        }
        
    }
    
    
     public static void main(String[] args) {
        RemoverProblem remover = new RemoverProblem();
    }
     
     private void removeDataset(int i) {
         plot.setDataset(i, null);
     }
     
     private void addDataset(AbstractXYDataset dataset) {
         
         boolean datasetFound = false;
         for (int i = 0; i < plot.getDatasetCount(); i++) {
             if (plot.getDataset(i) == null) {
                plot.setDataset(i, dataset);
                datasetFound = true;
                break;
             }
         }
         
         if ( !datasetFound) {
            plot.setDataset(plot.getDatasetCount(), dataset);
         }
     }
}

[tomkieffer]

Think I've found the error. I was forgetting changing seconds to minutes and actually created the huge data load.

But check the running demo prvided in my previous post anyway, just to see that there really is no problem with the remove code. I've thought for a long time that there should be some kind of removing method in the plot classes.

Sorry for bothering with this (in the end stupid) problem.