This problem is hard to reproduce in a small test case, but I can explain what is happening:

I've got a TreeView that due to a layout change is no longer needed. There however still is a (weak) Bidirectional binding, which causes a call to TreeView#setRoot (showRoot is false) -- the TreeView itself however is no longer part of the Scene, and is about to be garbage collected (and in fact, other weak bindings that the Tree might use internally may have already been garbage collected).

What happens next is that the TreeView gets stuck in a loop creating new cells, leading to the OutOfMemoryError. I've attached the sources to my b92 JavaFX build, and found that the loop is occuring in VirtualFlow#addTrailingCells. Code:

<pre>
    private boolean addTrailingCells(boolean fillEmptyCells) {
        // If cells is empty then addLeadingCells bailed for some reason and
        // we're hosed, so just punt
        if (cells.isEmpty()) return false;
        
        // While we have not yet laid out so many cells that they would fall
        // off the flow, so we will continue to create and add cells. When the
        // offset becomes greater than the width/height of the flow, then we
        // know we cannot add any more cells.
        T startCell = cells.getLast();
        double offset = getCellPosition(startCell) + getCellLength(startCell);
        int index = startCell.getIndex() + 1;
        boolean filledWithNonEmpty = index <= cellCount;

        while (offset < viewportLength) {
            if (index >= cellCount) {
                if (offset < viewportLength) filledWithNonEmpty = false;
                if (! fillEmptyCells) return filledWithNonEmpty;
            }
            T cell = getAvailableCell(index);
            setCellIndex(cell, index);
            resizeCellSize(cell); // resize happens after config!
            cells.addLast(cell);

            // Position the cell and update the max pref
            positionCell(cell, offset);
            maxPrefBreadth = Math.max(maxPrefBreadth, getCellBreadth(cell));

            offset += getCellLength(cell);
            cell.setVisible(true);
            ++index;
        }
</pre>

It keeps executing the while loop until it goes out of memory.

Unfortunately, I cannot inspect any of the local variables in this loop (probably due to Eclipse and Java8 not quite playing nice with each other), but I can give you the fields:

viewPortLength = 821
cellCount = 618+ (keeps going up, this is just were I hit the breakpoint, note that the tree only contains like 20 items, and my screen is not that big either)
cells.firstIndex = 31
cells.lastIndex = 870
maxPrefBreadth = 0.0

Now... as to my suspicion what is happening: offset is 0 or atleast smaller than the viewPortLength and is not increasing because getCellLength keeps returning 0 (as this can probably not be determined anymore due to lack of a Scene). I checked as best I could (without local variables being visible) and in getCellLength, it will call getHeight (cell is not null, cells are not fixed size).

The "fix" for me is simple, unbind the Bidrectional in time to prevent this from occuring, but I don't think it should be possible to create this situation at all.

And here's the stack trace (in case it is useful) -- it differs slightly each time, but it always contains "com.sun.javafx.scene.control.skin.TreeViewSkin.createCell(TreeViewSkin.java:229)" and its parents.

java.lang.OutOfMemoryError: Java heap space
at com.sun.javafx.event.CompositeEventHandler.createEventHandlerRecord(CompositeEventHandler.java:116)
at com.sun.javafx.event.CompositeEventHandler.addEventHandler(CompositeEventHandler.java:48)
at com.sun.javafx.event.EventHandlerManager.addEventHandler(EventHandlerManager.java:71)
at javafx.scene.Node.addEventHandler(Node.java:7785)
at javafx.scene.control.Control.<init>(Control.java:410)
at javafx.scene.control.Labeled.<init>(Labeled.java:115)
at javafx.scene.control.Cell.<init>(Cell.java:282)
at javafx.scene.control.IndexedCell.<init>(IndexedCell.java:62)
at javafx.scene.control.TreeCell.<init>(TreeCell.java:81)
at hs.mediasystem.ext.selectmedia.tree.TreeListPane$MediaItemTreeCell.<init>(TreeListPane.java:284)
at hs.mediasystem.ext.selectmedia.tree.TreeListPane$MediaItemTreeCell.<init>(TreeListPane.java:284)
at hs.mediasystem.ext.selectmedia.tree.TreeListPane$10.call(TreeListPane.java:204)
at hs.mediasystem.ext.selectmedia.tree.TreeListPane$10.call(TreeListPane.java:201)
at com.sun.javafx.scene.control.skin.TreeViewSkin.createCell(TreeViewSkin.java:229)
at com.sun.javafx.scene.control.skin.TreeViewSkin$1.call(TreeViewSkin.java:60)
at com.sun.javafx.scene.control.skin.TreeViewSkin$1.call(TreeViewSkin.java:58)
at com.sun.javafx.scene.control.skin.VirtualFlow.getAvailableCell(VirtualFlow.java:1748)
at com.sun.javafx.scene.control.skin.VirtualFlow.addTrailingCells(VirtualFlow.java:1147)
at com.sun.javafx.scene.control.skin.VirtualFlow.layoutChildren(VirtualFlow.java:1039)
at com.sun.javafx.scene.control.skin.VirtualFlow.setCellCount(VirtualFlow.java:216)
at com.sun.javafx.scene.control.skin.TreeViewSkin.updateRowCount(TreeViewSkin.java:215)
at com.sun.javafx.scene.control.skin.TreeViewSkin.setRoot(TreeViewSkin.java:201)
at com.sun.javafx.scene.control.skin.TreeViewSkin.handleControlPropertyChanged(TreeViewSkin.java:131)
at com.sun.javafx.scene.control.skin.BehaviorSkinBase$2.call(BehaviorSkinBase.java:179)
at com.sun.javafx.scene.control.skin.BehaviorSkinBase$2.call(BehaviorSkinBase.java:177)
at com.sun.javafx.scene.control.MultiplePropertyChangeListenerHandler$1.changed(MultiplePropertyChangeListenerHandler.java:55)
at javafx.beans.value.WeakChangeListener.changed(WeakChangeListener.java:87)
at com.sun.javafx.binding.ExpressionHelper$Generic.fireValueChangedEvent(ExpressionHelper.java:347)
at com.sun.javafx.binding.ExpressionHelper.fireValueChangedEvent(ExpressionHelper.java:80)
at javafx.beans.property.ObjectPropertyBase.fireValueChangedEvent(ObjectPropertyBase.java:103)
at javafx.beans.property.ObjectPropertyBase.markInvalid(ObjectPropertyBase.java:110)
at javafx.beans.property.ObjectPropertyBase.set(ObjectPropertyBase.java:143)