Use the test case from RT-37501 and click around for a while opening and closing different nodes. In many cases after a while the application suddenly hangs (spinning wheel) then the memory usage races up to over 1GB (for a simple tree that at that time displays a few hundred tree cells containing a simple string) and I get an OutOfMemoryError in the log:

Exception in thread "JavaFX Application Thread" java.lang.OutOfMemoryError: Java heap space
at javafx.scene.control.MultipleSelectionModelBase.shiftSelection(MultipleSelectionModelBase.java:222)
at javafx.scene.control.TreeView$TreeViewBitSetSelectionModel$2.handle(TreeView.java:1262)
at javafx.scene.control.TreeView$TreeViewBitSetSelectionModel$2.handle(TreeView.java:1167)
at javafx.event.WeakEventHandler.handle(WeakEventHandler.java:79)
at com.sun.javafx.event.CompositeEventHandler$WeakEventHandlerRecord.handleBubblingEvent(CompositeEventHandler.java:248)
at com.sun.javafx.event.CompositeEventHandler.dispatchBubblingEvent(CompositeEventHandler.java:80)
at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:238)
at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:191)
at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:58)
at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
at com.sun.javafx.event.EventUtil.fireEventImpl(EventUtil.java:74)
at com.sun.javafx.event.EventUtil.fireEvent(EventUtil.java:49)
at javafx.event.Event.fireEvent(Event.java:204)
at javafx.scene.control.TreeItem.fireEvent(TreeItem.java:738)
at javafx.scene.control.TreeItem.updateChildren(TreeItem.java:903)
at javafx.scene.control.TreeItem.access$100(TreeItem.java:203)
at javafx.scene.control.TreeItem$2.onChanged(TreeItem.java:412)
at com.sun.javafx.collections.ListListenerHelper$SingleChange.fireValueChangedEvent(ListListenerHelper.java:158)
at com.sun.javafx.collections.ListListenerHelper.fireValueChangedEvent(ListListenerHelper.java:72)
at javafx.collections.ObservableListBase.fireChange(ObservableListBase.java:233)
at javafx.collections.ListChangeBuilder.commit(ListChangeBuilder.java:482)
at javafx.collections.ListChangeBuilder.endChange(ListChangeBuilder.java:541)
at javafx.collections.ObservableListBase.endChange(ObservableListBase.java:205)
at javafx.collections.ModifiableObservableListBase.add(ModifiableObservableListBase.java:155)
at java.util.AbstractList.add(AbstractList.java:108)
at jfxtest.TreeTest$NumberTreeItem.getChildren(TreeTest.java:63)
at javafx.scene.control.TreeItem.updateExpandedDescendentCount(TreeItem.java:886)
at javafx.scene.control.TreeItem.getExpandedDescendentCount(TreeItem.java:875)
at javafx.scene.control.TreeItem.updateExpandedDescendentCount(TreeItem.java:888)
at javafx.scene.control.TreeItem.getExpandedDescendentCount(TreeItem.java:875)
at javafx.scene.control.TreeItem.updateExpandedDescendentCount(TreeItem.java:888)
at javafx.scene.control.TreeItem.getExpandedDescendentCount(TreeItem.java:875)

...

The tree model in this case is infininite so it may be that something triggers a recursive traversal (which should not happen). On the other hand, I experienced the same behaviour when I more or less copied the sample from the TreeItem javadoc with a TreeView displaying the file system, which definitely is finite. No idea if this is large enough to explain the memory consumption but it may very well be.