FULL PRODUCT VERSION :
$ java -version
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
$ uname -a
Darwin scrappy.local 16.4.0 Darwin Kernel Version 16.4.0: Thu Dec 22 22:53:21 PST 2016; root:xnu-3789.41.3~3/RELEASE_X86_64 x86_64

C:\>VER

Microsoft Windows [Version 10.0.14393]


A DESCRIPTION OF THE PROBLEM :
This is an issue with a custom FileSystemView, where a static singleton isn't an option, running in a long-running client. Every time we instantiate a new CustomFileSystemView, the FileSystemView constructor adds another UIManager propertyChangeListener that will never be cleared:

    public FileSystemView() {
        final WeakReference<FileSystemView> weakReference = new WeakReference<FileSystemView>(this);

        UIManager.addPropertyChangeListener(new PropertyChangeListener() {
            public void propertyChange(PropertyChangeEvent evt) {
                FileSystemView fileSystemView = weakReference.get();

                if (fileSystemView == null) {
                    // FileSystemView was destroyed
                    UIManager.removePropertyChangeListener(this);
                } else {
                    if (evt.getPropertyName().equals("lookAndFeel")) {
                        fileSystemView.useSystemExtensionHiding =
                                UIManager.getDefaults().getBoolean("FileChooser.useSystemExtensionHiding");
                    }
                }
            }
        });
    }

This PropertyChangeListener consumes a small amount of memory that adds up over the long run, ultimately triggering an OOME. Because this is inherent in the FileSystemView constructor, it happens in all operating system environments.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
This will ultimately trigger an OOME with any -Xmx value; this small value is just to speed up the test:

$ javac fsv.java
$ java -Xmx1m fsv
...
2022
Used Memory:1026544
Free Memory:546320
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.beans.PropertyChangeSupport$PropertyChangeListenerMap.newArray(PropertyChangeSupport.java:519)
at java.beans.PropertyChangeSupport$PropertyChangeListenerMap.newArray(PropertyChangeSupport.java:506)
at java.beans.ChangeListenerMap.getListeners(ChangeListenerMap.java:186)
at java.beans.PropertyChangeSupport.getPropertyChangeListeners(PropertyChangeSupport.java:179)
at javax.swing.UIManager.getPropertyChangeListeners(UIManager.java:1213)
at fsv.main(fsv.java:22)

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Because we're instantiating and clearing the new CustomFileSystemView object in the loop, this should run forever without consuming extra memory.
ACTUAL -
Because the FileSystemView base class constructor creates a PropertyChangeListener that is never cleared, this will always run out of memory if run long enough.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import java.io.*;
import javax.swing.*;
import javax.swing.filechooser.*;

public class fsv
{
  public static void main(String[] args) throws Exception
  {
    Runtime runtime = Runtime.getRuntime();
    System.out.println("Total Memory:" + runtime.totalMemory());
    System.out.println("Max Memory:" + runtime.maxMemory());
    System.err.println(UIManager.getPropertyChangeListeners().length);
    while (true)
    {
      FileSystemView v;
      // v = FileSystemView.getFileSystemView(); // no accumulation
      // FileSystemView constructor accumlates PropertyChangeListeners
      v = new CustomFileSystemView();
      v = null;
      System.runFinalization();
      System.gc();
      System.err.println(UIManager.getPropertyChangeListeners().length);
      long usedMemory = runtime.totalMemory() - runtime.freeMemory();
      System.out.println("Used Memory:" + usedMemory);
      System.out.println("Free Memory:" + runtime.freeMemory());
    }
  }

  static class CustomFileSystemView extends FileSystemView
  {
    public File createNewFolder(File containingDir) throws IOException
    {
      return null;
    }
  }
}

---------- END SOURCE ----------