FULL PRODUCT VERSION :
java version "1.7.0_40"
Java(TM) SE Runtime Environment (build 1.7.0_40-b43)
Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, mixed mode)



ADDITIONAL OS VERSION INFORMATION :
Windows 7 [Version 6.1.7601] Service Pack 1


EXTRA RELEVANT SYSTEM CONFIGURATION :
VM arguments:
-Dsun.java2d.d3d=false

A DESCRIPTION OF THE PROBLEM :
In case of change display events (like changing resolution of screen) Swing/AWT application leaks out of memory.

Heap dump showed that memory was consumed by int[] arrays from Win32GraphicConf objects which were accumulated in volatile images cache of javax.swing.RepaintManager.
RepaintManager add special listener (RepaintManager .DisplayChangedHandler) for display change events in static initializing block (so listener is added only once on class construction):

public class javax.swing.RepaintManager
static{
...
GraphicsEnvironment ge = GraphicsEnvironment.
                getLocalGraphicsEnvironment();
        if (ge instanceof SunGraphicsEnvironment) {
            ((SunGraphicsEnvironment)ge).addDisplayChangedListener(
                    new DisplayChangedHandler());
        }
...
}

But because of displayChangedListeners are kept in WeakHashMap:

public class sun.awt.SunDisplayChanger {
...
private Map listeners = Collections.synchronizedMap(new WeakHashMap(1));
...

and there is no strong reference for RepaintManager .DisplayChangedHandler ??? this handler is becoming eligible for garbage collector and removed at any time.
As this listener is responsible for clearing volatile images for previous GraphicConfiguration ??? application is constantly leaking memory each time display change event is happened.
As fix can suggest to hold strong reference for RepaintManager .DisplayChangedHandler.
              


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Compile and run the attached Java program from a console with -Dsun.java2d.d3d=false option

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Application should not leak memory

RepaintManager.volatileMap should contain as much entries as graphic devises on running machine
ACTUAL -
Application runs out of memory

ERROR MESSAGES/STACK TRACES THAT OCCUR :
F 0811-1506:11,764 java.lang.OutOfMemoryError: Java heap space [AWT-EventQueue-3]
F 0811-1506:11,764 at java.awt.image.DataBufferInt.<init>(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at java.awt.image.Raster.createPackedRaster(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at java.awt.image.DirectColorModel.createCompatibleWritableRaster(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at java.awt.GraphicsConfiguration.createCompatibleImage(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at java.awt.GraphicsConfiguration.createCompatibleImage(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at sun.awt.image.SunVolatileImage.getBackupImage(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at sun.awt.image.VolatileSurfaceManager.getBackupSurface(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at sun.awt.image.VolatileSurfaceManager.initialize(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at sun.awt.image.SunVolatileImage.<init>(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at sun.awt.image.SunVolatileImage.<init>(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at java.awt.GraphicsConfiguration.createCompatibleVolatileImage(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at java.awt.GraphicsConfiguration.createCompatibleVolatileImage(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at javax.swing.RepaintManager.getVolatileOffscreenBuffer(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at javax.swing.RepaintManager$PaintManager.paint(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at javax.swing.BufferStrategyPaintManager.paint(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at javax.swing.RepaintManager.paint(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at javax.swing.JComponent.paint(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at java.awt.GraphicsCallback$PaintCallback.run(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at sun.awt.SunGraphicsCallback.runOneComponent(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at sun.awt.SunGraphicsCallback.runComponents(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at java.awt.Container.paint(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at javax.swing.RepaintManager$3.run(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at javax.swing.RepaintManager$3.run(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at java.security.AccessController.doPrivileged(Native Method) [AWT-EventQueue-3]
F 0811-1506:11,764 at java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at javax.swing.RepaintManager.paintDirtyRegions(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at javax.swing.RepaintManager.paintDirtyRegions(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at javax.swing.RepaintManager.prePaintDirtyRegions(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at javax.swing.RepaintManager.access$1000(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at javax.swing.RepaintManager$ProcessingRunnable.run(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at java.awt.event.InvocationEvent.dispatch(Unknown Source) [AWT-EventQueue-3]
F 0811-1506:11,764 at java.awt.EventQueue.dispatchEventImpl(Unknown Source) [AWT-EventQueue-3]

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import java.awt.BorderLayout;
import java.awt.Dimension;
import javax.swing.JLabel;

import javax.swing.JFrame;

import sun.awt.windows.WToolkit;

public class RepaintManagerDisplayListenerWeakReferenceDefect
{
JFrame frame;

private void createAndShowGUI() {
        frame = new JFrame("FrameDemo");
        frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        JLabel emptyLabel = new JLabel("");
        emptyLabel.setPreferredSize(new Dimension(800,600));
        frame.getContentPane().add(emptyLabel, BorderLayout.CENTER);
        frame.pack();
        frame.setVisible(true);
    }

    public static void main(String[] args) {
    
    System.out.println("JVM D3D arguments : " + "noddraw-" + System.getProperty("swing.volatileImageBufferEnabled"));
    
    final RepaintManagerDisplayListenerWeakReferenceDefect defectCase = new RepaintManagerDisplayListenerWeakReferenceDefect();
    
        javax.swing.SwingUtilities.invokeLater(new Runnable() {
            public void run() {
            defectCase.createAndShowGUI();
            }
        });

while(true){
javax.swing.SwingUtilities.invokeLater(new Runnable() {
 public void run() {
 int width = defectCase.frame.getWidth();
 int height = defectCase.frame.getHeight();
 defectCase.frame.setSize(height, width); //create volatile offscreen image
 defectCase.frame.pack();
 WToolkit.displayChanged(); // initiate display change event
 }});
 
try{
Thread.sleep(600);
} catch (InterruptedException e){}
}
    }
}
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Use next VM option to disable volatile images cache/buffer

-Dswing.volatileImageBufferEnabled=false