FULL PRODUCT VERSION :
java version "1.6.0_01"
Java(TM) SE Runtime Environment (build 1.6.0_01-b06)
Java HotSpot(TM) Client VM (build 1.6.0_01-b06, mixed mode, sharing)

ADDITIONAL OS VERSION INFORMATION :
Microsoft Windows XP [Version 5.1.2600]

EXTRA RELEVANT SYSTEM CONFIGURATION :
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3

A DESCRIPTION OF THE PROBLEM :
The memory used by a JApplet is not garbage collected by the VM until *after* an OutOfMemoryError is thrown. The problem is most noticeable for applets that allocate a significant amount of memory, but appears to happen if an extremely large number of small applets are loaded.

However, if the applet extends directly from java.applet.Applet instead of from JApplet, the memory is reclaimed correctly and no error is thrown. The problem occurs in both FireFox and Internet Explorer (v6 and 7) but occurs much sooner in FireFox.

I suspect the problem may be limited to multiple CPU systems (including dual-core and hyperthreaded CPUs), but can't be certain.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Take a very simple applet that users a large amount of memory:
package com.ephox.tiny;
import javax.swing.*;


public class TinyApplet extends JApplet {

public byte[] big = new byte[Integer.MAX_VALUE / 100];

}

Load the applet in a simple page in FireFox:
<applet code="com.ephox.tiny.TinyApplet" codebase="." width="100" height="100"></applet>

Refresh the page a number of times. How many times you need to refresh will depend on the amount of memory available to Java - with default settings it generally generates an error after 10-20 reloads.

If however the applet extends from Applet instead of JApplet the page can be refreshed hundreds of times without seeing an OutOfMemoryError.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The applet should be garbage collected before an OutOfMemoryError is thrown and thus only one applet (or at worst 2 while the page is reloading) should be in memory and the available heap space should not be exhausted. Essentially, the JApplet version should work like the Applet version.
ACTUAL -
The JApplet is held in memory until after an OutOfMemoryError is thrown (it then appears to be garbage collected correctly).

ERROR MESSAGES/STACK TRACES THAT OCCUR :
Java Plug-in 1.6.0_01
Using JRE version 1.6.0_01 Java HotSpot(TM) Client VM
User home directory = C:\Users\Ephox
network: Loading user-defined proxy configuration ...
network: Done.
network: Loading proxy configuration from Netscape Navigator ...
network: Error reading registry file: C:\Users\Ephox\AppData\Roaming\Mozilla\registry.dat
network: Done.
network: Loading browser proxy configuration ...
network: Done.
network: Proxy Configuration: Browser Proxy Configuration


----------------------------------------------------
c: clear console window
f: finalize objects on finalization queue
g: garbage collect
h: display this help message
l: dump classloader list
m: print memory usage
o: trigger logging
p: reload proxy configuration
q: hide console
r: reload policy configuration
s: dump system and deployment properties
t: dump thread list
v: dump thread stack
x: clear classloader cache
0-5: set trace level to <n>
----------------------------------------------------

basic: Referencing classloader: sun.plugin.ClassLoaderInfo@134eb84, refcount=1
basic: Added progress listener: sun.plugin.util.GrayBoxPainter@13b5a3a
basic: Loading applet ...
basic: Initializing applet ...
basic: Starting applet ...
basic: completed perf rollup
network: Cache entry not found [url: http://speedy/workspace/tinyapplet/eclipse/com/ephox/tiny/TinyApplet.class, version: null]
network: Connecting http://speedy/workspace/tinyapplet/eclipse/com/ephox/tiny/TinyApplet.class with proxy=DIRECT
network: Downloading resource: http://speedy/workspace/tinyapplet/eclipse/com/ephox/tiny/TinyApplet.class
Content-Length: 423
Content-Encoding: null
network: Wrote URL http://speedy/workspace/tinyapplet/eclipse/com/ephox/tiny/TinyApplet.class to File C:\Users\Ephox\AppData\LocalLow\Sun\Java\Deployment\cache\6.0\15\7165c6cf-1fe578cf-temp
basic: Stopping applet ...
basic: Removed progress listener: sun.plugin.util.GrayBoxPainter@13b5a3a
basic: Finding information ...
basic: Releasing classloader: sun.plugin.ClassLoaderInfo@134eb84, refcount=0
basic: Caching classloader: sun.plugin.ClassLoaderInfo@134eb84
basic: Current classloader cache size: 1
basic: Done ...
basic: Joining applet thread ...
basic: Destroying applet ...
basic: Disposing applet ...
basic: Quiting applet ...
basic: Joined applet thread ...
basic: Referencing classloader: sun.plugin.ClassLoaderInfo@134eb84, refcount=1
basic: Added progress listener: sun.plugin.util.GrayBoxPainter@1c64ed8
basic: Loading applet ...
basic: Initializing applet ...
basic: Starting applet ...
basic: completed perf rollup
basic: Stopping applet ...
basic: Removed progress listener: sun.plugin.util.GrayBoxPainter@1c64ed8
basic: Finding information ...
basic: Releasing classloader: sun.plugin.ClassLoaderInfo@134eb84, refcount=0
basic: Caching classloader: sun.plugin.ClassLoaderInfo@134eb84
basic: Current classloader cache size: 1
basic: Done ...
basic: Joining applet thread ...
basic: Destroying applet ...
basic: Disposing applet ...
basic: Quiting applet ...
basic: Joined applet thread ...
basic: Referencing classloader: sun.plugin.ClassLoaderInfo@134eb84, refcount=1
basic: Added progress listener: sun.plugin.util.GrayBoxPainter@cdba53
basic: Loading applet ...
basic: Initializing applet ...
basic: Starting applet ...
basic: completed perf rollup
java.lang.OutOfMemoryError: Java heap space
at com.ephox.tiny.TinyApplet.<init>(TinyApplet.java:7)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at java.lang.Class.newInstance0(Unknown Source)
at java.lang.Class.newInstance(Unknown Source)
at sun.applet.AppletPanel.createApplet(Unknown Source)
at sun.plugin.AppletViewer.createApplet(Unknown Source)
at sun.applet.AppletPanel.runLoader(Unknown Source)
at sun.applet.AppletPanel.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
basic: Exception: java.lang.OutOfMemoryError: Java heap space


REPRODUCIBILITY :
This bug can be reproduced often.

---------- BEGIN SOURCE ----------
package com.ephox.tiny;
import javax.swing.*;


public class TinyApplet extends JApplet {

public byte[] big = new byte[Integer.MAX_VALUE / 100];

}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Extend from Applet instead of JApplet. The layered pane functionality of the applet is no longer available however.