If you run an application like Hotjava on 1.2 beta3, you'll notice that
the Java heap keeps expanding until the system runs out of memory. It
does not matter how big you set -mx to be. Evantually you get an out
of memory.

There are memory leaks here and there. However, a more severe problem
is that the GC does not clear soft refs aggressively enough. Using
hprof, I discovered that Hotjava keeps tons of soft refs to images
around.

Looking at the gc.c code more carefully, it appears that the VM always
tries to expand the heap before setting the allocation state to YELLOW
or RED. Since setting the state to YELLOW or RED is the only way to
trigger soft refs to be cleared, this causes the VM to reach the
maximum heap size and run out of memory.

This needs to be fixed ASAP. I've experimented with exchanging the
ACT_DESPERATE and ACT_EXPAND, clearing the refs more aggresively, and
have seen a drastic reduction in the heap size.

We need to be at a point where the GC policy is not affected by
-mx setting at all.