Steve Byrne found and has tentatively fixed a bug in the handling of out-of-memory
situations. I want to report this to make sure we bring over the fix, and remember
to think about it a little more. The problem is that when out of memory, code in
gc.c may call runFinalization(), and currently does so with the heap locked (i.e.
HEAP_LOCKED() is true). Finalization code is arbitrary code, and if it causes
some other thread to try to allocate, it will deadlock with the finalizing thread.
The tentative fix is to unlock the heap around that call to runFinalization(). This
fix has been put into the HJ prebeta runtime. The thing to think about are what
the effects of this change are. It clearly causes a race between the finalizing
thread and some other thread that might deal with the out-of-memory situation
behind the back of the finalizing thread. I think that the code handling the out-
of-memory situation is resiliant enough that even if this race is lost, the system
will do very close to the right thing. But it ought to be thought about more.