RFE: Stall allocation requests while heap is full and GC locker is held

As implicit in the synopsis above, suppose the heap is getting
nearly full and a GC will shortly be needed to satisfy any new
requests. Suppose that, at this juncture, one thread T1 in a multi-threaded
application acquires the gc locker in order to do a pretty short duration
operation. At this point, let us say T1 gets descheduled and
another thread T2 makes a java heap allocation request that cannot
be satisfied without a GC.

As currently implemented, we may bring the world to a safepoint,
and attempt a GC, realize that gc locker has locked out GC and
return NULL, signifying our failure to allocate the requested
storage. T2 will get an OOM which it can decide to catch and
deal with as appropriate, or it might not catch it and get
terminated.

A different implementation possibility is to hold the request of
thread T2 in abeyance (not returning the NULL) until such time
as we are able to do a garbage collection (i.e. when the gc locker
is vacated). Would that be a more friendly or useful behaviour
for applications?

Of course, in such a design we would need to consider the possibility
that a thread that holds the gc locker is itself making the allocation
request. That should be easy to track with just a little state
(i think already present) in the thread and in such cases we can and
would return a NULL because we would otherwise risk deadlock.

However, there might be more subtle deadlocks possible if T1's
operation has a circular dependency on T2's allocation via
a dependency chain not directly visible to the JVM. Clearly,
that would be a violation of the JNI spec by the programmer,
as to the restrictions on using critical lockers, and the
JVM could just shrug off responsibility for such user violations.
###@###.### 10/28/04 19:13 GMT