My thread pool tests get StackOverflowError in sameVM mode.
It looks like this code in MainAction:

private void cleanup() {
cleanMode = true;
boolean someAlive = false;
Thread ta[] = new Thread[activeCount()];
enumerate(ta);
for (int i = 0; i < ta.length; i++) {
if (ta[i] != null &&
ta[i].isAlive() &&
ta[i] != Thread.currentThread() &&
!ta[i].isDaemon())
{
ta[i].interrupt();
someAlive = true;
//Thread.currentThread().yield();
}
}
if (someAlive) {
Thread.currentThread().yield();
cleanup();
}
} // cleanup()

will run out of stack very quickly even if the other threads are in fact
in the process of shutting down, especially if there are multiple CPUs.
Better would be to rewrite this code
to be an infloop rather than to recurse.

do { .... } while (someAlive)

Martin
More details: here is a jtr file fragment from running
test/java/util/concurrent/locks/Lock/CheckedLockLoops.java
on solaris-sparc:

#-----testresult-----
end=Wed Jan 24 22:54:31 PST 2007
execStatus=Error. Unexpected error caught from test java/util/concurrent/locks/Lock/CheckedLockLoops.java: java.lang.StackOverflowError
sections=script_messages Details

#section:script_messages
----------messages:(0/0)----------

#section:Details
----------messages:(0/0)----------
----------Stack trace:(1025/89016)----------
java.lang.StackOverflowError
at java.lang.Thread.isAlive(Native Method)
at java.lang.ThreadGroup.enumerate(ThreadGroup.java:408)
at java.lang.ThreadGroup.enumerate(ThreadGroup.java:357)
at com.sun.javatest.regtest.MainAction$SameVMThreadGroup.cleanup(MainAction.java:573)
at com.sun.javatest.regtest.MainAction$SameVMThreadGroup.cleanup(MainAction.java:587)
at com.sun.javatest.regtest.MainAction$SameVMThreadGroup.cleanup(MainAction.java:587)
....

Even if the test is hostile and has threads that never terminate, the proper
action is to time out and report this fact, not to die with StackOverflowError.