http://sqeweb.sfbay.sun.com/nfs/results/vm/gtee/HSX/PIT/VM/hs20/b03/jdk7b118/fastdebug/vm/windows-i586/server/comp/windows-i586_vm__server_comp_vm.runtime.testlist/analysis.html

...
Canceller #583 successfully joined CancellableTimer #583 and checks out OK
[2010-11-07T18:15:12.29] Canceller #573 successfully joined CancellableTimer #573 and checks out OK
[2010-11-07T18:15:12.29] Canceller #590 successfully joined CancellableTimer #590 and checks out Exception in thread "Thread-1357" Exception in thread "Thread-1355" Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
[2010-11-07T18:15:12.53] at java.lang.Thread.start0(Native Method)
[2010-11-07T18:15:12.53] at java.lang.Thread.start(Thread.java:684)
[2010-11-07T18:15:12.53] at runtime.threads.CancellableThreadTest.Canceller.run(CancellableThreadTest.java:56)
[2010-11-07T18:15:12.53] at java.lang.Thread.run(Thread.java:717)
[2010-11-07T18:15:12.53] java.lang.OutOfMemoryError: unable to create new native thread
[2010-11-07T18:15:12.53] at java.lang.Thread.start0(Native Method)
[2010-11-07T18:15:12.53] at java.lang.Thread.start(Thread.java:684)
[2010-11-07T18:15:12.53] at runtime.threads.CancellableThreadTest.CancellableThreadTest.main(CancellableThreadTest.java:195)
[2010-11-07T18:15:12.53] java.lang.OutOfMemoryError: unable to create new native thread
[2010-11-07T18:15:12.53] at java.lang.Thread.start0(Native Method)
[2010-11-07T18:15:12.53] at java.lang.Thread.start(Thread.java:684)
[2010-11-07T18:15:12.53] at runtime.threads.CancellableThreadTest.Canceller.run(CancellableThreadTest.java:56)
[2010-11-07T18:15:12.53] at java.lang.Thread.run(Thread.java:717)
[2010-11-07T18:15:12.53] OK
[2010-11-07T18:15:12.53] Canceller #592 successfully joined CancellableTimer #592 and checks out OK
[2010-11-07T18:15:12.53] Canceller #586 successfully joined CancellableTimer #586 and checks out OK
[2010-11-07T18:15:12.53] Canceller #580 successfully joined CancellableTimer #580 and checks out OK
[2010-11-07T18:15:12.53] Canceller #587 successfully joined CancellableTimer #587 and checks out OK
[2010-11-07T18:15:12.53] VM option '+StartAttachListener'
[2010-11-07T18:15:12.53] # Test level exit status: 1

Note, that test failure occurs on fastdebug bits only. With product bits of b118, the test passes. Note also, that the test passes with b116 both product and fastdebug bits.

An interesting this, is that specifying some small maximum heap size explicitily, e.g., 16m also makes the test pass. Maybe default maximium heap size is set up errorneously too small in the fastdebug version on b118?
The maximum heap size -Xmx only affects the Java heap and this tests has run out of C heap. Thread stacks are allocated on the C heap. The thread stack size is 1 or two pages higher in fastdebug/debug mode, but hasn't been changed in a long time.
Of course, I can't reproduce this on my machine but I think I have a lot of memory. Are there memory quotas on these test machines? Was this only seen on windows?