OOMDebuggee tests wrongly assume that OOM exception place is determinated:

OOMDebugge.java
17 public void run() {
18 try {
19 while (true) {
20 list.add(new byte[50000]);
21 }
22 } catch (OutOfMemoryError err) {
23 Util.getLogger().info("Expected OOM in Debuggee");
24 list.removeFirst();
25 }
26
27 Utils.sleep();
28 }

Really it may be occured anywhere inside java process, e.g.:
Expected behavior

$ /opt/jdk1.7.0_06b14/bin/java -Dvmsqe.magic=061912175940 -d64 -server -Xmx10M -XX:-UseGCOverheadLimit -Dscenario=tmtools.jstack.share.debuggee.OOMDebuggee -cp /net/vmsqe.russia/export/testbase/sqe/vm/7/build/execution/vm/bin/classes:/export/jdk/jdk1.7.0_06//lib/tools.jar tmtools.share.debuggee.DebuggeeStarter tmtools.jstack.share.debuggee.StackDebuggee 10000
Expected OOM in Debuggee

Unexpected behavior

$ /opt/jdk1.7.0_06b14/bin/java -Dvmsqe.magic=061912175940 -d64 -server -XX:+UseG1GC -Xmx10M -XX:-UseGCOverheadLimit -Dscenario=tmtools.jstack.share.debuggee.OOMDebuggee -cp /net/vmsqe.russia/export/testbase/sqe/vm/7/build/execution/vm/bin/classes:/export/jdk/jdk1.7.0_06//lib/tools.jar tmtools.share.debuggee.DebuggeeStarter tmtools.jstack.share.debuggee.StackDebuggee 10000

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "scenario"
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
Test fail with message in server.log:

[2012-06-18T23:30:50.09] Exception in thread "main" nsk.share.TestBug: Fail result from client: Debuggee start script failed with exit code 1
[2012-06-18T23:30:50.09] at tmtools.share.distributed.LocalDebuggeeController.startDebuggee(LocalDebuggeeController.java:133)
[2012-06-18T23:30:50.09] at tmtools.share.Util.startDebuggee(Util.java:83)
[2012-06-18T23:30:50.09] at tmtools.share.Test.<init>(Test.java:84)
[2012-06-18T23:30:50.09] at tmtools.share.TestSuite.<init>(TestSuite.java:131)