Any thread that throws an unchecked exception followed by another thread starting a thread ( or invoking any operation on the group requiring its monitor ) will encounter a hang. When run in samevm or agentvm mode.

See discussion on the openjdk core-libs-dev mailing list:
  http://mail.openjdk.java.net/pipermail/core-libs-dev/2011-October/007885.html

Minimal jtreg test to reproduce this:

/*
 * @test
 */
import java.util.concurrent.Phaser;

public class HangJtreg {
    static Phaser phaser = new Phaser(2);

    public static void main(String[] args) throws Exception {
        (new Thread(new Runnable() {
            public void run() {
                try {
                    throw new RuntimeException("thrown from ThrowingThread");
                } finally {
                    phaser.arrive();
                }
            }
        })).start();

        phaser.arriveAndAwaitAdvance();
        (new Thread()).start();
    }
}