assert((PENDING_EXCEPTION->is_a(SystemDictionary::OutOfMemoryError_klass())), "we expect only an OOM error here"); is an invalid assertion since pending_exception in Thread is handled specially for the VM thread and the compiler thread. It's filled in with Universe::vm_exception() which isn't actually an exception. It's failing with b4796926 on linux-amd64 very reliably. I've attached a copy of the test and you just run it like this:

java -server -Xcomp -XX:+ShowMessageBoxOnError -XX:+TraceExceptions -DHANGINGJAVA24521 -XX:-PrintVMOptions -XX:CompileThreshold=100 -XX:MaxPermSize=64M b4796926