Failure to allocate a JNI handle (due to malloc failure) is presently reported using vm_exit_out_of_memory (when creating a new JNIHandleBlock in JNIHandles::allocate_block). That's not the behavior described in the JNI specification, which says that for out of memory failures

- NewGlobalRef will return NULL.

- NewLocalRef may return NULL.

- NewWeakGlobalRef will return NULL and throw OutOfMemoryError.

- PushLocalFrame will return a negative number and a pending OutOfMemoryError.

Note that OutOfMemoryError doesn't really seem like the right exception for running out of system memory, but that's what the JNI spec says.