In fastdebug / slowdebug builds, should an error happen in a safepoint while JFR is recording, there is an assertion that does not allow the emergency operation to complete successfully. As a side effect, besides not dumping the JFR data, this can give misleading information in the hs_err.log file (due to recursive traps).

For example:

--------------- T H R E A D ---------------

Current thread (0x00007f4eb03d5800): VMThread "VM Thread" [stack: 0x00007f4e87422000,0x00007f4e87522000] [id=4219]

Stack: [0x00007f4e87422000,0x00007f4e87522000], sp=0x00007f4e87520110, free space=1016k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x185a46f] VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x25f
V [libjvm.so+0x185b28f] VMError::report_and_die(Thread*, void*, char const*, int, char const*, char const*, __va_list_tag*)+0x2f
V [libjvm.so+0xb50e30] report_vm_error(char const*, int, char const*, char const*, ...)+0x100
V [libjvm.so+0xf7ec4f] JfrTypeManager::write_type_set()+0x19f
V [libjvm.so+0xf46c79] JfrRecorderService::finalize_current_chunk_on_vm_error()+0x39
V [libjvm.so+0xf46dde] JfrRecorderService::rotate(int)+0x7e
V [libjvm.so+0xef8ef8] JfrEmergencyDump::on_vm_shutdown(bool)+0x278
V [libjvm.so+0x185a72e] VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x51e
V [libjvm.so+0x185b28f] VMError::report_and_die(Thread*, void*, char const*, int, char const*, char const*, __va_list_tag*)+0x2f
V [libjvm.so+0xb50e30] report_vm_error(char const*, int, char const*, char const*, ...)+0x100
V [libjvm.so+0x14b26e0] os::thread_cpu_time(Thread*, bool)+0x80
V [libjvm.so+0x178c506] Thread::print_on(outputStream*, bool) const+0x46
V [libjvm.so+0x17919de] JavaThread::print_on(outputStream*, bool) const+0x13e
V [libjvm.so+0x1795c09] Threads::print_on(outputStream*, bool, bool, bool, bool)+0x1c9
V [libjvm.so+0x189f042] VM_Operation::evaluate()+0x132
V [libjvm.so+0x189a6ce] VMThread::evaluate_operation(VM_Operation*) [clone .constprop.51]+0x18e
V [libjvm.so+0x189af87] VMThread::loop()+0x4d7
V [libjvm.so+0x189b593] VMThread::run()+0xd3
V [libjvm.so+0x14b3de0] thread_native_entry(Thread*)+0x100

The TypeSet write routine currently asserts a non-safepointed context, and this is what leads to this issue.

Fix can be quite simple, like:

diff -r 79baec7d831e src/hotspot/share/jfr/recorder/checkpoint/types/jfrTypeManager.cpp
--- a/src/hotspot/share/jfr/recorder/checkpoint/types/jfrTypeManager.cpp Tue Jul 03 10:47:50 2018 +0200
+++ b/src/hotspot/share/jfr/recorder/checkpoint/types/jfrTypeManager.cpp Tue Jul 03 12:02:18 2018 +0200
@@ -148,9 +148,8 @@
 }

 void JfrTypeManager::write_type_set() {
- assert(!SafepointSynchronize::is_at_safepoint(), "invariant");
   // can safepoint here because of Module_lock
- MutexLockerEx lock(Module_lock);
+ MutexLockerEx lock(SafepointSynchronize::is_at_safepoint() ? NULL : Module_lock);
   JfrCheckpointWriter writer(true, true, Thread::current());
   TypeSet set;
   set.serialize(writer);