It seems that JDK-8304303 introduced a memory leak. When running some jvmti tests the VM allocates a lot of memory without freeing it.
I could track it down with NMT (using -XX:MallocLimit=internal:3g -XX:NativeMemoryTracking=detail) to be allocated in SharedRuntime::notify_jvmti_unmount/mount.

---
Details:

[0x00007ff830d473fb] JNIHandleBlock::allocate_handle+0x1ab
[0x00007ff831008859] SharedRuntime::notify_jvmti_unmount+0x29
[0x0000029f1fed4eac]
                             (malloc=2818841744 type=Internal #9523114)

[0x00007ff830d473fb] JNIHandleBlock::allocate_handle+0x1ab
[0x00007ff831008659] SharedRuntime::notify_jvmti_mount+0x29
[0x0000029f1fed4bac]
                             (malloc=403292896 type=Internal #1362476)

[0x00007ff8310e5725] ConcurrentHashTable<SymbolTableConfig,11>::internal_insert_get<SymbolTableLookup,`ConcurrentHashTable<SymbolTableConfig,11>::insert<SymbolTableLookup>'::`2'::NOP>+0xd5
[0x00007ff8310e668e] SymbolTable::do_add_if_needed+0x14e
[0x00007ff8310e7ff3] SymbolTable::new_symbols+0x73
[0x00007ff830af3146] ClassFileParser::parse_constant_pool_entries+0x996
                             (malloc=25286400 type=Symbol #189455)
...
----