By default, CMS does class unloading after a concurrent marking cycle. When the amount of metadata hits the high water mark (capacity_until_GC), a concurrent cycle is initiated, the high water mark is increased and the thread can proceed with the allocation. If CMSClassUnloadingEnabled is turned off we don't unload classes after a concurrent cycle, but instead relies on Full GCs to reclaim metadata memory. The current code skips triggering a concurrent cycle if CMSClassUnloadingEnabled is turned off, but it still allows the high water mark to be increased and the thread can proceed with the allocation without triggering a Full GC first. This defeats the entire purpose of the high water mark.

The suggested fix is to not allow the increase of the high water mark and the allocation if CMSClassUnloadingEnabled is turned off.

This is the suggested fix for JDK 8 and pre-G1 Class Unloading JDK 9 code:
diff -r 3976a725c192 src/share/vm/gc_implementation/shared/vmGCOperations.cpp
--- a/src/share/vm/gc_implementation/shared/vmGCOperations.cpp Mon Jul 07 10:18:33 2014 +0200
+++ b/src/share/vm/gc_implementation/shared/vmGCOperations.cpp Thu Jul 10 09:47:33 2014 +0200
@@ -212,10 +212,10 @@
     if (UseConcMarkSweepGC) {
       if (CMSClassUnloadingEnabled) {
         MetaspaceGC::set_should_concurrent_collect(true);
+ // For CMS expand since the collection is going to be concurrent.
+ _result =
+ _loader_data->metaspace_non_null()->expand_and_allocate(_size, _mdtype);
       }
- // For CMS expand since the collection is going to be concurrent.
- _result =
- _loader_data->metaspace_non_null()->expand_and_allocate(_size, _mdtype);
     }
     if (_result == NULL) {
       // Don't clear the soft refs yet.

This code was refactored in the G1 Class Unloading changes, but the problem still exists there.