The Klass objects are allocated into a class space list which is limited and fixed size. These pointers are allocated in chunks per class loader, which will have unallocated space at the end. These objects are variable sized because they include oopmaps and for the instance, and vtables for virtual calls, so a simple non-fragmenting allocation scheme can't be used.

The reason that this space is fixed size is because it's allocated with the Java heap so that the encoding and decoding compressed class can use the same compressed oops/klass pointer base and alignment. The size is also fixed because no matter where you allocate this space, reserving a new virtual space probably will not get an address range where the same encoding/decoding algorithm can be used. The code for the encoding/decoding algorithm is generated at startup time and is performance sensitive.

So this class space list is effectively the new PermGen. There's a ClassMetaspaceSize global but it's set to a byte size, which is only an estimate of number of classes that can be loaded before running out of class space. Running out of class metaspace gives an OOM "out of metadata space" and the VM cannot continue unless the application starts dropping references to class loaders (not easy to do). We've seen this error running nashorn tests. Resizing the class metaspace chunks for jsr292 anonymous classes stopped this error for nashorn so it's not a pri1 right now. It will be for someone though.

One of the other things that is odd about this class virtual space list in the code is that when it gets full it triggers a GC but the high water mark GC triggering is done for metaspace for data not classes. It's inconsistent in the code.

A fix that I've been testing out is what I call indirect compressed class pointers. The oops in Java heap point to an entry in an array of Klass* where Klass objects are allocated in the same metaspace as the other class data. The indirect class pointers are compressed into 32 bit value. The indirect class pointer array is still a fixed size but can allocate 20K which can have 20K/8 classes loaded. There is no wastage in this array since the indirect class pointers are all fixed size (ie pointers).

The indirection in encoding/decoding has a minimal performance cost < 1% in refworkload for solaris/sparc but none significant observed for linux/x64. Our testing ruled out BiasedLocking and GC times, so it appears that the generated code is incurring the extra cost.