We sometimes get an allocation failure from here:

https://github.com/openjdk/leyden/blob/74c0bfad65145a068ca6d9f51b38ae38a1fbc206/src/hotspot/share/oops/trainingData.cpp#L921

It's because Metaspace::allocate() sometimes needs to run GC before making the allocation, even though there's still available space. (I think the reason is to avoid fragmentation.)

However, we are allocating training data in many cases where a GC cannot be triggered. So trainingData.cpp calls the non-TRAPS version of Metaspace::allocate(), which will simply return nullptr if it thinks a GC is required.

Plus, sometimes we allocate TD while holding a lock. This means we cannot call the TRAPS version of Metaspace::allocate() -- only the TRAPS variant can trigger GC.

Work-around:

Instead of allocating from Metaspace, just allocate the TrainingData from C_HEAP, which doesn't care about GC or locks being held.

Even though the TrainingData types are logically a subclass of MetaspaceObj, there's no requirement by HotSpot that they must be allocated from Metaspace.