Shrinking of Metaspace high-water-mark causes incorrect OutOfMemoryErrors or back-to-back GCs

Before more memory is committed for the Metaspace we check that we don't go past MetaspaceGC::_capacity_until_GC, which acts as a high-water-mark for the Metaspace. If we do, we trigger a Metadata GC before allowing the Metadata allocation.

Psuedo code for the Metadata allocation failure path:
if (HWM reached) {
  do Metadata GC (1)
  expand Metaspace and allocate (2)
  if (allocation failed) {
    release all soft-references and GC
    allocate
    if (allocation failed) {
      throw OOME
    }
  }
}

It's expected that the expansion of the Metaspace and the allocation of Metadata in (2) should succeed unless we can't commit more memory from the OS or the MaxMetaspaceSize has been reached.

However, there's a possibility that Metadata GC in (1) could lower the HWM although we actually need to increase it. This will break some assumptions in (2) and we fail to allocate Metadata.

The effect is that we get an OutOfMemoryError or back-to-back GCs.

ILW = HML => P2

Impact: High
 The JVM will most likely shutdown hang doing back-to-back GCS

Likelihood: Medium
 Not seen in our usual testing, bug can be reproduced. Applications with large, temporary spikes in Metaspace are more likely to hit this bug.

Workaround: Low
 Use -XX:MaxMetaspaceFreeRatio=100. This will turn off HWM shrinking.
 or, set a high MetaspaceSize, which will set the initial HWM value. Metadata GCs will be triggered at a later point.