There was a bug reported against JDK8 on the mlvm-dev list.

The following Groovy program works with 7u40 but sometimes fails with JDK8:
---
   println "Started"
   for(int i = 0; i < 100; i++) {
       print "."
       for (int j=0;j<100000;j++) {
           Container c = new Container()
           c.run()
       }
   }
   println "\ndone"

public class Container implements Runnable {
 public Container() {}
 public void run() {
   GroovyShell gs = new GroovyShell()
   Script script = gs.parse("")
   script.run()
 }
}
---

I ran the test with both JDK7u40 and JDK8 and I see a steady increase of Permgen/Metaspace memory.

In JDK7u40 this continues until the Permgen is full. At that point a Full GC manages to clear out almost the entire Permgen.

With JDK8 the memory is just increasing indefinitely, unless we limit the metsapce by fore exmple -XX:MaxMetaspaceSize=86m. At that point we get an metaspace OOME instead.

My guess was that this is caused by SoftReferences and a JFR recording showed that to be plausible. The test added ~100000 SoftReferences during the 1 minute recording I took.

I also verified that adding:
Universe::heap()->collector_policy()->set_should_clear_all_soft_refs(true)
 and triggering a GC, before deciding to throw an OOME clears out the metaspace.