ADDITIONAL SYSTEM INFORMATION :
Tested on the latest OSX with Ubuntu Linux using
the latest builds from https://adoptopenjdk.net/upstream.html
8u222-b10 - no problem
11.0.4+11 - leak
jdk-12.0.2+10 - leak

A DESCRIPTION OF THE PROBLEM :
Using jdk 11 and 12 we experience a memory leak using nashorn (we know that is deprecated)

from the heap dump we can see the number of java.lang.invoke.LambdaForm and java.lang.invoke.BoundMethodHandle$Species_LL going up each iteration.

this does not happen by switching to jdk 8

REGRESSION : Last worked in version 8u221

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
create an instance of the nashorn script engine and invoke a java injected class method in a loop.
with a jdk 11 or 12 you can see that the memory keeps going up and from the heap dump you can see a lot of LambdaForm and BoundMethodHandle objects. while with jdk 8 this does not happen.

ACTUAL -
from the heap dump you can see the number of java.lang.invoke.LambdaForm and java.lang.invoke.BoundMethodHandle$Species_LL going up each iteration.

  Class Name | Objects | Shallow Heap
  -------------------------------------------------------------------------------------------------------
  java.lang.invoke.LambdaForm$Name | 4,857 | 155,424
  |- java.lang.Object[] | 4,375 | 124,784
  |- java.lang.invoke.LambdaForm$Name[] | 862 | 49,000
  | |- java.lang.invoke.LambdaForm | 857 | 41,136
  | | |- java.lang.invoke.BoundMethodHandle$Species_LL | 938 | 37,520
  | | | |- java.lang.invoke.BoundMethodHandle$Species_LL | 386 | 15,440
  | | | | |- java.lang.invoke.DirectMethodHandle | 314 | 10,048
  | | | | |- jdk.nashorn.internal.runtime.AccessorProperty | 165 | 9,240
  | | | | |- java.lang.invoke.LambdaForm | 40 | 1,920
  | | | | |- java.lang.Object[] | 41 | 1,880
  | | | | |- java.lang.invoke.MethodHandle[] | 28 | 896
  | | | | |- java.lang.invoke.BoundMethodHandle$Species_LLIL | 12 | 576
  | | | | |- java.lang.invoke.DirectMethodHandle$Accessor | 14 | 560
  | | | | |- jdk.nashorn.internal.runtime.Context$BuiltinSwitchPoint | 22 | 528
  | | | | |- java.lang.invoke.BoundMethodHandle$Species_LLI | 12 | 480
  | | | | |- java.lang.invoke.BoundMethodHandle$Species_LLLL | 10 | 480
  | | | | |- java.lang.invoke.SwitchPoint | 10 | 240
  | | | | |- java.lang.invoke.BoundMethodHandle$Species_LL | 4 | 160
  | | | | |- jdk.nashorn.internal.runtime.linker.InvokeByName | 4 | 96
  | | | | |- java.util.concurrent.ConcurrentHashMap$Node | 2 | 64
  | | | | |- java.lang.Class | 1 | 40
  | | | | |- jdk.nashorn.internal.runtime.ScriptFunctionData$GenericInvokers| 1 | 24
  | | | | '- Total: 16 entries | |
  | | | |- java.lang.invoke.DirectMethodHandle | 322 | 10,304
  | | | |- jdk.nashorn.internal.runtime.AccessorProperty | 168 | 9,408
  | | | |- java.lang.invoke.BoundMethodHandle$Species_LLL | 57 | 2,280
  -------------------------------------------------------------------------------------------------------


---------- BEGIN SOURCE ----------
package test;

import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;

public class TestJsMemLeak {
  private static final String JS_TEST_NAME = "test";
  private static final String JS_TEST_SOURCE =
      "for (var i = 0; i < 10; ++i) {\n" +
      " JsJavaUtil.testFunc();\n" +
      "}";

  public static final class JsJavaUtil {
    private long counter = 0;

    public long testFunc() {
      return counter++;
    }
  }

  public static void main(final String[] args) throws Exception {
    System.setProperty("nashorn.args", "--no-java -doe --language=es5");

    for (long i = 0; true; ++i) {
      final ScriptEngine scriptEngine = new ScriptEngineManager().getEngineByName("nashorn");
      scriptEngine.put("JsJavaUtil", new JsJavaUtil());
      scriptEngine.put("script", JS_TEST_SOURCE);
      scriptEngine.put("scriptName", JS_TEST_NAME);
      scriptEngine.eval("load({ script: script, name: scriptName })");
      System.gc();
      if (i % 100 == 0) {
        System.out.println(Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory());
      }
    }
  }
}

---------- END SOURCE ----------

FREQUENCY : always