jdk.nashorn.internal.ir.debug.ObjectSizeCalculator.getEffectiveMemoryLayoutSpecification.getEffectiveMemoryLayoutSpecification() does not recognize OpenJDK's HotSpot VM, see below link for detailed code:

http://hg.openjdk.java.net/jdk9/jdk9/nashorn/file/e56cd9b5660f/src/jdk.scripting.nashorn/share/classes/jdk/nashorn/internal/ir/debug/ObjectSizeCalculator.java#l397

It was hard coded as:

    public static MemoryLayoutSpecification getEffectiveMemoryLayoutSpecification() {
        final String vmName = System.getProperty("java.vm.name");
        if (vmName == null || !vmName.startsWith("Java HotSpot(TM) ")) {
            throw new UnsupportedOperationException(
                    "ObjectSizeCalculator only supported on HotSpot VM");
        }

But for OpenJDK, value of "java.vm.name" property is "OpenJDK 64-Bit Server VM", not "Java HotSpot(TM)" (which is only valid for Oracle JDK).