Implementation of JEP 371: Hidden Classes

Summary

Introduce hidden classes, which are classes that cannot be used directly by the bytecode of other classes. Hidden classes are intended for use by frameworks that generate classes at run time and use them indirectly, via reflection. A hidden class may be defined as a member of a nest for access control purposes, and may be unloaded while its defining loader is reachable.

Problem

The method sun.misc.Unsafe::defineAnonymousClass is a powerful internal API used by language implementions, such as for those for lambda expressions and Nashorn bindings. However, the VM-anonymous classes defined by this method are ill-defined and require special-case access checks. It is time to provide a standard mechanism to replace VM-anonymous classes.

The goals of JEP 371:

• Allow frameworks to define classes as non-discoverable implementation details of the framework, so that they cannot be linked against by other classes nor discovered through reflection.
• Support extending an access-control nest with non-discoverable classes.
• Support aggressive unloading of non-discoverable classes, so that frameworks have the flexibility to define as many as they need.
• Deprecate the non-standard API sun.misc.Unsafe::defineAnonymousClass, with the intent to deprecate it for removal in a future release.
• Do not change the Java programming language in any way.

Solution

This JEP proposes to extend the Lookup API to support defining a hidden class that can only be accessed by reflection. A hidden class is not discoverable by the JVM during bytecode linkage, nor by programs making explicit use of class loaders (via, e.g., Class::forName and ClassLoader::loadClass). Optionally, a hidden class can be created as a member of an access control nest, and a hidden class can have the same strong relationship with its defining loader as a normal class has with its own defining loader.

This CSR proposes to add a new method Lookup::defineHiddenClass for creating a hidden class. The API specification specifies the algorithm of the creation of a hidden class. The JVM itself never creates a hidden class, and hidden class creation has no impact on the JVM Specification.

Lookup::defineHiddenClass takes an option ClassOption::NESTMATE to specify if the newly created hidden class is added as a new member to an existing nest, and an option ClassOption::STRONG to specify if the newly created hidden class has the same strong relationship with its defining class loader as the normal class has with its own defining loader (this guarantees that the hidden class may be unloaded if and only if its defining class loader is reclaimed).

The dynamic nest membership enjoyed by hidden classes requires an update to the JVM Specification's rules for access control. Hidden classes cannot participate in the nest membership defined by NestHost and NestMembers attributes because they do not have binary names that can be expressed, in internal form, in those attributes. However, Lookup::defineHiddenClass can arrange for a hidden class to be a member of the same nest as the lookup class (Lookup::lookupClass) This means that, from the JVM's point of view, a class's nest membership may have been determined prior to access control; in that case, we wish for the JVM to use that membership during access control, and more specifically during the "nestmate test" (JVMS 5.4.4).

Separately, we wish to reduce the failure modes of access control in connection with nest membership. The nestmate test introduced in Java 11 is brittle because any failure to determine a nest host results in the JVM throwing a LinkageError other than an IllegalAccessError. This behavior is surprising, and inconsistent with how both Lookup::defineHiddenClass and Class::getNestHost behave when they experience problems trying to determine a nest host. Accordingly, if the JVM attempts to determine a nest host for a class, but various ancillary exceptions occur, then the JVM should swallow them, and determine that the class is its own nest host. In this way, and like the API, the JVM never fails to determine a nest host. The nestmate test may then succeed or fail in the normal way, based on the nests determined for the accessor and accessee.

Specification

Attached jep-371-specdiff-v3.zip and Draft-JVMS-HiddenClasses.pdf are the specdiff and JVMS change for (1) to (9) below. jep-371-specdiff-inc.zip for (10) to (15).

API javadoc/specdiff for reference: http://cr.openjdk.java.net/~mchung/valhalla/webrevs/hidden-classes/api/java.base/module-summary.html http://cr.openjdk.java.net/~mchung/valhalla/webrevs/hidden-classes/specdiff/

Summary of API changes:

1) java.lang.invoke.MethodHandles.Lookup::defineHiddenClass(byte[] bytes, boolean initialize, ClassOption... options)

This is the API to create a hidden class from the given bytes. The JVM never creates hidden classes itself. The API specification specifies the algorithm for the creation of a hidden class. Hidden class creation has no impact to the JVMS.

The support for a hidden class to be added to an existing nest has impact on JVMS section 5.4.4 "access control" and nest host determination (see below).

2) New enum class java.lang.invoke.MethodHandles.Lookup.ClassOption

NESTMATE and STRONG options allow a hidden class to optionally be a nestmate of the lookup class and have the same strong relationship with its defining loader as a normal class has with its own defining loader.

By default, a hidden class belongs to the nest of itself and it may be unloaded while its defining class loader is still alive.

3) New Class::isHidden method

4) Class::getName of a hidden class will return an invalid binary name containing / character (see JVMS change). Tools and applications that rely on the returned name being a valid binary name will need to be updated to handle hidden classes properly.

5) Class::getNestMembers is changed to not throw an exception if it fails to validate the nest membership of any member listed in the NestMembers attribute. Instead, Class::getNestMembers returns the nest host plus the members listed in the NestMembers attributes that were successfully resolved and determined to have the same nest host as this class. So it may return fewer members than listed in the NestMembers attribute.

6) java.lang.reflect.Field::set and other setX methods throw IllegalAccessError if the field is final and its declaring class is a hidden class.

7) Deprecate sun.misc.Unsafe::defineAnonymousClass API with removal=false.

+     *
+     * @deprecated Use the {@link java.lang.invoke.MethodHandles.Lookup#defineHiddenClass(byte[], boolean, MethodHandles.Lookup.ClassOption...)}
+     * method.
+     *
      * @param hostClass context for linkage, access control, protection domain, and class loader
      * @param data      bytes of a class file
      * @param cpPatches where non-null entries exist, they replace corresponding CP entries in data
      */
     @ForceInline
+    @Deprecated(since = "15", forRemoval = false)
     public Class<?> defineAnonymousClass(Class<?> hostClass, byte[] data, Object[] cpPatches) {
         return theInternalUnsafe.defineAnonymousClass(hostClass, data, cpPatches);
     }

8) JVM TI GetClassSignature returns a JNI-style signature and it returns a name for a hidden class containing ".".

9) JVMS

JDK-8231313 has the proposed JVMS 5.4.4 change to enhance access control for dynamically-determined nest membership. The draft JVMS change is cut-n-paste below. Attached Draft-JVMS-HiddenClasses.pdf is the better formatted version.

The second half of 5.4.4 is modified as follows, using * to indicate addition, and ~~~ to indication deletion.

If R is not accessible to D, then access control throws an IllegalAccessError.

~~~If R is public, protected, or has default access, then access control throws an IllegalAccessError.~~~ ~~~If R is private, then the nestmate test failed, and access control fails for the same reason.~~~~

Otherwise, access control succeeds.

A nest is a set of classes and interfaces that allow mutual access to their private members. One of the classes or interfaces is the nest host. It enumerates the classes and interfaces which belong to the nest, using the NestMembers attribute (§4.7.29). Each of them in turn designates it as the nest host, using the NestHost attribute (§4.7.28). A class or interface which lacks a NestHost attribute belongs to the nest hosted by itself; if it also lacks a NestMembers attribute, this nest is a singleton consisting only of the class or interface itself. The nest host for a given class or interface (that is, the nest to which the class or interface belongs) is determined by the Java Virtual Machine as part of access control, rather than when the class or interface is loaded. Certain methods of the Java SE Platform API may determine the nest host for a given class or interface prior to access control, in which case the Java Virtual Machine respects the prior determination during access control.

To determine whether a class or interface C belongs to the same nest as a class or interface D, the nestmate test is applied. C and D belong to the same nest if and only if the nestmate test succeeds. The nestmate test is as follows:

• If C and D are the same class or interface, then the nestmate test succeeds.

• Otherwise, the following steps are performed, in order:

  1. ~~~The nest host of D, H, is determined (below). If an exception is thrown, then the nestmate test fails for the same reason.~~~ Let H be the nest host of D, if the nest host of D has previously been determined. If the nest host of D has not previously been determined, then it is determined using the algorithm below, yielding H.

  2. ~~~The nest host of C, H', is determined (below). If an exception is thrown, then the nestmate test fails for the same reason.~~~ Let H' be the nest host of C, if the nest host of C has previously been determined. If the nest host of C has not previously been determined, then it is determined using the algorithm below, yielding H'.

  3. H and H' are compared. If H and H' are the same class or interface, then the nestmate test succeeds. Otherwise, the nestmate test fails. ~~~by throwing an IllegalAccessError.~~~

The nest host of a class or interface M is determined as follows:

• If M lacks a NestHost attribute, then M is its own nest host.

• Otherwise, M has a NestHost attribute, and its host_class_index item is used as an index into the run-time constant pool of M. The symbolic reference at that index is resolved to a class or interface H (§5.4.3.1). Then:

  ~~~During resolution of this symbolic reference, any of the exceptions pertaining to class or interface resolution can be thrown. Otherwise, resolution of H succeeds.~~~ If resolution of this symbolic reference fails, then M is its own nest host. Any exception thrown as a result of failure of class or interface resolution is not rethrown.

  ~~~If any of the following is true, an IncompatibleClassChangeError is thrown:~~~ Otherwise, if resolution succeeds but any of the following is true, then M is its own nest host:

  • H is not in the same run-time package as M.
  • H lacks a NestMembers attribute.
  • H has a NestMembers attribute, but there is no entry in its classes array that refers to a class or interface with the name N, where N is the name of M.

  Otherwise, H is the nest host of M.

Attached jep-371-specdiff-inc.zip shows the spec change for the following items:

This spec update is regarding the new form of a descriptor string for a hidden class or interface:

    "L" + N + "." + <suffix> + ";"

where N is the binary name encoded in the internal form of the bytes passed to Lookup::defineHiddenClass.

10) java.lang.invoke.TypeDescriptor and java.lang.invoke.TypeDescriptor.OfField and java.lang.invoke.TypeDescriptor.OfMethod spec are changed to specify that it can represent runtime entities that cannot be described nominally. TypeDescriptor::descriptorString returns a string that does not conform to JVMS 4.3 if this entity cannot be described nominally.

11) Class::descriptorString returns a string which is not a valid type descriptor if this Class object represents a hidden class or interface or represents an array class whose element type is a hidden class or interface. Such class or interface cannot be described nominally.

This is a behaviorlal incompatible change that may impact existing libraries that pass the resulting string to an API that requires it be a valid type descriptor. Existing code using the java.lang.constant API needs update to prepare hidden classes regardless of this change.

Class::describeConstable returns an empty optional for a Class object that cannot be described nominally.

12) Similarly java.lang.invoke.MethodType::descriptorString and MethodType::toMethodDescriptorString returns a string which is not a valid method descriptor if this method type cannot be described nominally. MethodType::describeConstable returns an empty optional.

13) com.sun.jdi.Type::signature and com.sun.jdi.event.ClassUnloadEvent::classSignature and JDWP corresponding commands are updated to return the string returned by Class::descriptorString and it includes . character which is illegal in a type descriptor.

Debugger tools need update to support hidden classes.

14) Type::name, com.sun.jdi.ReferenceType::name and com.sun.jdi.event.ClassUnloadEvent::className and JDWP corresponding commands are updated to return the string returned by Class::getName that may not be a binary name.

This has been a long standing behavior due to VM anonymous classes. This is not new to debuggers.

15) Spec clarification in JVM TI, JDWP, JDI and java.lang.instrument.Instrumentation in (a) APIs to return loaded classes include classes created by class loaders as well as Java SE APIs such as hidden classes (b) APIs to return classes who can be found by a class loader (aka initiating class loader) that do not include hidden classes.

The javadoc of the following APIs are updated and no behavior change:

- JVM TI `GetLoadedClasses`
- JVM TI  `GetClassLoaderReference`
- JVM TI `ClassLoad` event
- `java.lang.instrument.Instrumentation::getAllLoadedClasses`
- `java.lang.instrument.Instrumentation::getInitiatedClasses
- `com.sun.jdi.ClassLoaderReference::definedClasses`
- `com.sun.jdi.ClassLoaderReference::visibleClasses`