Implementation of Foreign Linker API (Incubator)

Summary

Enhance the incubator module, jdk.incubator.foreign (which is currently used by the Foreign Memory Access API), to add a new API, referred to as the Foreign Linker API, that is designed to facilitate direct and efficient access to foreign functions via the MethodHandle API. This API provides the fundamental building blocks to replace JNI.

Problem

To date, interacting with native libraries from Java can be a painful process. With the Java Native Interface (JNI), users have to declare native methods, then compile their classes which special flags, so that javac will also emit synthetic headers containing C entry points, which the user has then to define and compile (using a platform compiler such as gcc or clang) into a shared library; the shared library has then to be made available at runtime (e.g. via System::loadLibrary) so that the JVM will be able to link native method calls. Needless to say, this process is convoluted and error prone, and fails to scale when users have to provide Java bindings for entire native libraries (which might sometimes contain thousands of functions). Moreover, the presence of this intermediate JNI glue code makes projects relying on native methods harder to deploy (as the glue code will likely vary across platform) and to maintain (as such code will need to be updated every time the underlying native library is updated). For this reasons, Java developers cannot easily access high-quality native libraries, or have to resort to third-party frameworks in order to automate some of the steps associated with JNI.

Solution

The Foreign Linker API addresses the aforementioned problems by providing a way to link a native function, defined in some native library, directly, using Java code; the result of this linking operation will be a MethodHandle instance which, when called (e.g. using MethodHandle::invokeExact) will trigger the corresponding call to the native function. Since it is sometimes helpful to also pass Java code as data to foreign functions, the ForeignLinker API also provides a way to turn an existing Java MethodHandle instance into a MemorySegment which can then be passed (as a function pointer) to a native method handle call. The Foreign Linker API can thus allow Java programs to interact with native libraries without the need of any intervening glue code (rather, the glue code is generated dynamically, by the Foreign Linker runtime).

Specification

The implementation of the memory access API exports the following interfaces in the package jdk.incubator.foreign, defined in module jdk.incubator.foreign:

LibraryLookup               A lookup class which allows clients to load libraries and lookup symbols inside these libraries.
CLinker                     The main Foreign Linker API implementation.
FunctionDescriptor          An aggregate of `MemoryLayout`s describing the signature of the target foreign function.
NativeScope                 A helper class which allows clients to manage logically related off-heap memory allocations.

In traditional Java/JNI scenarios, this is done via the System::loadLibrary and System::load methods, which internally map into calls to, for instance, dlopen. The Foreign Linker API provides a simple library-lookup abstraction via the LibraryLookup class (similar to a method-handle lookup), which provides capabilities to look up named symbols in a given native library. We can obtain a library lookup in three different ways:

• LibraryLookup::ofDefault — returns the library lookup which can see all the symbols that have been loaded with the VM (useful to access symbols in the C standard library)
• LibraryLookup::ofPath — creates a library lookup associated with the library found at the given absolute path.
• LibraryLookup::ofLibrary — creates a library lookup associated with the library with given name (this might require setting the java.library.path variable appropriately).

The CLinker interface is the cornerstone of the Foreign Linker API. This abstraction plays a dual role. First, for downcalls (e.g. calls from Java to native code), the CLinker::downcallHandle method can be used to model native functions as plain MethodHandle objects. Second, for upcalls (e.g. calls from native back to Java code), the CLinker::upcallStub method can be used to convert an existing MethodHandle (which might point to some Java method) into a MemorySegment, which can then be passed to a native function as a function pointer. Both methods accept a FunctionDescriptor instance, which is an aggregate of memory layouts which is used to describe the signature of a foreign function in full. Moreover, the CLinker interface defines many layout constants, one for each main C primitive type. These layouts can be combined using a FunctionDescriptor to describe the signature of a C function; these layouts contain special classification attributes which are used by the Foreign Linker API runtime in order to correctly convert Java arguments into native arguments (and back). Finally, the CLinker API defines several helper functions which allow clients e.g. to convert a Java string into a NULL-terminated C string and back.

The NativeScope abstraction allows client to allocate multiple segments which share the same temporal bounds; that is, all segments allocated via a NativeScope instance will remain alive until the NativeScope itself is alive (in fact, NativeScope implements the AutoCloseable interface, and can be used in a try-with-resource statements). This is very useful when allocating and managing multiple, logically related, off-heap memory segments.

Note that many of the functionalities provided by the Foreign Linker API are fundamentally unsafe. That is, there is no way for the Foreign Linker runtime to verify that e.g. the signature of a native function in a FunctionDescriptor is correct, since the underlying shared library typically contains no type information (unless the library is compiled with debugging information). For this reason, many methods in this API are marked as restricted methods (this is a concept that has been introduced in the Foreign Memory Access API), and can only be invoked when a read-only JDK property, namely -Dforeign.restricted is set; this property can assume several values - the default value is deny, which will trigger a hard exception each time a restricted method is called. Developers can override this property value from the command line, to e.g. permit, which will allow calls to this method to succeed.

Note: this way of accessing restricted foreign functionalities through a runtime property is a pragmatic compromise, which will be replaced by a more robust mechanism (e.g. based on the module system) by the time the API exits the incubation stage.

Here are some useful links which should help in navigating through the changes in the API.

Javadoc:

http://cr.openjdk.java.net/~mcimadamore/8254231_v2/javadoc

Specdiff (delta, relative to JDK-8254163)

http://cr.openjdk.java.net/~mcimadamore/8254231_v3/specdiff_out

In addition, a specdiffs of the changes as of November 10th 2020 has been attached to this CSR.