Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8312523

Implementation of Foreign Function & Memory API

    XMLWordPrintable

Details

    • CSR
    • Resolution: Approved
    • P3
    • 22
    • core-libs
    • None
    • source, binary, behavioral
    • medium
    • Some APIs have been removed/renamed. Clients will have to switch to alternatives. For var/method handles derived from memory layouts, clients will have to provide the additional base offset coordinate.
    • Java API
    • SE

    Description

      Summary

      This CSR refers to the finalization of the Foreign Function & Memory (FFM) API that first started previewing in Java 19.

      Problem

      Feedback and careful examination of the FFM API revealed the following issues:

      1. The FFM API offers ways to easily translate Java strings into native strings. But, the produced native strings are currently limited to the UTF-8 encoding.
      2. The MemoryLayout::sequenceLayout(MemoryLayout) factory method, which produces a sequence layout with a maximum element size (dervied from Long.MAX_VALUE) is a pitfall. If a user simply forgets to specify the size of the sequence, issues can occur later down the line. Such as: failure to allocate such a sequnce, or failure to link a function using a struct layout containing one of these sequences.
      3. While jextract can be used to automatically derive the layouts of native types, clients that don't use jextract are left figuring out what the layout of a native type is on their own.
      4. There are cases where the VarHandles and MethodHandles derived from memory layouts fall short, because a certain case can not be fully represented using memory layouts, for instance because it involves an array whose size is not statically known.
      5. The MemorySegment::segmentOffset method is no longer needed. The same thing can be trivially done by taking the difference between the base address of 2 segments.
      6. Calls to SegmentAllocator::allocateArray can be ambiguous. If this method is called with a single long argument x, which is intended to allocate an array containing the single long value x, instead clients will get an array of x uninitialized longs.
      7. The documentation of variadic functions mentions prototype-less functions, which are not variadic according to the C spec. This spec is also not as clear as could be.
      8. The Linker API is currently optional, which makes it impossible for the JDK to use it itself to implement access to native functions.
      9. The FMM API is currently in preview. We want to move it out of preview.
      10. The name of the isTrivial linker option is not descriptive enough of what the linker option does.
      11. It is currently possible to combine the isTrivial linker option with the captureCallState linker option, which constrains the implementation of future features.
      12. The Arena::allocate method is currently a default method. But, it is an important method that implements of Arena should override.
      13. When using an unsupported access mode and accessing a mis-aligned address using a memory access var handle, we currently throw an IllegalArgumentException, instead of an UnsupportedOperationException.
      14. There are several SegmentAllocator::allocateFrom methods that allocate an initialize the allocated memory. For those methods, we can skip memory zeroing of the allocated memory, since we know we'll be overwriting the memory right away any ways. But, the current API only allows initializing the allocated memory from a primitive value or a Java array.
      15. Executable jar files are allowed to specify Add-Opens and Add-Exports in their manifests, which function equivalently to the --add-opens and --add-exports command line flags of the java launcher. However, there is no equivalent manifest attribute for --enable-native-access.

      Solution

      The proposed solutions to each of these issues, in order, are:

      1. Expand the set of supported encodings to all encodings found in the java.nio.charset.StandardCharsets class.
      2. Remove the MemoryLayout::sequenceLayout(MemoryLayout) factory method. Clients should instead use the MemoryLayout::sequenceLayout(long, MemoryLayout) factory method, and specify the element count that they want explicitly.
      3. A new API, Linker::canonicalLayouts(), is added. For native linkers, this API can be used to find the memory layouts of the most common primitive types of the C language.
      4. Change the var and method handles derived from memory layouts to accept an additional 'base offset' coordinate. When a handle is used, the value of this coordinate is added to the offset computed done by the handle. This effectively allows these handles to be composed with other offset computation code. This can help address cases where a particular memory access can not be fully represented using the memory layout and layout path APIs.
      5. Remove the MemorySegment::segmentOffset method.
      6. Rename the methods in SegmentAllocator that allocate and intialize a memory segment to 'allocateFrom'. This avoids the aforementioned ambiguity.
      7. Clarfiy the documentation of variadic functions, and drop the references to prototype-less functions.
      8. Make the linker a required API, but dropping the exception specification on the Linker::nativeLinker method stating that on unsupported platforms it throws an UnsupportedOperationException.
      9. Move the FFM API out of preview, by removing @PreviewFeature annotations, and updating @since tags in javadoc.
      10. Rename the Linker.Option.isTrivial API to 'critical'. The name name leans on the established meaning of 'critical' in the context of JNI. We have to keep the name somewhat vague to avoid making too many promises about what it does, as not every linker implementation is required to implement it.
      11. Disallow isTrivial from being combined with the Linker.Option.captureCallState option. This keeps the door open for future enhancements.
      12. Make the Arena::allocate method abstract.
      13. Update the implementation to throw an UnsupportedOperationException for unsupported access modes, even in the case of a mis-aligned access
      14. Add a new SegmentAllocator::allocateFrom overload that allows initializing the allocated memory from an arbitrary memory segment.
      15. Add an Enable-Native-Access jar manifest attribute that functions equivalently to the --enable-native-access command line flag of the java launcher.

      Specification

      A specdiff of the changes is available below:

      Note that particular item #4 has a wide spread impact on the javadoc of the API.

      . #15 does not reflect in any javadoc changes, so I'll describe the feature here:

      The value of the Enable-Native-Access manifest attribute is, at least for the time being, restricted to the 'ALL-UNNAMED', indicating that native access is enabled for unnamed modules. This is similar to how the Add-Opens/Add-Exports attributes only grant access to unnamed modules. When a value other than 'ALL-UNNAMED' is specified, the JVM will be abort during launch with an error message.

      References

      1. https://github.com/openjdk/panama-foreign/pull/836
      2. https://github.com/openjdk/panama-foreign/pull/838
      3. https://github.com/openjdk/panama-foreign/pull/839
      4. https://github.com/openjdk/panama-foreign/pull/840
      5. https://github.com/openjdk/panama-foreign/pull/841
      6. https://github.com/openjdk/panama-foreign/pull/845
      7. https://github.com/openjdk/panama-foreign/pull/846
      8. https://github.com/openjdk/panama-foreign/pull/850
      9. https://github.com/openjdk/panama-foreign/pull/853
      10. https://github.com/openjdk/panama-foreign/pull/859
      11. https://github.com/openjdk/panama-foreign/pull/856
      12. no patch in panama-foreign
      13. https://github.com/openjdk/panama-foreign/pull/876
      14. https://github.com/openjdk/panama-foreign/pull/878
      15. https://github.com/openjdk/panama-foreign/pull/843

      Attachments

        Issue Links

          Activity

            People

              jvernee Jorn Vernee
              jvernee Jorn Vernee
              Maurizio Cimadamore
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: