Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8346239

Improve memory efficiency of JimageDiffGenerator

XMLWordPrintable

    • b05
    • generic

        In certain JDK build configurations the binary files of the JDK can become quite large. One example is libjvm.so (the largest that I know of). If, for example the JDK is being built with --with-native-debug-symbols=internal in a fastdebug configuration, libjvm.so can become as large as 750MB on a x86_64 Linux system.

        In order to generate a JEP 493 enabled runtime, the JimageDiffGenerator comes into play which compares all files originating from JMOD source to the optimized versions that have been processed by the jlink plug-in pipeline. Each and every file is being compared byte-by-byte and the current implementation is too memory hungry since it reads in both copies (of a file) into memory and then compares them using Arrays.equals().

        This can cause jlink/runtimeImage test failures on some systems as witnessed below (and also in JDK-8344036). Note that for default JDK builds tests in runtimeImage folder generate a linkable runtime on-the-fly and then run the tests.

        On linux-aarch64 platform: $ make test TEST="test/jdk/tools/jlink/runtimeImage"

        Failing tests:
        TEST: tools/jlink/runtimeImage/AddOptionsTest.java
        TEST: tools/jlink/runtimeImage/BasicJlinkMissingJavaBase.java
        TEST: tools/jlink/runtimeImage/BasicJlinkTest.java
        TEST: tools/jlink/runtimeImage/CustomModuleJlinkTest.java
        TEST: tools/jlink/runtimeImage/GenerateJLIClassesTest.java
        TEST: tools/jlink/runtimeImage/KeepPackagedModulesFailTest.java
        TEST: tools/jlink/runtimeImage/JavaSEReproducibleTest.java
        TEST: tools/jlink/runtimeImage/ModifiedFilesExitTest.java
        TEST: tools/jlink/runtimeImage/ModifiedFilesWarningTest.java
        TEST: tools/jlink/runtimeImage/MultiHopTest.java
        TEST: tools/jlink/runtimeImage/PackagedModulesVsRuntimeImageLinkTest.java
        TEST: tools/jlink/runtimeImage/PatchedJDKModuleJlinkTest.java
        TEST: tools/jlink/runtimeImage/SystemModulesTest.java
        TEST: tools/jlink/runtimeImage/SystemModulesTest2.java

        On linux-aarch64 platform: $ make test TEST="tools/jlink/runtimeImage/AddOptionsTest.java"

        ```
        STDOUT:
        STDERR:
        jlink options: --output java-base-with-opts-jlink-tmp --add-modules jdk.jlink,java.base --generate-linkable-runtime --keep-packaged-modules=java-base-with-opts-jlink-tmp/jmods
        Error: Java heap space
        java.lang.OutOfMemoryError: Java heap space
                at java.base/java.io.InputStream.readNBytes(InputStream.java:447)
                at java.base/java.io.InputStream.readAllBytes(InputStream.java:348)
                at jdk.jlink/jdk.tools.jlink.plugin.ResourcePoolEntry.contentBytes(ResourcePoolEntry.java:127)
                at jdk.jlink/jdk.tools.jlink.internal.runtimelink.ResourcePoolReader.getResourceBytes(ResourcePoolReader.java:56)
                at jdk.jlink/jdk.tools.jlink.internal.runtimelink.JimageDiffGenerator.generateDiff(JimageDiffGenerator.java:91)
                at jdk.jlink/jdk.tools.jlink.internal.ImageFileCreator.addResourceDiffFiles(ImageFileCreator.java:333)
                at jdk.jlink/jdk.tools.jlink.internal.ImageFileCreator.generateJImage(ImageFileCreator.java:240)
                at jdk.jlink/jdk.tools.jlink.internal.ImageFileCreator.writeImage(ImageFileCreator.java:195)
                at jdk.jlink/jdk.tools.jlink.internal.ImageFileCreator.create(ImageFileCreator.java:126)
                at jdk.jlink/jdk.tools.jlink.internal.JlinkTask$ImageHelper.retrieve(JlinkTask.java:1015)
                at jdk.jlink/jdk.tools.jlink.internal.ImagePluginStack.operate(ImagePluginStack.java:194)
                at jdk.jlink/jdk.tools.jlink.internal.JlinkTask.createImage(JlinkTask.java:483)
                at jdk.jlink/jdk.tools.jlink.internal.JlinkTask.run(JlinkTask.java:293)
                at jdk.jlink/jdk.tools.jlink.internal.Main.run(Main.java:50)
                at jdk.jlink/jdk.tools.jlink.internal.Main$JlinkToolProvider.run(Main.java:67)
                at tests.JImageGenerator$JLinkTask.call(JImageGenerator.java:713)
                at AbstractLinkableRuntimeTest.createRuntimeLinkImage(AbstractLinkableRuntimeTest.java:274)
                at AbstractLinkableRuntimeTest.createJavaImageRuntimeLink(AbstractLinkableRuntimeTest.java:126)
                at AbstractLinkableRuntimeTest.createJavaImageRuntimeLink(AbstractLinkableRuntimeTest.java:121)
                at AddOptionsTest.runTest(AddOptionsTest.java:64)
                at AbstractLinkableRuntimeTest.run(AbstractLinkableRuntimeTest.java:62)
                at AddOptionsTest.main(AddOptionsTest.java:49)
                at java.base/java.lang.invoke.LambdaForm$DMH/0x000000f001002000.invokeStatic(LambdaForm$DMH)
                at java.base/java.lang.invoke.LambdaForm$MH/0x000000f001003400.invoke(LambdaForm$MH)
                at java.base/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
                at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:155)
                at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
                at java.base/java.lang.reflect.Method.invoke(Method.java:565)
                at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
                at java.base/java.lang.Thread.runWith(Thread.java:1460)
                at java.base/java.lang.Thread.run(Thread.java:1447)

        java.lang.AssertionError: Unexpected failure: 4
                at tests.Result.assertSuccess(Result.java:80)
                at AbstractLinkableRuntimeTest.createRuntimeLinkImage(AbstractLinkableRuntimeTest.java:274)
                at AbstractLinkableRuntimeTest.createJavaImageRuntimeLink(AbstractLinkableRuntimeTest.java:126)
                at AbstractLinkableRuntimeTest.createJavaImageRuntimeLink(AbstractLinkableRuntimeTest.java:121)
                at AddOptionsTest.runTest(AddOptionsTest.java:64)
                at AbstractLinkableRuntimeTest.run(AbstractLinkableRuntimeTest.java:62)
                at AddOptionsTest.main(AddOptionsTest.java:49)
                at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
                at java.base/java.lang.reflect.Method.invoke(Method.java:565)
                at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
                at java.base/java.lang.Thread.run(Thread.java:1447)

        JavaTest Message: Test threw exception: java.lang.AssertionError: Unexpected failure: 4
        JavaTest Message: shutting down test

        STATUS:Failed.`main' threw exception: java.lang.AssertionError: Unexpected failure: 4
        ```

              sgehwolf Severin Gehwolf
              fyang Fei Yang
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: