-
Bug
-
Resolution: Fixed
-
P4
-
24, 25
-
b05
-
generic
Issue | Fix Version | Assignee | Priority | Status | Resolution | Resolved In Build |
---|---|---|---|---|---|---|
JDK-8347619 | 24.0.1 | Severin Gehwolf | P4 | Resolved | Fixed | b02 |
In certain JDK build configurations the binary files of the JDK can become quite large. One example is libjvm.so (the largest that I know of). If, for example the JDK is being built with --with-native-debug-symbols=internal in a fastdebug configuration, libjvm.so can become as large as 750MB on a x86_64 Linux system.
In order to generate a JEP 493 enabled runtime, the JimageDiffGenerator comes into play which compares all files originating from JMOD source to the optimized versions that have been processed by the jlink plug-in pipeline. Each and every file is being compared byte-by-byte and the current implementation is too memory hungry since it reads in both copies (of a file) into memory and then compares them using Arrays.equals().
This can cause jlink/runtimeImage test failures on some systems as witnessed below (and also inJDK-8344036). Note that for default JDK builds tests in runtimeImage folder generate a linkable runtime on-the-fly and then run the tests.
On linux-aarch64 platform: $ make test TEST="test/jdk/tools/jlink/runtimeImage"
Failing tests:
TEST: tools/jlink/runtimeImage/AddOptionsTest.java
TEST: tools/jlink/runtimeImage/BasicJlinkMissingJavaBase.java
TEST: tools/jlink/runtimeImage/BasicJlinkTest.java
TEST: tools/jlink/runtimeImage/CustomModuleJlinkTest.java
TEST: tools/jlink/runtimeImage/GenerateJLIClassesTest.java
TEST: tools/jlink/runtimeImage/KeepPackagedModulesFailTest.java
TEST: tools/jlink/runtimeImage/JavaSEReproducibleTest.java
TEST: tools/jlink/runtimeImage/ModifiedFilesExitTest.java
TEST: tools/jlink/runtimeImage/ModifiedFilesWarningTest.java
TEST: tools/jlink/runtimeImage/MultiHopTest.java
TEST: tools/jlink/runtimeImage/PackagedModulesVsRuntimeImageLinkTest.java
TEST: tools/jlink/runtimeImage/PatchedJDKModuleJlinkTest.java
TEST: tools/jlink/runtimeImage/SystemModulesTest.java
TEST: tools/jlink/runtimeImage/SystemModulesTest2.java
On linux-aarch64 platform: $ make test TEST="tools/jlink/runtimeImage/AddOptionsTest.java"
```
STDOUT:
STDERR:
jlink options: --output java-base-with-opts-jlink-tmp --add-modules jdk.jlink,java.base --generate-linkable-runtime --keep-packaged-modules=java-base-with-opts-jlink-tmp/jmods
Error: Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.base/java.io.InputStream.readNBytes(InputStream.java:447)
at java.base/java.io.InputStream.readAllBytes(InputStream.java:348)
at jdk.jlink/jdk.tools.jlink.plugin.ResourcePoolEntry.contentBytes(ResourcePoolEntry.java:127)
at jdk.jlink/jdk.tools.jlink.internal.runtimelink.ResourcePoolReader.getResourceBytes(ResourcePoolReader.java:56)
at jdk.jlink/jdk.tools.jlink.internal.runtimelink.JimageDiffGenerator.generateDiff(JimageDiffGenerator.java:91)
at jdk.jlink/jdk.tools.jlink.internal.ImageFileCreator.addResourceDiffFiles(ImageFileCreator.java:333)
at jdk.jlink/jdk.tools.jlink.internal.ImageFileCreator.generateJImage(ImageFileCreator.java:240)
at jdk.jlink/jdk.tools.jlink.internal.ImageFileCreator.writeImage(ImageFileCreator.java:195)
at jdk.jlink/jdk.tools.jlink.internal.ImageFileCreator.create(ImageFileCreator.java:126)
at jdk.jlink/jdk.tools.jlink.internal.JlinkTask$ImageHelper.retrieve(JlinkTask.java:1015)
at jdk.jlink/jdk.tools.jlink.internal.ImagePluginStack.operate(ImagePluginStack.java:194)
at jdk.jlink/jdk.tools.jlink.internal.JlinkTask.createImage(JlinkTask.java:483)
at jdk.jlink/jdk.tools.jlink.internal.JlinkTask.run(JlinkTask.java:293)
at jdk.jlink/jdk.tools.jlink.internal.Main.run(Main.java:50)
at jdk.jlink/jdk.tools.jlink.internal.Main$JlinkToolProvider.run(Main.java:67)
at tests.JImageGenerator$JLinkTask.call(JImageGenerator.java:713)
at AbstractLinkableRuntimeTest.createRuntimeLinkImage(AbstractLinkableRuntimeTest.java:274)
at AbstractLinkableRuntimeTest.createJavaImageRuntimeLink(AbstractLinkableRuntimeTest.java:126)
at AbstractLinkableRuntimeTest.createJavaImageRuntimeLink(AbstractLinkableRuntimeTest.java:121)
at AddOptionsTest.runTest(AddOptionsTest.java:64)
at AbstractLinkableRuntimeTest.run(AbstractLinkableRuntimeTest.java:62)
at AddOptionsTest.main(AddOptionsTest.java:49)
at java.base/java.lang.invoke.LambdaForm$DMH/0x000000f001002000.invokeStatic(LambdaForm$DMH)
at java.base/java.lang.invoke.LambdaForm$MH/0x000000f001003400.invoke(LambdaForm$MH)
at java.base/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:155)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:565)
at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
at java.base/java.lang.Thread.runWith(Thread.java:1460)
at java.base/java.lang.Thread.run(Thread.java:1447)
java.lang.AssertionError: Unexpected failure: 4
at tests.Result.assertSuccess(Result.java:80)
at AbstractLinkableRuntimeTest.createRuntimeLinkImage(AbstractLinkableRuntimeTest.java:274)
at AbstractLinkableRuntimeTest.createJavaImageRuntimeLink(AbstractLinkableRuntimeTest.java:126)
at AbstractLinkableRuntimeTest.createJavaImageRuntimeLink(AbstractLinkableRuntimeTest.java:121)
at AddOptionsTest.runTest(AddOptionsTest.java:64)
at AbstractLinkableRuntimeTest.run(AbstractLinkableRuntimeTest.java:62)
at AddOptionsTest.main(AddOptionsTest.java:49)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:565)
at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
at java.base/java.lang.Thread.run(Thread.java:1447)
JavaTest Message: Test threw exception: java.lang.AssertionError: Unexpected failure: 4
JavaTest Message: shutting down test
STATUS:Failed.`main' threw exception: java.lang.AssertionError: Unexpected failure: 4
```
In order to generate a JEP 493 enabled runtime, the JimageDiffGenerator comes into play which compares all files originating from JMOD source to the optimized versions that have been processed by the jlink plug-in pipeline. Each and every file is being compared byte-by-byte and the current implementation is too memory hungry since it reads in both copies (of a file) into memory and then compares them using Arrays.equals().
This can cause jlink/runtimeImage test failures on some systems as witnessed below (and also in
On linux-aarch64 platform: $ make test TEST="test/jdk/tools/jlink/runtimeImage"
Failing tests:
TEST: tools/jlink/runtimeImage/AddOptionsTest.java
TEST: tools/jlink/runtimeImage/BasicJlinkMissingJavaBase.java
TEST: tools/jlink/runtimeImage/BasicJlinkTest.java
TEST: tools/jlink/runtimeImage/CustomModuleJlinkTest.java
TEST: tools/jlink/runtimeImage/GenerateJLIClassesTest.java
TEST: tools/jlink/runtimeImage/KeepPackagedModulesFailTest.java
TEST: tools/jlink/runtimeImage/JavaSEReproducibleTest.java
TEST: tools/jlink/runtimeImage/ModifiedFilesExitTest.java
TEST: tools/jlink/runtimeImage/ModifiedFilesWarningTest.java
TEST: tools/jlink/runtimeImage/MultiHopTest.java
TEST: tools/jlink/runtimeImage/PackagedModulesVsRuntimeImageLinkTest.java
TEST: tools/jlink/runtimeImage/PatchedJDKModuleJlinkTest.java
TEST: tools/jlink/runtimeImage/SystemModulesTest.java
TEST: tools/jlink/runtimeImage/SystemModulesTest2.java
On linux-aarch64 platform: $ make test TEST="tools/jlink/runtimeImage/AddOptionsTest.java"
```
STDOUT:
STDERR:
jlink options: --output java-base-with-opts-jlink-tmp --add-modules jdk.jlink,java.base --generate-linkable-runtime --keep-packaged-modules=java-base-with-opts-jlink-tmp/jmods
Error: Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.base/java.io.InputStream.readNBytes(InputStream.java:447)
at java.base/java.io.InputStream.readAllBytes(InputStream.java:348)
at jdk.jlink/jdk.tools.jlink.plugin.ResourcePoolEntry.contentBytes(ResourcePoolEntry.java:127)
at jdk.jlink/jdk.tools.jlink.internal.runtimelink.ResourcePoolReader.getResourceBytes(ResourcePoolReader.java:56)
at jdk.jlink/jdk.tools.jlink.internal.runtimelink.JimageDiffGenerator.generateDiff(JimageDiffGenerator.java:91)
at jdk.jlink/jdk.tools.jlink.internal.ImageFileCreator.addResourceDiffFiles(ImageFileCreator.java:333)
at jdk.jlink/jdk.tools.jlink.internal.ImageFileCreator.generateJImage(ImageFileCreator.java:240)
at jdk.jlink/jdk.tools.jlink.internal.ImageFileCreator.writeImage(ImageFileCreator.java:195)
at jdk.jlink/jdk.tools.jlink.internal.ImageFileCreator.create(ImageFileCreator.java:126)
at jdk.jlink/jdk.tools.jlink.internal.JlinkTask$ImageHelper.retrieve(JlinkTask.java:1015)
at jdk.jlink/jdk.tools.jlink.internal.ImagePluginStack.operate(ImagePluginStack.java:194)
at jdk.jlink/jdk.tools.jlink.internal.JlinkTask.createImage(JlinkTask.java:483)
at jdk.jlink/jdk.tools.jlink.internal.JlinkTask.run(JlinkTask.java:293)
at jdk.jlink/jdk.tools.jlink.internal.Main.run(Main.java:50)
at jdk.jlink/jdk.tools.jlink.internal.Main$JlinkToolProvider.run(Main.java:67)
at tests.JImageGenerator$JLinkTask.call(JImageGenerator.java:713)
at AbstractLinkableRuntimeTest.createRuntimeLinkImage(AbstractLinkableRuntimeTest.java:274)
at AbstractLinkableRuntimeTest.createJavaImageRuntimeLink(AbstractLinkableRuntimeTest.java:126)
at AbstractLinkableRuntimeTest.createJavaImageRuntimeLink(AbstractLinkableRuntimeTest.java:121)
at AddOptionsTest.runTest(AddOptionsTest.java:64)
at AbstractLinkableRuntimeTest.run(AbstractLinkableRuntimeTest.java:62)
at AddOptionsTest.main(AddOptionsTest.java:49)
at java.base/java.lang.invoke.LambdaForm$DMH/0x000000f001002000.invokeStatic(LambdaForm$DMH)
at java.base/java.lang.invoke.LambdaForm$MH/0x000000f001003400.invoke(LambdaForm$MH)
at java.base/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:155)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:565)
at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
at java.base/java.lang.Thread.runWith(Thread.java:1460)
at java.base/java.lang.Thread.run(Thread.java:1447)
java.lang.AssertionError: Unexpected failure: 4
at tests.Result.assertSuccess(Result.java:80)
at AbstractLinkableRuntimeTest.createRuntimeLinkImage(AbstractLinkableRuntimeTest.java:274)
at AbstractLinkableRuntimeTest.createJavaImageRuntimeLink(AbstractLinkableRuntimeTest.java:126)
at AbstractLinkableRuntimeTest.createJavaImageRuntimeLink(AbstractLinkableRuntimeTest.java:121)
at AddOptionsTest.runTest(AddOptionsTest.java:64)
at AbstractLinkableRuntimeTest.run(AbstractLinkableRuntimeTest.java:62)
at AddOptionsTest.main(AddOptionsTest.java:49)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:565)
at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
at java.base/java.lang.Thread.run(Thread.java:1447)
JavaTest Message: Test threw exception: java.lang.AssertionError: Unexpected failure: 4
JavaTest Message: shutting down test
STATUS:Failed.`main' threw exception: java.lang.AssertionError: Unexpected failure: 4
```
- backported by
-
JDK-8347619 Improve memory efficiency of JimageDiffGenerator
- Resolved
- relates to
-
JDK-8344036 Tests tools/jlink/runtimeImage fail on AIX after JDK-8311302
- Resolved
-
JDK-8347334 JimageDiffGenerator code clean-ups
- Resolved
- links to
-
Commit(master) openjdk/jdk24u/0243349d
-
Commit(master) openjdk/jdk/f696d9c5
-
Review(master) openjdk/jdk24u/13
-
Review(master) openjdk/jdk/22835
(2 links to)