-
Bug
-
Resolution: Unresolved
-
P4
-
9, 11, 17, 20, 21
-
generic
-
generic
ADDITIONAL SYSTEM INFORMATION :
openjdk 17.0.8 2023-07-18
OpenJDK Runtime Environment (build 17.0.8+7-Ubuntu-123.04)
A DESCRIPTION OF THE PROBLEM :
Module configuration resolution (specifically `Resolver#makeGraph`) is very slow for large numbers of automatic modules. I believe it scales in O(N^3) where N is the number of automatic modules. See below for crude numbers.
With the current readability graph resolution logic in `makeGraph`, each automatic module directly depends on all automatic modules, and all automatic modules transitively depend on all automatic modules. Hence, transitive readability propagation is very slow.
Assuming we have N automatic modules, the following should happen:
- For each automatic module M: (N entries)
- For each direct dependency D: (N-1 entries, all automatic modules that are not M)
- For each transitive dependency: (N-1 entries, all automatic modules that are not D)
- Check `m1Reads.contains(m3)`.
This gives O(N^3) operations for the graph computation.
A fix could be to handle automatic module resolution in a separate step after the propagation of transitive dependencies. If there is interest, I can work on providing a fix myself.
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run the attached code and look at the timings.
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Readability graph resolution for < 1000 modules should take a lot less than 1 second, certainly not over 90s on my machine.
ACTUAL -
I understand this is a very crude benchmark and that's not how you should write them in Java, but this gives a simple file to reproduce the issue. Here are my results:
Time for 100 automatic modules = 0.2s
Time for 200 automatic modules = 0.8s
Time for 300 automatic modules = 2.2s
Time for 400 automatic modules = 6.4s
Time for 500 automatic modules = 8.5s
Time for 600 automatic modules = 22.8s
Time for 700 automatic modules = 49.9s
Time for 800 automatic modules = 44.7s
Time for 900 automatic modules = 66.3s
Time for 1000 automatic modules = 95.1s
---------- BEGIN SOURCE ----------
public class BigLayerBenchmark {
public static void main(String... args) {
for (int modCount = 100; modCount <= 1000; modCount += 100) {
Map<String, ModuleReference> modules = new HashMap<>();
for (int i = 0; i < modCount; ++i) {
String name = "testmodule" + i;
var desc = ModuleDescriptor.newAutomaticModule(name).build();
modules.put(name, new ModuleReference(desc, null) {
@Override
public ModuleReader open() throws IOException {
throw new UnsupportedOperationException();
}
});
}
var references = new HashSet<>(modules.values());
var finder = new ModuleFinder() {
@Override
public Optional<ModuleReference> find(String name) {
return Optional.ofNullable(modules.get(name));
}
@Override
public Set<ModuleReference> findAll() {
return references;
}
};
long start = System.nanoTime();
ModuleLayer.boot().configuration().resolve(finder, ModuleFinder.ofSystem(), modules.keySet());
long end = System.nanoTime();
System.out.printf("Time for %4d automatic modules = %.1fs\n", modCount, (end - start) / 1e9);
}
}
}
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
My current workaround is to use deep reflection into java.lang.module to remove the automatic flag for module descriptors before resolution, add it back after resolution, and manually fix the readability graph. Something like this: https://gist.github.com/Technici4n/2a74e0e000fded940cfa79914123c812, but more testing is still required before I can deploy it.
openjdk 17.0.8 2023-07-18
OpenJDK Runtime Environment (build 17.0.8+7-Ubuntu-123.04)
A DESCRIPTION OF THE PROBLEM :
Module configuration resolution (specifically `Resolver#makeGraph`) is very slow for large numbers of automatic modules. I believe it scales in O(N^3) where N is the number of automatic modules. See below for crude numbers.
With the current readability graph resolution logic in `makeGraph`, each automatic module directly depends on all automatic modules, and all automatic modules transitively depend on all automatic modules. Hence, transitive readability propagation is very slow.
Assuming we have N automatic modules, the following should happen:
- For each automatic module M: (N entries)
- For each direct dependency D: (N-1 entries, all automatic modules that are not M)
- For each transitive dependency: (N-1 entries, all automatic modules that are not D)
- Check `m1Reads.contains(m3)`.
This gives O(N^3) operations for the graph computation.
A fix could be to handle automatic module resolution in a separate step after the propagation of transitive dependencies. If there is interest, I can work on providing a fix myself.
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run the attached code and look at the timings.
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Readability graph resolution for < 1000 modules should take a lot less than 1 second, certainly not over 90s on my machine.
ACTUAL -
I understand this is a very crude benchmark and that's not how you should write them in Java, but this gives a simple file to reproduce the issue. Here are my results:
Time for 100 automatic modules = 0.2s
Time for 200 automatic modules = 0.8s
Time for 300 automatic modules = 2.2s
Time for 400 automatic modules = 6.4s
Time for 500 automatic modules = 8.5s
Time for 600 automatic modules = 22.8s
Time for 700 automatic modules = 49.9s
Time for 800 automatic modules = 44.7s
Time for 900 automatic modules = 66.3s
Time for 1000 automatic modules = 95.1s
---------- BEGIN SOURCE ----------
public class BigLayerBenchmark {
public static void main(String... args) {
for (int modCount = 100; modCount <= 1000; modCount += 100) {
Map<String, ModuleReference> modules = new HashMap<>();
for (int i = 0; i < modCount; ++i) {
String name = "testmodule" + i;
var desc = ModuleDescriptor.newAutomaticModule(name).build();
modules.put(name, new ModuleReference(desc, null) {
@Override
public ModuleReader open() throws IOException {
throw new UnsupportedOperationException();
}
});
}
var references = new HashSet<>(modules.values());
var finder = new ModuleFinder() {
@Override
public Optional<ModuleReference> find(String name) {
return Optional.ofNullable(modules.get(name));
}
@Override
public Set<ModuleReference> findAll() {
return references;
}
};
long start = System.nanoTime();
ModuleLayer.boot().configuration().resolve(finder, ModuleFinder.ofSystem(), modules.keySet());
long end = System.nanoTime();
System.out.printf("Time for %4d automatic modules = %.1fs\n", modCount, (end - start) / 1e9);
}
}
}
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
My current workaround is to use deep reflection into java.lang.module to remove the automatic flag for module descriptors before resolution, add it back after resolution, and manually fix the readability graph. Something like this: https://gist.github.com/Technici4n/2a74e0e000fded940cfa79914123c812, but more testing is still required before I can deploy it.
- relates to
-
JDK-8320716 ResolvedModule::reads includes self when configuration contains two or more automatic modules
- Resolved
- links to
-
Review openjdk/jdk/15926