-
Enhancement
-
Resolution: Fixed
-
P4
-
17, 20, 21
-
b20
Issue | Fix Version | Assignee | Priority | Status | Resolution | Resolved In Build |
---|---|---|---|---|---|---|
JDK-8308131 | 20u-cpu | Aleksey Shipilev | P4 | Resolved | Fixed | master |
JDK-8307361 | 20.0.2 | Aleksey Shipilev | P4 | Resolved | Fixed | b05 |
JDK-8316312 | 17.0.10-oracle | Calvin Cheung | P4 | Resolved | Fixed | b01 |
JDK-8307534 | 17.0.8 | Aleksey Shipilev | P4 | Resolved | Fixed | b02 |
Async monitor deflation is supposed to deal with this, by triggering the cleanup when `MonitorUsedDeflationThreshold` (`MUDT`) is reached. But the apparent problem with that heuristics is that `MUDT` is the percent of "ceiling", which is derived roughly as `max(#threads*AvgMonitorsPerThreadEstimate, #max_monitors_ever_observed)`, plus additive adjustments when deflation does not make progress. For the systems that run thousands of threads, we can have a very high ceiling.
Also, AFAIU, the ceiling can get arbitrarily high, if we had historical spike in the number of monitors, or we did some past async deflations without a progress. The ceiling seems to never go down! (Which is a good thing in buildings, but not in this heuristics code.) So even if we set `MUDT` to lowest value, 1, the ceiling might get so large eventually, the heuristics would never fire after some point.
Back-envelope calculation: even without involving the historical ceiling adjustments, just the static calculation for the system with 13K threads (real-life number) and default `AMPTE` = 1024, yields the ceiling of about 12M. Which means the default `MUDT` = 90 would not trigger cleanup until we have at least 11M monitors, which at ~200 bytes per monitor translates to >2 GB of native memory.
This started to be a problem in JDK 17, because the work done in JDK 15..16 (
I have a minimal example showing this behavior:
```
import java.util.concurrent.CountDownLatch;
public class Monitors {
static final int THREAD_COUNT = Integer.getInteger("threads", 2000);
static final int MONITOR_COUNT = Integer.getInteger("monitorsPerThread", 800);
static final CountDownLatch STARTED = new CountDownLatch(THREAD_COUNT);
static final CountDownLatch LOCKED = new CountDownLatch(THREAD_COUNT);
static final CountDownLatch HANG = new CountDownLatch(1);
public static void main(String... args) throws Exception {
System.out.println("Initializing");
for (int c = 0; c < THREAD_COUNT; c++) {
Thread t = new Thread(() -> {
try {
STARTED.countDown();
STARTED.await();
} catch (InterruptedException e) {}
for (int l = 0; l < MONITOR_COUNT; l++) {
try {
Object o = new Object();
synchronized (o) {
o.wait(1);
}
} catch (InterruptedException e) {}
}
try {
LOCKED.countDown();
HANG.await();
} catch (InterruptedException e) {}
});
t.start();
}
STARTED.await();
System.out.println("Started");
LOCKED.await();
System.out.println("Locked");
System.in.read();
HANG.countDown();
}
}
```
Run with:
```
$ java -XX:NativeMemoryTracking=summary -Xss256k Monitors.java
Initializing
Started
Locked
<in another terminal>
$ ps x -o pid,rss,command | grep java
67999 704656 .../java -XX:NativeMemoryTracking=summary -Xss256k Monitors.java
$ jcmd 67999 VM.native_memory
...
- Object Monitors (reserved=325001KB, committed=325001KB)
(malloc=325001KB #1600007)
```
So, out of 704M of RSS, 325M is taken by inflated object monitors, and there is 1.6M of them (2000 threads, 800 monitors each).
I see these ways out of this:
0. Ask users who know about this problem to drop their `MUDT` to much lower value, so that deflation would be triggered more often. This mitigates the issue, but this does not change the default behavior, which means other users are still exposed to this problem.
1. Drop `MUDT` to much lower default value, so that cleanups are more frequent. I think this is safe to do from latency perspective, because the deflation would still be performed asynchronously. The problem with the arbitrarily high ceiling is still present. This might also have throughput regressions since deflater thread would be more active in normal conditions.
2. Drop `AMPTE` to much lower default value, so that monitor population ceiling is not that large. This have the same implications as lowering `MUDT`.
3. Amend VM to request async deflation from safepoint, for example calling a light-weight version of `ObjectSynchronizer::request_deflate_idle_monitors` from safepoint cleanup path. This would be similar to the behavior of the JDK 11 -- piggybacking the cleanup requests on safepoint invocations, but with the benefit of being completely asynchronous.
4. Introduce the additional `GuaranteedAsyncDeflationInterval`, which would normally be larger than `AsyncDeflationInterval`, but which would trigger the deflation even when threshold is not reached. Some large default value, like 60s, should serve long-running systems well without incurring significant work on the deflater thread.
I like (4) quite a bit better, because it acts like a safety rail should the normal heuristics fail.
The threshold heuristics fixes can then proceed at leisurely pace, all the while being covered by this safety net.
- backported by
-
JDK-8307361 Guarantee eventual async monitor deflation
-
- Resolved
-
-
JDK-8307534 Guarantee eventual async monitor deflation
-
- Resolved
-
-
JDK-8308131 Guarantee eventual async monitor deflation
-
- Resolved
-
-
JDK-8316312 Guarantee eventual async monitor deflation
-
- Resolved
-
- duplicates
-
JDK-8264420 Allow MonitorUsedDeflationThreshold=0 for aggressive deflation of all eligible monitors
-
- Closed
-
- relates to
-
JDK-8291418 adjust monitor deflation logging and deflate_idle_monitors use
-
- Resolved
-
-
JDK-8306825 Monitor deflation might be accidentally disabled by zero intervals
-
- Resolved
-
-
JDK-8226416 MonitorUsedDeflationThreshold can cause repeated async deflation requests
-
- Resolved
-
-
JDK-8306774 Make runtime/Monitor/GuaranteedAsyncDeflationIntervalTest.java more reliable
-
- Resolved
-
-
JDK-8153224 Monitor deflation prolong safepoints
-
- Resolved
-
- links to
-
Commit openjdk/jdk17u-dev/547667df
-
Commit openjdk/jdk20u/5d2c2556
-
Commit openjdk/jdk/6b81342c
-
Review openjdk/jdk17u-dev/1316
-
Review openjdk/jdk20u/61
-
Review openjdk/jdk/13474