Details
-
Bug
-
Resolution: Unresolved
-
P3
-
17.0.3
-
x86_64
-
linux
Description
ADDITIONAL SYSTEM INFORMATION :
Linux images-i-0fbf4c4bbd98a90e3 5.4.0-1080-aws #87~18.04.1-Ubuntu SMP Fri Jun 10 18:32:41 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
openjdk version "17.0.3" 2022-04-19 LTS
OpenJDK Runtime Environment Zulu17.34+20-SA (build 17.0.3+7-LTS)
OpenJDK 64-Bit Server VM Zulu17.34+20-SA (build 17.0.3+7-LTS, mixed mode, sharing)
A DESCRIPTION OF THE PROBLEM :
We've discovered in our JDK 8 to 17 upgrades that the switch to Unified Logging for GC logging has made GC logging far more susceptible to write latency. In environments with IOPS/throughput available on the disk containing logs, for instance we commonly use 'standard' EBS volumes for instance storage, write latency spikes now cause significant outliers during evacuation pauses.
The difference appears Unified Logging calling fflush for every event. The previous GC logging implementation benefited from buffering from fwrite (I believe the buffer is 8192 bytes in this case).
We'll configure -Xlog:async for our applications as we upgrade, however given how often this has been rediscovered by folks over the years, seems like Unified Logging should be buffered by default given it's behavior and the potential impact to JVM performance.
- https://engineering.linkedin.com/blog/2016/02/eliminating-large-jvm-gc-pauses-caused-by-background-io-traffic
- https://aws.amazon.com/blogs/developer/asynchronous-logging-corretto-17/
- https://mail.openjdk.org/pipermail/hotspot-dev/2019-August/039130.html
Some additional background here https://twitter.com/DannyThomas/status/1547367078864523264?s=20&t=EkYmLyXEGOmoRoVA_qrkvw.
REGRESSION : Last worked in version 8u333
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. Enable GC logging
2. Induce I/O latency by saturating the logging volume and note the impact to GC evaluation pause times
CUSTOMER SUBMITTED WORKAROUND :
Enable asynchronous logging via -Xlog:async.
FREQUENCY : often
Linux images-i-0fbf4c4bbd98a90e3 5.4.0-1080-aws #87~18.04.1-Ubuntu SMP Fri Jun 10 18:32:41 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
openjdk version "17.0.3" 2022-04-19 LTS
OpenJDK Runtime Environment Zulu17.34+20-SA (build 17.0.3+7-LTS)
OpenJDK 64-Bit Server VM Zulu17.34+20-SA (build 17.0.3+7-LTS, mixed mode, sharing)
A DESCRIPTION OF THE PROBLEM :
We've discovered in our JDK 8 to 17 upgrades that the switch to Unified Logging for GC logging has made GC logging far more susceptible to write latency. In environments with IOPS/throughput available on the disk containing logs, for instance we commonly use 'standard' EBS volumes for instance storage, write latency spikes now cause significant outliers during evacuation pauses.
The difference appears Unified Logging calling fflush for every event. The previous GC logging implementation benefited from buffering from fwrite (I believe the buffer is 8192 bytes in this case).
We'll configure -Xlog:async for our applications as we upgrade, however given how often this has been rediscovered by folks over the years, seems like Unified Logging should be buffered by default given it's behavior and the potential impact to JVM performance.
- https://engineering.linkedin.com/blog/2016/02/eliminating-large-jvm-gc-pauses-caused-by-background-io-traffic
- https://aws.amazon.com/blogs/developer/asynchronous-logging-corretto-17/
- https://mail.openjdk.org/pipermail/hotspot-dev/2019-August/039130.html
Some additional background here https://twitter.com/DannyThomas/status/1547367078864523264?s=20&t=EkYmLyXEGOmoRoVA_qrkvw.
REGRESSION : Last worked in version 8u333
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. Enable GC logging
2. Induce I/O latency by saturating the logging volume and note the impact to GC evaluation pause times
CUSTOMER SUBMITTED WORKAROUND :
Enable asynchronous logging via -Xlog:async.
FREQUENCY : often