ADDITIONAL SYSTEM INFORMATION :
OS: Linux xxxxxxxxxxxx 3.10.0-1062.20.1.el7.x86_64 #1 SMP Wed Feb 26 09:34:15 EST 2020 x86_64 x86_64 x86_64 GNU/Linux
Java: 15.0.1

A DESCRIPTION OF THE PROBLEM :
ZGC: Failed to truncate backing file (Permission denied) at startup after 14.0.2 to 15.0.1 upgrade

I found a possible regression in customer production environment linked to JDK-8245203 "ZGC: Don't track size in ZPhysicalMemoryBacking" or JDK15 "Fixed support for transparent huge pages".
After upgrade from jdk-14.0.2 to jdk-15.0.1, the JVM (using ZGC) is crashing at startup with the error message "Failed to truncate backing file (Permission denied)"
 
This error message was introduced in changelist http://hg.openjdk.java.net/jdk-updates/jdk15u/rev/556d5070c458
Cf source code : http://hg.openjdk.java.net/jdk-updates/jdk15u/file/556d5070c458/src/hotspot/os/linux/gc/z/zPhysicalMemoryBacking_linux.cpp
 
The issue was seen in Prod, not in Test, nor in labs, I suspect that Heap Backing Filesystem is not correctly set to tmpfs in Prod or a permission issue on a directory/file owned by root.

[2020-11-20T16:56:31.952+0530][debug][gc,heap] Minimum heap 96636764160 Initial heap 96636764160 Maximum heap 289910292480
[2020-11-20T16:56:31.953+0530][info ][gc,init] Initializing The Z Garbage Collector
[2020-11-20T16:56:31.953+0530][info ][gc,init] Version: 15.0.1+9-18 (release)
[2020-11-20T16:56:31.953+0530][info ][gc,init] NUMA Support: Disabled
[2020-11-20T16:56:31.953+0530][info ][gc,init] CPUs: 70 total, 45 available
[2020-11-20T16:56:31.953+0530][info ][gc,init] Memory: 307200M
[2020-11-20T16:56:31.953+0530][info ][gc,init] Large Page Support: Disabled
[2020-11-20T16:56:31.953+0530][info ][gc,init] Workers: 8 parallel, 8 concurrent
[2020-11-20T16:56:31.956+0530][info ][gc,init] Address Space Type: Contiguous/Unrestricted/Complete
[2020-11-20T16:56:31.956+0530][info ][gc,init] Address Space Size: 4423680M x 3 = 13271040M
[2020-11-20T16:56:31.956+0530][info ][gc,init] Heap Backing File: /memfd:java_heap
[2020-11-20T16:56:31.956+0530][error][gc ] Failed to truncate backing file (Permission denied)
[2020-11-20T16:56:31.992+0530][info ][gc,init] Runtime Workers: 8 parallel
[2020-11-20T16:56:31.994+0530][info ][gc ] Using The Z Garbage Collector

$ /opt/3rd_party/jdk_installed/jdk-15.0.1/bin/java -version
openjdk version "15.0.1" 2020-10-20
OpenJDK Runtime Environment (build 15.0.1+9-18)
OpenJDK 64-Bit Server VM (build 15.0.1+9-18, mixed mode, sharing)
 
$ /opt/3rd_party/jdk_installed/jdk-15.0.1/bin/java -XX:+UseZGC -version
[0.009s][error][gc] Failed to truncate backing file (Permission denied)
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
 
$ /opt/3rd_party/jdk_installed/jdk-14.0.2/bin/java
-XX:+UnlockExperimentalVMOptions -XX:+UseZGC -version
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-46)
OpenJDK 64-Bit Server VM (build 14.0.2+12-46, mixed mode)
 





-----------------------------------------------
[Additional details]
 
1. Not working in Prod with 15.0.1 :
openjdk version "15.0.1" 2020-10-20
OpenJDK Runtime Environment (build 15.0.1+9-18)
OpenJDK 64-Bit Server VM (build 15.0.1+9-18, mixed mode, sharing)
 
[2020-11-20T16:56:31.952+0530][debug][gc,heap] Minimum heap 96636764160 Initial heap 96636764160 Maximum heap 289910292480
[2020-11-20T16:56:31.953+0530][info ][gc,init] Initializing The Z Garbage Collector
[2020-11-20T16:56:31.953+0530][info ][gc,init] Version: 15.0.1+9-18 (release)
[2020-11-20T16:56:31.953+0530][info ][gc,init] NUMA Support: Disabled
[2020-11-20T16:56:31.953+0530][info ][gc,init] CPUs: 70 total, 45 available
[2020-11-20T16:56:31.953+0530][info ][gc,init] Memory: 307200M
[2020-11-20T16:56:31.953+0530][info ][gc,init] Large Page Support: Disabled
[2020-11-20T16:56:31.953+0530][info ][gc,init] Workers: 8 parallel, 8 concurrent
[2020-11-20T16:56:31.956+0530][info ][gc,init] Address Space Type: Contiguous/Unrestricted/Complete
[2020-11-20T16:56:31.956+0530][info ][gc,init] Address Space Size: 4423680M x 3 = 13271040M
[2020-11-20T16:56:31.956+0530][info ][gc,init] Heap Backing File: /memfd:java_heap
[2020-11-20T16:56:31.956+0530][error][gc ] Failed to truncate backing file (Permission denied)
[2020-11-20T16:56:31.992+0530][info ][gc,init] Runtime Workers: 8 parallel
[2020-11-20T16:56:31.994+0530][info ][gc ] Using The Z Garbage Collector
 
$ /opt/3rd_party/jdk_installed/jdk-15.0.1/bin/java -Xmx512m -XX:+PrintFlagsFinal -version | egrep "UseHugeTLBFS|UseLargePages|UseSHM|UseTransparentHugePages"
bool UseHugeTLBFS = false {product} {default}
bool UseLargePages = false {pd product} {default}
bool UseLargePagesInMetaspace = false {product} {default}
bool UseLargePagesIndividualAllocation = false {pd product} {default}
bool UseSHM = false {product} {default}
bool UseTransparentHugePages = false {product} {default}
 
 
2. Working fine in Prod with 14.0.2 :
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-46)
OpenJDK 64-Bit Server VM (build 14.0.2+12-46, mixed mode, sharing)
 
[2020-11-20T17:10:57.378+0530][debug][gc,heap] Minimum heap 96636764160 Initial heap 96636764160 Maximum heap 289910292480
[2020-11-20T17:10:57.378+0530][info ][gc,init] Initializing The Z Garbage Collector
[2020-11-20T17:10:57.378+0530][info ][gc,init] Version: 14.0.2+12-46 (release)
[2020-11-20T17:10:57.378+0530][info ][gc,init] NUMA Support: Disabled
[2020-11-20T17:10:57.378+0530][info ][gc,init] CPUs: 70 total, 45 available
[2020-11-20T17:10:57.378+0530][info ][gc,init] Memory: 307200M
[2020-11-20T17:10:57.378+0530][info ][gc,init] Large Page Support: Disabled
[2020-11-20T17:10:57.378+0530][info ][gc,init] Medium Page Size: 32M
[2020-11-20T17:10:57.378+0530][info ][gc,init] Workers: 8 parallel, 8 concurrent
[2020-11-20T17:10:57.381+0530][info ][gc,init] Address Space Type: Contiguous/Unrestricted/Complete
[2020-11-20T17:10:57.381+0530][info ][gc,init] Address Space Size: 4423680M x 3 = 13271040M
[2020-11-20T17:10:57.381+0530][info ][gc,init] Heap backed by file: /memfd:java_heap
[2020-11-20T17:10:57.381+0530][info ][gc,init] Min Capacity: 92160M
[2020-11-20T17:10:57.381+0530][info ][gc,init] Initial Capacity: 92160M
[2020-11-20T17:10:57.381+0530][info ][gc,init] Max Capacity: 276480M
[2020-11-20T17:10:57.381+0530][info ][gc,init] Max Reserve: 48M
[2020-11-20T17:10:57.381+0530][info ][gc,init] Pre-touch: Disabled
[2020-11-20T17:10:57.381+0530][info ][gc,init] Available space on backing filesystem: N/A
[2020-11-20T17:11:21.585+0530][info ][gc,init] Uncommit: Enabled, Delay: 300s
[2020-11-20T17:11:21.618+0530][info ][gc,init] Runtime Workers: 8 parallel
[2020-11-20T17:11:21.620+0530][info ][gc ] Using The Z Garbage Collector
 
Working fine in Lab with 15.0.1 :
$ /opt/3rd_party/jdk_installed/jdk-15.0.1/bin/java -Xms90g -Xmx270g -XX:-UseNUMA -XX:+UseZGC -XX:ParallelGCThreads=8 -Xlog:gc*,gc+heap=debug,gc+age=trace,safepoint:stdout:time,level,tags HelloWorld.java
[2020-11-20T14:16:08.635+0000][debug][gc,heap] Minimum heap 96636764160 Initial heap 96636764160 Maximum heap 289910292480
[2020-11-20T14:16:08.635+0000][info ][gc,init] Initializing The Z Garbage Collector
[2020-11-20T14:16:08.635+0000][info ][gc,init] Version: 15.0.1+9-18 (release)
[2020-11-20T14:16:08.635+0000][info ][gc,init] NUMA Support: Disabled
[2020-11-20T14:16:08.635+0000][info ][gc,init] CPUs: 64 total, 64 available
[2020-11-20T14:16:08.635+0000][info ][gc,init] Memory: 257775M
[2020-11-20T14:16:08.635+0000][info ][gc,init] Large Page Support: Disabled
[2020-11-20T14:16:08.635+0000][info ][gc,init] Workers: 8 parallel, 8 concurrent
[2020-11-20T14:16:08.636+0000][info ][gc,init] Address Space Type: Contiguous/Unrestricted/Complete
[2020-11-20T14:16:08.636+0000][info ][gc,init] Address Space Size: 4423680M x 3 = 13271040M
[2020-11-20T14:16:08.636+0000][info ][gc,init] Heap Backing File: /memfd:java_heap
[2020-11-20T14:16:08.637+0000][info ][gc,init] Heap Backing Filesystem: tmpfs (0x1021994)
[2020-11-20T14:16:08.637+0000][info ][gc,init] Min Capacity: 92160M
[2020-11-20T14:16:08.637+0000][info ][gc,init] Initial Capacity: 92160M
[2020-11-20T14:16:08.637+0000][info ][gc,init] Max Capacity: 276480M
[2020-11-20T14:16:08.637+0000][info ][gc,init] Max Reserve: 48M
[2020-11-20T14:16:08.637+0000][info ][gc,init] Medium Page Size: 32M
[2020-11-20T14:16:08.637+0000][info ][gc,init] Pre-touch: Disabled
[2020-11-20T14:16:08.637+0000][info ][gc,init] Available space on backing filesystem: N/A
[2020-11-20T14:16:08.637+0000][info ][gc,init] Uncommit: Enabled
[2020-11-20T14:16:08.637+0000][info ][gc,init] Uncommit Delay: 300s
[2020-11-20T14:16:24.652+0000][info ][gc,init] Runtime Workers: 8 parallel
[2020-11-20T14:16:24.653+0000][info ][gc ] Using The Z Garbage Collector

REGRESSION : Last worked in version 14

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
/opt/3rd_party/jdk_installed/jdk-15.0.1/bin/java -XX:+UseZGC -version

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
openjdk version "15.0.1" 2020-10-20
OpenJDK Runtime Environment (build 15.0.1+9-18)
OpenJDK 64-Bit Server VM (build 15.0.1+9-18, mixed mode, sharing)
 
ACTUAL -
[0.009s][error][gc] Failed to truncate backing file (Permission denied)
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.


FREQUENCY : always