-
Enhancement
-
Resolution: Fixed
-
P3
-
8, 9, 10
-
b31
-
linux
From TJ Fontaine:
Diagnostic commands (i.e. jcmd, jstack, etc) fail to attach to a target JVM that is inside a container (e.g. Docker).
A Linux container often isolates a process in a PID and Mount namespace that is separate from the "root container" (analogous to the hypervisor/dom0 in hardware virtualization environments, or the global zone on Solaris). A target JVM that is isolated in either a PID namespace, or a Mount namespace will fail the attach sequence.
When the target JVM is in its own PID namespace the pid of the process is distinct from what the real pid of the process as it relates to the root container. For example, in the root container you can observe a JVM with a pid of 17734, however if that JVM is running inside a Docker container the pid inside its PID namespace is likely 1. So when the target JVM receives the SIGQUIT it looks in /proc/self/cwd/ for .attach_pid1 however the external attaching JVM has created the file /proc/17734/cwd/.attach_pid17734. Given this discrepancy the target JVM will output to stderr thread status, since /proc/self/cwd/.attach_pid1 doesn't exist and won't continue with the attach sequence.
The solution is to parse /proc/pid/status for the field NSpid (available since Linux 4.1) which contains a list of pids, where the last entry is the "innermost" PID namespace value. (Namespaces can be stacked, unlike Solaris Zones which have a virtualization depth of 1)
The rest of the Linux attach sequence assumes a shared mount namespace by waiting for /tmp/.java_pid17734 to appear. But if the attaching process is in a separate namespace because the target JVM is in a mount namepsace (or in a chroot as well) the unix domain socket for attaching won't appear.
Instead the attach sequence should resolve file names relative to /proc/17734/root which has a materialized view of the rootfs for the target.
Diagnostic commands (i.e. jcmd, jstack, etc) fail to attach to a target JVM that is inside a container (e.g. Docker).
A Linux container often isolates a process in a PID and Mount namespace that is separate from the "root container" (analogous to the hypervisor/dom0 in hardware virtualization environments, or the global zone on Solaris). A target JVM that is isolated in either a PID namespace, or a Mount namespace will fail the attach sequence.
When the target JVM is in its own PID namespace the pid of the process is distinct from what the real pid of the process as it relates to the root container. For example, in the root container you can observe a JVM with a pid of 17734, however if that JVM is running inside a Docker container the pid inside its PID namespace is likely 1. So when the target JVM receives the SIGQUIT it looks in /proc/self/cwd/ for .attach_pid1 however the external attaching JVM has created the file /proc/17734/cwd/.attach_pid17734. Given this discrepancy the target JVM will output to stderr thread status, since /proc/self/cwd/.attach_pid1 doesn't exist and won't continue with the attach sequence.
The solution is to parse /proc/pid/status for the field NSpid (available since Linux 4.1) which contains a list of pids, where the last entry is the "innermost" PID namespace value. (Namespaces can be stacked, unlike Solaris Zones which have a virtualization depth of 1)
The rest of the Linux attach sequence assumes a shared mount namespace by waiting for /tmp/.java_pid17734 to appear. But if the attaching process is in a separate namespace because the target JVM is in a mount namepsace (or in a chroot as well) the unix domain socket for attaching won't appear.
Instead the attach sequence should resolve file names relative to /proc/17734/root which has a materialized view of the rootfs for the target.
- relates to
-
JDK-8193710 jcmd -l and jps commands do not list Java processes running in Docker containers
-
- Resolved
-
-
JDK-8255008 Serviceability tools don't fully support Containers
-
- Open
-
-
JDK-8226919 attach in linux hangs due to permission denied accessing /proc/pid/root
-
- Resolved
-
-
JDK-8228343 JCMD and attach fail to work across Linux Container boundary
-
- Resolved
-
-
JDK-8307977 jcmd and jstack broken for target processes running with elevated capabilities
-
- Closed
-
1.
|
docs need to describe special considerations for using the Attach API with docker |
|
Resolved | Savitha Satish | 2018-01-31 |
2.
|
[Doc Task] - Tools guide to be updated to accomodate Attach API changes |
|
Resolved | Clifford Wayne (Inactive) | 2018-02-09 |