Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8325363

Container support does not detect effective limits in cgroup hierarchies

XMLWordPrintable

    • x86_64
    • linux

      ADDITIONAL SYSTEM INFORMATION :
      All current releases of the JDK / Any Linux distribution with cgroup support, particularly when running with SystemD init

      A DESCRIPTION OF THE PROBLEM :
      Container support does not correctly detect effective limits in containers. It's a particular concern when using SystemD as the init system, because it creates a cgroup for every unit.

      It appears that the current implementation expects a cgroup under resource control to have the controller files available in the current cgroup (i.e. /proc/self/cgroup), but that's only the case if the subtree has been allowed control of the resources and a limit has been explicitly configured for that cgroup.

      This appeared to work with cgroup v1 because under SystemD the cgroup file points to the nearest cgroup with a limit configured.

      STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
      On any Linux distribution with SystemD and cgroup v2 support:

      $ sudo systemctl set-property user.slice MemoryMax=256M
      $ sudo systemctl set-property user.slice CPUQuota=20%

      $ cat /sys/fs/cgroup/user.slice/cpu.max
      20000 100000

      $ cat /sys/fs/cgroup/user.slice/memory.max
      268435456

      $ java -XshowSettings:system -Xlog:os+container=trace -version
      [0.000s][trace][os,container] OSContainer::init: Initializing Container Support
      [0.000s][debug][os,container] Detected optional pids controller entry in /proc/cgroups
      [0.000s][debug][os,container] Detected cgroups v2 unified hierarchy
      [0.000s][trace][os,container] Path to /cpu.max is /sys/fs/cgroup/user.slice/user-xxxxx.slice/session-xxxxx.scope/cpu.max
      [0.000s][debug][os,container] Open of file /sys/fs/cgroup/user.slice/user-xxxxx.slice/session-xxxxx.scope/cpu.max failed, No such file or directory
      [0.000s][trace][os,container] CPU Quota is: -2
      [0.000s][trace][os,container] Path to /cpu.max is /sys/fs/cgroup/user.slice/user-xxxxx.slice/session-xxxxx.scope/cpu.max
      [0.000s][debug][os,container] Open of file /sys/fs/cgroup/user.slice/user-xxxxx.slice/session-xxxxx.scope/cpu.max failed, No such file or directory
      [0.000s][trace][os,container] CPU Period is: -2
      [0.000s][trace][os,container] OSContainer::active_processor_count: 2
      [0.000s][trace][os,container] CgroupSubsystem::active_processor_count (cached): 2
      [0.000s][trace][os,container] total physical memory: 8107233280
      [0.000s][trace][os,container] Path to /memory.max is /sys/fs/cgroup/user.slice/user-xxxxx.slice/session-xxxxx.scope/memory.max
      [0.000s][trace][os,container] Raw value for memory limit is: max
      [0.000s][trace][os,container] Memory Limit is: Unlimited
      [0.000s][debug][os,container] container memory limit unlimited: -1, using host value 8107233280
      [0.001s][trace][os,container] CgroupSubsystem::active_processor_count (cached): 2
      [0.012s][trace][os,container] CgroupSubsystem::active_processor_count (cached): 2
      [0.049s][trace][os,container] total physical memory: 8107233280
      [0.050s][trace][os,container] Path to /memory.max is /sys/fs/cgroup/user.slice/user-xxxxx.slice/session-xxxxx.scope/memory.max
      [0.050s][trace][os,container] Raw value for memory limit is: max
      [0.050s][trace][os,container] Memory Limit is: Unlimited
      [0.050s][debug][os,container] container memory limit unlimited: -1, using host value 8107233280
      Operating System Metrics:
          Provider: cgroupv2
      [0.060s][trace][os,container] Path to /cpu.max is /sys/fs/cgroup/user.slice/user-xxxxx.slice/session-xxxxx.scope/cpu.max
      [0.060s][debug][os,container] Open of file /sys/fs/cgroup/user.slice/user-xxxxx.slice/session-xxxxx.scope/cpu.max failed, No such file or directory
      [0.060s][trace][os,container] CPU Quota is: -2
      [0.060s][trace][os,container] Path to /cpu.max is /sys/fs/cgroup/user.slice/user-xxxxx.slice/session-xxxxx.scope/cpu.max
      [0.060s][debug][os,container] Open of file /sys/fs/cgroup/user.slice/user-xxxxx.slice/session-xxxxx.scope/cpu.max failed, No such file or directory
      [0.060s][trace][os,container] CPU Period is: -2
      [0.060s][trace][os,container] OSContainer::active_processor_count: 2
          Effective CPU Count: 2
          CPU Period: -1
          CPU Quota: -1
          CPU Shares: -1
          List of Processors: N/A
          List of Effective Processors: N/A
          List of Memory Nodes: N/A
          List of Available Memory Nodes: N/A
          Memory Limit: Unlimited
          Memory Soft Limit: 0.00K
          Memory & Swap Limit: Unlimited
          Maximum Processes Limit: Unlimited

      openjdk version "21" 2023-09-19


      EXPECTED VERSUS ACTUAL BEHAVIOR :
      EXPECTED -
      The lowest limit from the current or any ancestor cgroup is used as the effective limit for that resource.
      ACTUAL -
      The configured limit for the current cgroup is used, which is usually the default maximum available for the system, causing ancestors, and most importantly the root cgroup limits to be ignored. Our container platform mounts in the cgroup limits into the container filesystem, to make them visible to the container.

      CUSTOMER SUBMITTED WORKAROUND :
      Disable container support and explicitly configure CPU and memory limits with flags.

      FREQUENCY : always

            spayne Sandra Payne
            spayne Sandra Payne
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: