Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8370572

Cgroups hierarchical memory limit is not honored after JDK-8322420

XMLWordPrintable

      JDK-8322420 removed the handling of `hierarchical_memory_limit`:
      https://github.com/openjdk/jdk/commit/55a7cf14453b6cd1de91362927b2fa63cba400a1#diff-8910f554ed4a7bc465e01679328b3e9bd64ceaa6c85f00f0c575670e748ebba9L118-L131

      It seems to be the reason for reports that ECS tasks are no longer honoring the memory limit:
       https://github.com/adoptium/adoptium-support/issues/1293
       https://github.com/corretto/corretto-21/issues/135

      We initially found this in Corretto 21.0.9. I have deployed `public.ecr.aws/amazoncorretto/amazoncorretto:21` image to ECS, Fargate, 2GB task, and overridden Docker command to `java,-XX:InitialRAMPercentage=25,-XX:MaxRAMPercentage=25,-Xlog:os+container=trace,-Xlog:gc+init,-version`, and it showed me this:

      ```
      [0.002s][trace][os,container] total physical memory: 4037939200
      ...
      [0.002s][trace][os,container] Memory Limit is: 9223372036854771712
      [0.002s][debug][os,container] container memory limit ignored: 9223372036854771712, using host value 4037939200
      ...
      [0.013s][info ][gc,init ] Heap Initial Capacity: 964M
      [0.013s][info ][gc,init ] Heap Max Capacity: 964M
      ...
      openjdk version "21.0.9" 2025-10-21 LTS
      OpenJDK Runtime Environment Corretto-21.0.9.10.1 (build 21.0.9+10-LTS)
      OpenJDK 64-Bit Server VM Corretto-21.0.9.10.1 (build 21.0.9+10-LTS, mixed mode, sharing)
      ```

      So regardless this is 2G task, we detect 4G as the memory limit, which gives 25% heap of 1G, which leads to OOM later.

      In fact, this also reproduces with JDK mainline nightly, with a similar result:

      ```
      | [0.001s][trace][os,container] OSContainer::init: Initializing Container Support
      | [0.001s][debug][os,container] Detected optional pids controller entry in /proc/cgroups
      | [0.001s][debug][os,container] Detected cgroups hybrid or legacy hierarchy, using cgroups v1 controllers
      ...
      | [0.002s][trace][os,container] Path to /memory.limit_in_bytes is /sys/fs/cgroup/memory/memory.limit_in_bytes
      | [0.002s][trace][os,container] Memory Limit is: 9223372036854771712
      | [0.002s][debug][os,container] container memory limit ignored: 9223372036854771712, upper bound is 4037939200
      ...
      | [0.019s][info ][gc,init ] Heap Initial Capacity: 964M
      | [0.019s][info ][gc,init ] Heap Max Capacity: 964M
      ...
      | openjdk version "26-testing" 2026-03-17
      | OpenJDK Runtime Environment (build 26-testing-builds.shipilev.net-openjdk-jdk-b5938-20251024-1037)
      | OpenJDK 64-Bit Server VM (build 26-testing-builds.shipilev.net-openjdk-jdk-b5938-20251024-1037, mixed mode, sharing)
      ```

      If you ask how ECS configures cgroups, it would say this:

      ```
      /sys/fs/cgroup/memory/memory.use_hierarchy:1
      /sys/fs/cgroup/memory/memory.stat:hierarchical_memory_limit 2147483648
      ...
      /sys/fs/cgroup/memory/memory.limit_in_bytes:9223372036854771712
      ```

      So there is _no_ `memory.limit_in_bytes` set, but `hierarchical_memory_limit` is still there. New JDK code leans heavily on `memory.limit_in_bytes` and therefore sees no limit. Older JDK code used to look at `hierarchical_memory_limit` and worked fine.

      My brief reading suggests that walking the hierarchy and looking for `memory.limit_in_bytes` is fine for cgroups V2, but for cgroups V1 we should still rely on `hierarchical_memory_limit`. I have not been able to see clear docs on this.

            shade Aleksey Shipilev
            shade Aleksey Shipilev
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: