Cgroups hierarchical memory limit is not honored after JDK-8322420

XMLWordPrintable

    • b23

        JDK-8322420 removed the handling of `hierarchical_memory_limit`:
        https://github.com/openjdk/jdk/commit/55a7cf14453b6cd1de91362927b2fa63cba400a1#diff-8910f554ed4a7bc465e01679328b3e9bd64ceaa6c85f00f0c575670e748ebba9L118-L131

        It seems to be the reason for reports that ECS tasks are no longer honoring the memory limit:
         https://github.com/adoptium/adoptium-support/issues/1293
         https://github.com/corretto/corretto-21/issues/135

        We initially found this in Corretto 21.0.9. I have deployed `public.ecr.aws/amazoncorretto/amazoncorretto:21` image to ECS, Fargate, 2GB task, and overridden Docker command to `java,-XX:InitialRAMPercentage=25,-XX:MaxRAMPercentage=25,-Xlog:os+container=trace,-Xlog:gc+init,-version`, and it showed me this:

        ```
        [0.002s][trace][os,container] total physical memory: 4037939200
        ...
        [0.002s][trace][os,container] Memory Limit is: 9223372036854771712
        [0.002s][debug][os,container] container memory limit ignored: 9223372036854771712, using host value 4037939200
        ...
        [0.013s][info ][gc,init ] Heap Initial Capacity: 964M
        [0.013s][info ][gc,init ] Heap Max Capacity: 964M
        ...
        openjdk version "21.0.9" 2025-10-21 LTS
        OpenJDK Runtime Environment Corretto-21.0.9.10.1 (build 21.0.9+10-LTS)
        OpenJDK 64-Bit Server VM Corretto-21.0.9.10.1 (build 21.0.9+10-LTS, mixed mode, sharing)
        ```

        So regardless this is 2G task, we detect 4G as the memory limit, which gives 25% heap of 1G, which leads to OOM later.

        In fact, this also reproduces with JDK mainline nightly, with a similar result:

        ```
        | [0.001s][trace][os,container] OSContainer::init: Initializing Container Support
        | [0.001s][debug][os,container] Detected optional pids controller entry in /proc/cgroups
        | [0.001s][debug][os,container] Detected cgroups hybrid or legacy hierarchy, using cgroups v1 controllers
        ...
        | [0.002s][trace][os,container] Path to /memory.limit_in_bytes is /sys/fs/cgroup/memory/memory.limit_in_bytes
        | [0.002s][trace][os,container] Memory Limit is: 9223372036854771712
        | [0.002s][debug][os,container] container memory limit ignored: 9223372036854771712, upper bound is 4037939200
        ...
        | [0.019s][info ][gc,init ] Heap Initial Capacity: 964M
        | [0.019s][info ][gc,init ] Heap Max Capacity: 964M
        ...
        | openjdk version "26-testing" 2026-03-17
        | OpenJDK Runtime Environment (build 26-testing-builds.shipilev.net-openjdk-jdk-b5938-20251024-1037)
        | OpenJDK 64-Bit Server VM (build 26-testing-builds.shipilev.net-openjdk-jdk-b5938-20251024-1037, mixed mode, sharing)
        ```

        If you ask how ECS configures cgroups, it would say this:

        ```
        /sys/fs/cgroup/memory/memory.use_hierarchy:1
        /sys/fs/cgroup/memory/memory.stat:hierarchical_memory_limit 2147483648
        ...
        /sys/fs/cgroup/memory/memory.limit_in_bytes:9223372036854771712
        ```

        So there is _no_ `memory.limit_in_bytes` set, but `hierarchical_memory_limit` is still there. New JDK code leans heavily on `memory.limit_in_bytes` and therefore sees no limit. Older JDK code used to look at `hierarchical_memory_limit` and worked fine.

        My brief reading suggests that walking the hierarchy and looking for `memory.limit_in_bytes` is fine for cgroups V2, but for cgroups V1 we should still rely on `hierarchical_memory_limit`. I have not been able to see clear docs on this.

              Assignee:
              Aleksey Shipilev
              Reporter:
              Aleksey Shipilev
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated:
                Resolved: