cgroups v2 support hierarchical groups. Limits from outer groups apply also for the inner groups. To calculate an effective limit for the current nested group one needs to calculate a minimum of all the limits in the current group and its parent groups up to the root one.

Detecting resource limits with cgroups v2
https://mail.openjdk.org/pipermail/container-discuss/2023-November/000001.html

Current Hotspot does not seem to support it:
$ cgcreate -g memory:foo/bar
$ echo $[100*1024*1024] >/sys/fs/cgroup/foo/memory.max
grep "" /sys/fs/cgroup{,/foo{,/bar}}/memory.max
grep: /sys/fs/cgroup/memory.max: No such file or directory
/sys/fs/cgroup/foo/memory.max:104857600
/sys/fs/cgroup/foo/bar/memory.max:max
$ cgexec -g memory:foo/bar java -Xlog:os+container=trace -version|&grep 'Memory Limit'
[0.001s][trace][os,container] Memory Limit is: -1
[0.001s][trace][os,container] Memory Limit is: Unlimited
$ cgdelete -r -g memory:foo

$ cgcreate -g memory:foo/bar
$ echo $[100*1024*1024] >/sys/fs/cgroup/foo/bar/memory.max
$ grep "" /sys/fs/cgroup{,/foo{,/bar}}/memory.max
grep: /sys/fs/cgroup/memory.max: No such file or directory
/sys/fs/cgroup/foo/memory.max:max
/sys/fs/cgroup/foo/bar/memory.max:104857600
$ cgexec -g memory:foo/bar java -Xlog:os+container=trace -version|&grep 'Memory Limit'
[0.001s][trace][os,container] Memory Limit is: 104857600
[0.001s][trace][os,container] Memory Limit is: 104857600
[0.023s][trace][os,container] Memory Limit is: 104857600
[0.023s][trace][os,container] Memory Limit is: 104857600
$ cgdelete -r -g memory:foo