With regard to virtual memory, both the memory operations (reserve, uncommit, etc) and the VirtualMemoryTracker accounting should be done together atomically. This is so that the NMT tracking information is updated the same order the memory operations actually happened.

This is currently not the case for os::commit_memory and os::reserve_memory. In those cases, the actual memory operation is done before acquiring the the NMT mutex. And the the virtual memory accounting is done later in MemTracker, after the lock has been acquired.

Additionally, the point at which the NMT lock is acquired is inconsistent across different memory operations. For example, os::release_memory, os::uncommit_memory, and os::unmap_memory don't suffer from the same problem because the NMT lock is acquired outside of MemTracker.

There are at least two possible solutions:
1. Move all NMT lock acquisitions up out of MemTracker and into os:: methods. This will keep the memory operations and accounting atomic, and also improve consistency.
2. Move the locking down the stack into platform specific code. This is a probably a better solution, but it diverges more from the existing code. Ex. lock around { munmap() + MemTracker::record }. This will help reduce the scope of the NMT lock and prevent possible deadlocks with other mutexes.

See this previous review for more context: https://github.com/openjdk/jdk/pull/20852#issuecomment-2364273437