Update 2024-06-09:

both THP-mode- and SHM-mode-tests had already been removed. All that remains of the original issue scope is the hugetlbfs sanity test.

Original description follows:

Upon startup, when probing kernel support for large pages on Linux, we attempt to allocate one page of the desired default hugepage size. If that succeeds, we count this as success.

In particular, these tests are:

- `os::Linux::transparent_huge_pages_sanity_check` - for probing THP support
- `os::Linux::hugetlbfs_sanity_check` - for probing mmap-based TLBFS support
- `os::Linux::shm_hugetlbfs_sanity_check` - for probing shmat-based TLBFS support

These tests, in their current forms, have the following issues:
- `os::Linux::shm_hugetlbfs_sanity_check()` only tests the default pagesize, essentially ignoring the pagesize argument. To make this work, one would have to pass `SHM_HUGE_2MB` etc. (note that the linux manpage explicitly mentions "2M" and "1G" as page sizes; these are only valid on x64, and the question arises how old and outdated this interface actually is. In contrast, mmap supports explicit hugepage sizes in a more generic manner. This suggests that mmap is the way to go, and we should maybe get rid of the shm tlbfs mode).
- `os::Linux::transparent_huge_pages_sanity_check` just won't work at all. In my tests, the result is always positive even if THPs are disabled on the system (separate issue opened: JDK-8310261)
- Neither of these tests check all page sizes. So, only the first (default) large page size is checked, we are still in the dark about other page sizes.

Furthermore, the point of these tests is questionable since they do not guarantee in any way that later allocations with this page size will succeed.

Furthermore, they incur costs: each probing costs at least two system calls (mmap and munmap), regardless of whether allocation succeeds. Then, the mmaps, if successful, actually allocate memory. In the case of huge pages, this may be a lot of memory (e.g. 1G pages). This is unnecessary work at startup.

And for explicit mmap-based TLBFS support, we may do this multiple times, too.

Argument: these tests should be replaced with explicit readings of configuration parameters from procfs:

- explicit hugepage support should just scan `nr_hugepages` and `nr_overcommit_hugepages` from the associated hugepage setting. The beauty of this would be that this would work for both smat- and mmap-based TLBFS.
- THP-mode should scan `/sys/kernel/mm/transparent_hugepages/enabled`.