We have a helper class responsible for /proc/self/smaps parsing available in test/hotspot/jtreg/runtime/os/TestTracePageSizes.java. Similar functionality is implemented in a new test:
test/hotspot/jtreg/gc/TestTransparentHugePagesHeap.java

We should look at moving the SmapsParser class into the testlib to make it possible to use it in both tests.