The new test VendorInfoPluginsTest.java starts and crashes a VM without setting a value for the java heap. The VM will then determine the java heap to use using a heuristic approach. This could lead to very big heaps (I have seen more than 30 GB on our bigger AIX test machines).
The second test crashes the VM to parse the output for the vendor info. On windows no minidump file is created by default and on linux the core file is much smaller because on linux only the already used memory is included in the core dump. On AIX the whole reserved memory block seem to be dumped to the core file. This could result in very big core files written to disk and a big spike in used memory.

The problem can be easily reproduced by executing:
make test TEST="tools/jlink/plugins/VendorInfoPluginsTest.java" on an AIX

An easy solution would be to disable core file writing with -XX:-CreateCoredumpOnCrash (because the test does not need a core fiel) and setting a small heap with -Xmx64m.