After push of JDK-8307478 , the following test started to fail on AIX :
com/sun/tools/attach/warnings/DynamicLoadWarningTest.java;
The problem was described in JDK-8309549 with a first try of a fix.
A second fix via JDK-8310191 was necessary.
Both fixes just disable the specific subtest on AIX, without correction of the root cause.
The root cause is, that dlopen() on AIX returns different handles every time, even if you load a library twice. There is no official AIX API available to get this information on a different way.
My proposal is, to use the stat64x API with the fields st_device and st_inode. After a dlopen() the stat64x() API is called additionally to get this information which is then stored parallel to the library handle in the jvmtiAgent. For AIX we then can compare these values instead of the library handle and get the same functionality as on linux.