(synopsis is provisional, change as you see fit)

This is confusing, because the JNI verification is not supposed to change the result. The test code does not look all that suspicious as well. So maybe the issue is somewhere in Xcheck:jni code rather than in the test or graphics code. Therefore I provisionally put it into hotspot/runtime.

% make test TEST=sun/awt/image/ImageRepresentation/LUTCompareTest.java

<passes>

% make test TEST=sun/awt/image/ImageRepresentation/LUTCompareTest.java TEST_VM_OPTS=-Xcheck:jni

Result color: ffff0000
STDERR:
java.lang.RuntimeException: Test FAILED!
        at LUTCompareTest.checkResults(LUTCompareTest.java:133)
        at LUTCompareTest.main(LUTCompareTest.java:78)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        at java.base/java.lang.reflect.Method.invoke(Method.java:565)
        at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
        at java.base/java.lang.Thread.run(Thread.java:1516)