It happens even in interpreted mode, so something is fishy in runtime, I guess.

$ CONF=linux-x86-server-fastdebug make images run-test TEST=compiler/floatingpoint/NaNTest.java TEST_VM_OPTS="-XX:UseSSE=1 -Xint"

STDOUT:
### NanTest started
Written and read back float values match
0x7F800001 0x7F800001
STDERR:
java.lang.RuntimeException: Original and read back double values mismatch
0xFFF0000000000001 0xFFF8000000000001

at compiler.floatingpoint.NaNTest.testDouble(NaNTest.java:56)
at compiler.floatingpoint.NaNTest.main(NaNTest.java:69)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:298)
at java.base/java.lang.Thread.run(Thread.java:831)