FULL PRODUCT VERSION :
Any update of 32-bit JDK 6, JDK 7, JDK 8

ADDITIONAL OS VERSION INFORMATION :
32-bit Linux OS, 32-bit Oracle Solaris OS (for example Oracle Solaris 10 8/11 s10x_u10wos_17b X86)

A DESCRIPTION OF THE PROBLEM :
Infinite recursion in AWT native function J2DXErrHandler() causes a stack overflow and a crash under certain circumstances on 32-bit Linux OS or 32-bit Oracle Solaris OS. An approximation of J2DXErrHandler() function in pseudo-code is provided below.

J2DXErrHandler(parms) {
    if (errorBelongsToJvm(parms))
        letJvmHandleIt(parms);
    else if (pOldHandler)
        pOldHandler(parms);
}
 
The problem occurs, if another thread (for example, GTK thread) is doing the same sort of thing concurrently. This can lead to the following situation.

JVM thread: Sets J2DXErrHandler(), saves ANY_PREVIOUS_HANDLER as previous
    GTK thread: Sets some GTK_HANDLER, saves J2DXErrHandler() as previous
JVM thread: Restores ANY_PREVIOUS_HANDLER
    GTK thread: Restores J2DXErrHandler()
JVM thread: Sets J2DXErrHandler(), saves J2DXErrHandler() as previous

Then, if J2DXErrHandler() is called for an error that JVM doesn't own, this error handler calls a previous handler, which was saved, and passes the error for processing. Unfortunately the previous handler is also set to J2DXErrHandler(), thus the result is an infinite recursion, a stack overflow, and a crash.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. Make sure that OS used for this purpose is 32-bit Linux OS or 32-bit Oracle Solaris OS.
2. Download the attached "J2DXErrHandlerTestcase.zip" or "J2DXErrHandlerTestcase-Solaris.zip" archive depending on the OS type and unpack it.
3. Execute: java -cp xtest.jar -Djava.library.path=. xtest
    or unpack "src.zip", compile the test case using "makefile" script and execute the previously mentioned command line.
4. When Java2Demo window appears select the "Images" tab.
5. Select the menu item "Options->Background Color".
6. After the window appears, hit "Escape" to close it.
7. Repeat steps 5 and 6 in quick succession until the process crashes (may take 10-20 attempts).

Notes:
- The problem can be reproduced using tabs other than "Images".
- The problem may also be easier to reproduce on multi-core hosts.

REPRODUCIBILITY :
The problem is intermittent and may be reproduced using the test case after different number of attempts.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
$ /shared/jdks/oracle/6/jdk6_37/bin/java -cp xtest.jar -Djava.library.path=. xtest
Loading library
Starting xtest
Starting xtest: xesetTime=10 xeclearTime=10
Opening X display
Starting thread...
Waiting for thread to run...
Thread started
Thread running
Launch java2 demo
ODD 0x9fd9f950 != 0xb6b4b902
ODD 0x9f45b280 != 0xb6b4b902
ODD 0x9f465c40 != 0xb6b4b902
Launch complete
ODD 0x9f465c40 != 0xb6b4b902
URK! newXeh called
URK! newXeh called
URK! newXeh called
URK! newXeh called
ODD 0x9f465c40 != 0xb6b4b902
ODD 0x9f465c40 != 0xb6b4b902
Segmentation fault