An organisation within Ericsson bumped into the problem described below.
I would like to ask your help in order to investigate this problem a bit further.
Since it seems to be a bug in the JDK implementation a bug report might be necessary. In that case, could you direct me (or the request itself) towards the proper organisation?
I also attach a file that can help the investigation.
-------------------------------------------------------
Problem description:
We have a build framework (written in scripts) that calls javac on SUSE SLES9 workskations. In some cases javac crashes with Hotspot error, and this is preventing some of our mission critical applications from being built, which is very severe. As this does not happen all the time, happens on different computers, and as far as we know, unexpectedly, it has been very hard for us to pin point and isolate the problem. We have had a few ideas of why this happens, such as maybe a known bug in JDK version 1.5.0_6 which is the version we were using when we first spotted the problem. We have since then upgraded JDK to version 1.5.0_12 and unfortunately the problem has not been solved. We have tried to set the JVM_FLAGS to "-server -XX:-EliminateLocks" and the result is indifferent. Another explanation is that the system in special circumstances behave in such a way that causes the Java crash. We believe that this is not the case as the problem happens on several computers under different conditions. We are out of ideas and hope that you can help us to find a solution to the problem. Below I have provided you the javac command issued when we get the Hotspot error, and attached an hs_err_pidxxxxx log file.
javac command issued when we build application:
/local/tmp/jdk_1.5.0_12/JDK_LM/jdk/bin/javac -encoding utf-8 -classpath '/vobs/tas/tas_code/imsas/src/30_PlatformSupportLayer_SA/Oam_SE/Pm_OB/ImsAsPmf_SWI/PmMtasWeShare/generated/telorb/PmMtasWeShareIntra/PmMtasWeShareIntra_SWI/user/jsrc:/vobs/tas/tas_code/imsas/src/30_PlatformSupportLayer_SA/Oam_SE/Pm_OB/ImsAsPmf_SWI/PmMtasWeShare/generated/telorb/PmMtasWeShareIntra/PmMtasWeShareIntra_SWI/jsrc:/proj/tas/opt/tsp/icp5300.2/TADE-5.3-RM5300.0-M-0-2/system/TelORB/CXA1100926_1_R3H03-TelORB_API/included/SWI_D1-JavaDpi/jclasses:/proj/tas/opt/tsp/icp5300.2/TADE-5.3-RM5300.0-M-0-2/system/TelORB/CXA1100926_1_R3H03-TelORB_API/included/SWI_D10-JavaBroker/jclasses:/proj/tas/opt/tsp/icp5300.2/TADE-5.3-RM5300.0-M-0-2/system/TelORB/CXA1100934_1_R3A01-PerformanceMgmt_API/included/SWI_PMINTRASCANNER-PMINTRASCANNER/jclasses' -d /vobs/tas/tas_code/imsas/src/30_PlatformSupportLayer_SA/Oam_SE/Pm_OB/ImsAsPmf_SWI/PmMtasWeShare/generated/telorb/PmMtasWeShareIntra/PmMtasWeShareIntra_SWI/jclasses -deprecation /vobs/tas/tas_code/imsas/src/30_PlatformSupportLayer_SA/Oam_SE/Pm_OB/ImsAsPmf_SWI/PmMtasWeShare/generated/telorb/PmMtasWeShareIntra/PmMtasWeShareIntra_SWI/user/jsrc/se/ericsson/telorb/delos/PmMtasWeShareIntraProcess_Name.java /vobs/tas/tas_code/imsas/src/30_PlatformSupportLayer_SA/Oam_SE/Pm_OB/ImsAsPmf_SWI/PmMtasWeShare/generated/telorb/PmMtasWeShareIntra/PmMtasWeShareIntra_SWI/jsrc/PmMtasWeShareIntra.java
Or in short:
/local/tmp/jdk_1.5.0_12/JDK_LM/jdk/bin/javac -encoding utf-8 -classpath <paths> -d <path> -depcrecation <path> <java file>
Error Message outputted to standard out:
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (0xb) at pc=0x402e2666, pid=16450, tid=1316027312
#
# Java VM: Java HotSpot(TM) Client VM (1.5.0_12-b04 mixed mode)
# Problematic frame:
# V [libjvm.so+0x17a666]
#
# An error report file with more information is saved as hs_err_pid16450.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
Thanks in advance.
Best Regards,
Ericsson has come back with more information.
-------------------------------------------------------------------------------
We have now tested with JDK 6 aswell and we get the Java HotSpot error:
#
# An unexpected error has been detected by Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x06164192, pid=23226, tid=1130638256 # # Java VM: Java HotSpot(TM) Client VM (1.6.0_03-b05 mixed mode, sharing) # Problematic frame:
# V [libjvm.so+0x164192]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
I have attached the error log as hs_err_pid23226.log
Interestingly we also get the same HotSpot error in a different situation:
From one of our developers:
I get the following error when testing my Eclipse plugin by running as "Eclipse application" from Eclipse 3.3.1.1 with TPTP 4.4.0.3 and 4.4.1 candidate build.
The vmargs are:
-Xms64m -Xmx768m -XrunpiAgent:server=enabled I then "Attach to agent" through TPTP in Eclipse. I've previously started TPTP Agent 4.4.0.3.
When profiling the application with TPTP, the Eclipse instance being profiled is extremely slow, always peeking the processor. Usually within a few minutes of profiling, the Eclipse instance being profiled closes down, and a short Java Hotspot Server error message is shown in the terminal, with reference to such a HS-dump.
The error message is the same as the previous error we had, which is interesting. We can reproduce the error described above any time we want.
I have attached the logfile hs_err_pid17748.log when this happens.
All the crashes appear to result from the execution of os::write_memory_serialization_page since we're in the _trans state.
hs_err_pid17748.log:Current thread (0x0805f048): JavaThread "main" [_thread_in_vm_trans, id=17748]
hs_err_pid23226.log:Current thread (0x42c06800): JavaThread "CompilerThread0" daemon [_thread_in_native_trans, id=23232]
hs_err_pid28206:Current thread (0x080c7a50): JavaThread "Signal Dispatcher" daemon [_thread_in_vm_trans, id=28210]
Decoding the hs_err logs suggests that the register state is getting corrupted at the point where we die.
hs_err_pid23226.log:
siginfo:si_signo=11, si_errno=0, si_code=0, si_addr=0x00000000;;
Registers:
EAX=0x40595000, EBX=0x42c06800, ECX=0x00000d00, EDX=0x00000ffc
ESP=0x43641c70, EBP=0x43641cc8, ESI=0x43641db0, EDI=0x43641dec
EIP=0x06164192, CR2=0x40595d00, EFLAGS=0x00210206
;; 06164192 c7 04 01 01 00 00 00 movl $0x1,(%ecx,%eax,1)
The reported fault location is 0 but ecx + eax is 0x40595d00.
hs_err_pid17748.log:
siginfo:si_signo=11, si_errno=0, si_code=0, si_addr=0x00002025;;
Registers:
EAX=0x40b2e000, EBX=0x407127f0, ECX=0x00000ffc, EDX=0x00000c10
ESP=0xbfff9bd8, EBP=0xbfff9c20, ESI=0x0805f048, EDI=0x5c849198
EIP=0x4048a5ef, CR2=0x40b2ec10, EFLAGS=0x00010202
;; 4048a5ef c7 04 02 01 00 00 00 movl $0x1,(%edx,%eax,1)
again the si_addr is not 0x40b2ec10
hs_err_pid28206:
siginfo:si_signo=11, si_errno=0, si_code=0, si_addr=0x00000000;;
Registers:
EAX=0x40910000, EBX=0x404f542c, ECX=0x00000ffc, EDX=0x00000e94
ESP=0x4e70ee50, EBP=0x4e70ee68, ESI=0x080c7a50, EDI=0x4e70eee0
EIP=0x402e2666, CR2=0x40910e94, EFLAGS=0x00210202
;; 402e2666 c7 04 02 01 00 00 00 movl $0x1,(%edx,%eax,1)
si_addr is 0 but edx + eax = 0x40910e94
It might be interesting to try using -XX:+UseMemBar which switches away from using the memory serialize page.
Ericsson informs us that -XX:+UseMemBar did not help.
They are going to look into what they might want to do next,
but indicated that this was not a blocking issue.
I would like to ask your help in order to investigate this problem a bit further.
Since it seems to be a bug in the JDK implementation a bug report might be necessary. In that case, could you direct me (or the request itself) towards the proper organisation?
I also attach a file that can help the investigation.
-------------------------------------------------------
Problem description:
We have a build framework (written in scripts) that calls javac on SUSE SLES9 workskations. In some cases javac crashes with Hotspot error, and this is preventing some of our mission critical applications from being built, which is very severe. As this does not happen all the time, happens on different computers, and as far as we know, unexpectedly, it has been very hard for us to pin point and isolate the problem. We have had a few ideas of why this happens, such as maybe a known bug in JDK version 1.5.0_6 which is the version we were using when we first spotted the problem. We have since then upgraded JDK to version 1.5.0_12 and unfortunately the problem has not been solved. We have tried to set the JVM_FLAGS to "-server -XX:-EliminateLocks" and the result is indifferent. Another explanation is that the system in special circumstances behave in such a way that causes the Java crash. We believe that this is not the case as the problem happens on several computers under different conditions. We are out of ideas and hope that you can help us to find a solution to the problem. Below I have provided you the javac command issued when we get the Hotspot error, and attached an hs_err_pidxxxxx log file.
javac command issued when we build application:
/local/tmp/jdk_1.5.0_12/JDK_LM/jdk/bin/javac -encoding utf-8 -classpath '/vobs/tas/tas_code/imsas/src/30_PlatformSupportLayer_SA/Oam_SE/Pm_OB/ImsAsPmf_SWI/PmMtasWeShare/generated/telorb/PmMtasWeShareIntra/PmMtasWeShareIntra_SWI/user/jsrc:/vobs/tas/tas_code/imsas/src/30_PlatformSupportLayer_SA/Oam_SE/Pm_OB/ImsAsPmf_SWI/PmMtasWeShare/generated/telorb/PmMtasWeShareIntra/PmMtasWeShareIntra_SWI/jsrc:/proj/tas/opt/tsp/icp5300.2/TADE-5.3-RM5300.0-M-0-2/system/TelORB/CXA1100926_1_R3H03-TelORB_API/included/SWI_D1-JavaDpi/jclasses:/proj/tas/opt/tsp/icp5300.2/TADE-5.3-RM5300.0-M-0-2/system/TelORB/CXA1100926_1_R3H03-TelORB_API/included/SWI_D10-JavaBroker/jclasses:/proj/tas/opt/tsp/icp5300.2/TADE-5.3-RM5300.0-M-0-2/system/TelORB/CXA1100934_1_R3A01-PerformanceMgmt_API/included/SWI_PMINTRASCANNER-PMINTRASCANNER/jclasses' -d /vobs/tas/tas_code/imsas/src/30_PlatformSupportLayer_SA/Oam_SE/Pm_OB/ImsAsPmf_SWI/PmMtasWeShare/generated/telorb/PmMtasWeShareIntra/PmMtasWeShareIntra_SWI/jclasses -deprecation /vobs/tas/tas_code/imsas/src/30_PlatformSupportLayer_SA/Oam_SE/Pm_OB/ImsAsPmf_SWI/PmMtasWeShare/generated/telorb/PmMtasWeShareIntra/PmMtasWeShareIntra_SWI/user/jsrc/se/ericsson/telorb/delos/PmMtasWeShareIntraProcess_Name.java /vobs/tas/tas_code/imsas/src/30_PlatformSupportLayer_SA/Oam_SE/Pm_OB/ImsAsPmf_SWI/PmMtasWeShare/generated/telorb/PmMtasWeShareIntra/PmMtasWeShareIntra_SWI/jsrc/PmMtasWeShareIntra.java
Or in short:
/local/tmp/jdk_1.5.0_12/JDK_LM/jdk/bin/javac -encoding utf-8 -classpath <paths> -d <path> -depcrecation <path> <java file>
Error Message outputted to standard out:
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (0xb) at pc=0x402e2666, pid=16450, tid=1316027312
#
# Java VM: Java HotSpot(TM) Client VM (1.5.0_12-b04 mixed mode)
# Problematic frame:
# V [libjvm.so+0x17a666]
#
# An error report file with more information is saved as hs_err_pid16450.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
Thanks in advance.
Best Regards,
Ericsson has come back with more information.
-------------------------------------------------------------------------------
We have now tested with JDK 6 aswell and we get the Java HotSpot error:
#
# An unexpected error has been detected by Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x06164192, pid=23226, tid=1130638256 # # Java VM: Java HotSpot(TM) Client VM (1.6.0_03-b05 mixed mode, sharing) # Problematic frame:
# V [libjvm.so+0x164192]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
I have attached the error log as hs_err_pid23226.log
Interestingly we also get the same HotSpot error in a different situation:
From one of our developers:
I get the following error when testing my Eclipse plugin by running as "Eclipse application" from Eclipse 3.3.1.1 with TPTP 4.4.0.3 and 4.4.1 candidate build.
The vmargs are:
-Xms64m -Xmx768m -XrunpiAgent:server=enabled I then "Attach to agent" through TPTP in Eclipse. I've previously started TPTP Agent 4.4.0.3.
When profiling the application with TPTP, the Eclipse instance being profiled is extremely slow, always peeking the processor. Usually within a few minutes of profiling, the Eclipse instance being profiled closes down, and a short Java Hotspot Server error message is shown in the terminal, with reference to such a HS-dump.
The error message is the same as the previous error we had, which is interesting. We can reproduce the error described above any time we want.
I have attached the logfile hs_err_pid17748.log when this happens.
All the crashes appear to result from the execution of os::write_memory_serialization_page since we're in the _trans state.
hs_err_pid17748.log:Current thread (0x0805f048): JavaThread "main" [_thread_in_vm_trans, id=17748]
hs_err_pid23226.log:Current thread (0x42c06800): JavaThread "CompilerThread0" daemon [_thread_in_native_trans, id=23232]
hs_err_pid28206:Current thread (0x080c7a50): JavaThread "Signal Dispatcher" daemon [_thread_in_vm_trans, id=28210]
Decoding the hs_err logs suggests that the register state is getting corrupted at the point where we die.
hs_err_pid23226.log:
siginfo:si_signo=11, si_errno=0, si_code=0, si_addr=0x00000000;;
Registers:
EAX=0x40595000, EBX=0x42c06800, ECX=0x00000d00, EDX=0x00000ffc
ESP=0x43641c70, EBP=0x43641cc8, ESI=0x43641db0, EDI=0x43641dec
EIP=0x06164192, CR2=0x40595d00, EFLAGS=0x00210206
;; 06164192 c7 04 01 01 00 00 00 movl $0x1,(%ecx,%eax,1)
The reported fault location is 0 but ecx + eax is 0x40595d00.
hs_err_pid17748.log:
siginfo:si_signo=11, si_errno=0, si_code=0, si_addr=0x00002025;;
Registers:
EAX=0x40b2e000, EBX=0x407127f0, ECX=0x00000ffc, EDX=0x00000c10
ESP=0xbfff9bd8, EBP=0xbfff9c20, ESI=0x0805f048, EDI=0x5c849198
EIP=0x4048a5ef, CR2=0x40b2ec10, EFLAGS=0x00010202
;; 4048a5ef c7 04 02 01 00 00 00 movl $0x1,(%edx,%eax,1)
again the si_addr is not 0x40b2ec10
hs_err_pid28206:
siginfo:si_signo=11, si_errno=0, si_code=0, si_addr=0x00000000;;
Registers:
EAX=0x40910000, EBX=0x404f542c, ECX=0x00000ffc, EDX=0x00000e94
ESP=0x4e70ee50, EBP=0x4e70ee68, ESI=0x080c7a50, EDI=0x4e70eee0
EIP=0x402e2666, CR2=0x40910e94, EFLAGS=0x00210202
;; 402e2666 c7 04 02 01 00 00 00 movl $0x1,(%edx,%eax,1)
si_addr is 0 but edx + eax = 0x40910e94
It might be interesting to try using -XX:+UseMemBar which switches away from using the memory serialize page.
Ericsson informs us that -XX:+UseMemBar did not help.
They are going to look into what they might want to do next,
but indicated that this was not a blocking issue.