P2
Run details page: http://aurora.ru.oracle.com/functional/faces/RunDetails.xhtml?names=388624.VMSQE.PIT-145#vm.gc%20%5B!desktop%5D%20(tonga)_gc/gctests/LoadUnloadGC
RULE gc/gctests/LoadUnloadGC Timeout none
Reproducible: 100%
Steps to reproduce: go to scaaa107 and launch local.rerun.sh.
Core file presents. A lot of core files. I wrote script that takes core each 5 seconds and it revealed that the only alive thread is always stuck in following point:
Thread 21 (Thread 0xe1caeb70 (LWP 15286)):
#0 DependencySignatureBuffer::add_if_missing (this=0xe1ca9594, sig=0xe0560640) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/code/dependencies.hpp:547
#1 0xf6e95f3b in nmethod::check_all_dependencies (changes=0xe1ca9668) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/code/nmethod.cpp:2184
#2 0xf69cbe91 in CodeCache::mark_for_deoptimization (changes=0xe1ca9668) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/code/codeCache.cpp:626
#3 0xf7034393 in Universe::flush_dependents_on (dependee=0xe1ca9698) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/memory/universe.cpp:1189
#4 0xf6fd9cbf in SystemDictionary::add_to_hierarchy (k=0xe1ca975c, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/classfile/systemDictionary.cpp:1654
#5 0xf6fe1fcb in SystemDictionary::define_instance_class (k=0xe1ca9920, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/classfile/systemDictionary.cpp:1443
#6 0xf6fe6053 in SystemDictionary::resolve_from_stream (class_name=0xe16123d8, class_loader=..., protection_domain=..., st=0xe1ca99e4, verify=1, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/classfile/systemDictionary.cpp:1155
#7 0xf6cd57ae in jvm_define_class_common(JNIEnv_*, char const*, _jobject*, signed char const*, int, _jobject*, char const*, unsigned char, Thread*) (func=0xe1ca9fa0) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/prims/jvm.cpp:907
#8 0xf6cd7a4e in JVM_DefineClassWithSource (env=0xe270c15c, name=0xe1ca9eec "nsk/share/gc/Class244", loader=0xe1ca9fa0, buf=0xe1904d20 "\312\376\272\276", len=253, pd=0xe1ca9fcc, source=0x0) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/prims/jvm.cpp:927
#9 0xf66d5086 in Java_java_lang_ClassLoader_defineClass1 () from /scratch/local/repr1/resources/jdk/jre/lib/i386/libjava.so
#10 0xf471b84b in ?? ()
#11 0xf47185f8 in ?? ()
#12 0xf6bf429d in JavaCalls::call_helper (result=0xe1caa598, m=0xe1caa3e8, args=0xe1caa464, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/javaCalls.cpp:402
#13 0xf6ec7899 in os::os_exception_wrapper (f=0xf6bf2ae0 <JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)>, value=0xe1caa598, method=0xe1caa3e8, args=0xe1caa464, thread=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/os/linux/vm/os_linux.cpp:5132
#14 0xf6bf67f0 in JavaCalls::call_virtual (result=0xe1caa598, spec_klass=0xe1caa4ac, name=0xe3eda328, signature=0xe3edbac8, args=0xe1caa464, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/javaCalls.cpp:307
#15 0xf6bf7094 in JavaCalls::call_virtual (result=0xe1caa598, receiver=..., spec_klass=0xe1caa5b4, name=0xe3eda328, signature=0xe3edbac8, arg1=..., __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/javaCalls.cpp:217
#16 0xf6fe2ec3 in SystemDictionary::load_instance_class (class_name=0xe16123d8, class_loader=..., __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/classfile/systemDictionary.cpp:1366
#17 0xf6fe389d in SystemDictionary::resolve_instance_class_or_null (name=0xe16123d8, class_loader=..., protection_domain=..., __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/classfile/systemDictionary.cpp:779
#18 0xf6fe3e9f in SystemDictionary::resolve_or_null (class_name=0xe16123d8, class_loader=..., protection_domain=..., __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/classfile/systemDictionary.cpp:232
#19 0xf6fe511e in SystemDictionary::resolve_or_fail (class_name=0xe16123d8, class_loader=..., protection_domain=..., throw_error=1, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/classfile/systemDictionary.cpp:171
#20 0xf6a55749 in ConstantPool::klass_at_impl(constantPoolHandle, int, Thread*) (this=0xe1caac14, which=2, uncached=-495927296) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/oops/constantPool.cpp:233
#21 0xf691723d in ConstantPool::klass_at(int, Thread*) (this=0xe017acd0) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/handles.inline.hpp:140
#22 0xf6913a1b in Runtime1::patch_code (thread=0xe270c000, stub_id=load_klass_patching_id) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/c1/c1_Runtime1.cpp:874
#23 0xf6914cb6 in Runtime1::move_klass_patching (thread=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/c1/c1_Runtime1.cpp:1193
#24 0xf46622f3 in ?? ()
#25 0xf4bd3f62 in ?? ()
#26 0xf6bf429d in JavaCalls::call_helper (result=0xe1cad848, m=0xe1cad83c, args=0xe1cad78c, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/javaCalls.cpp:402
#27 0xf6ec7899 in os::os_exception_wrapper (f=0xf6bf2ae0 <JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)>, value=0xe1cad848, method=0xe1cad83c, args=0xe1cad78c, thread=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/os/linux/vm/os_linux.cpp:5132
#28 0xf6bf1097 in JavaCalls::call (result=0xe1cad848, method=0xe1cad83c, args=0xe1cad78c, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/javaCalls.cpp:307
#29 0xf6f4daaf in Reflection::invoke (klass=0xe1cad9d0, reflected_method=0xe1cad9b8, receiver=..., override=1, ptypes=..., rtype=T_VOID, args=..., is_method_invoke=0, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/reflection.cpp:1070
#30 0xf6f4ed4d in Reflection::invoke_constructor (constructor_mirror=0xe1cada80, args=..., __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/reflection.cpp:1164
#31 0xf6ce1f2f in JVM_NewInstanceFromConstructor (env=0xe270c15c, c=0xe1cadb20, args0=0x0) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/prims/jvm.cpp:4005
#32 0xf66d641b in Java_sun_reflect_NativeConstructorAccessorImpl_newInstance0 () from /scratch/local/repr1/resources/jdk/jre/lib/i386/libjava.so
#33 0xf450f49d in ?? ()
#34 0xf45040e4 in ?? ()
#35 0xf45040e4 in ?? ()
#36 0xf4504119 in ?? ()
#37 0xf45040e4 in ?? ()
#38 0xf45040e4 in ?? ()
#39 0xf45041b9 in ?? ()
#40 0xf450043d in ?? ()
#41 0xf6bf429d in JavaCalls::call_helper (result=0xe1cae020, m=0xe1cadee8, args=0xe1cadf68, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/javaCalls.cpp:402
#42 0xf6ec7899 in os::os_exception_wrapper (f=0xf6bf2ae0 <JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)>, value=0xe1cae020, method=0xe1cadee8, args=0xe1cadf68, thread=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/os/linux/vm/os_linux.cpp:5132
#43 0xf6bf67f0 in JavaCalls::call_virtual (result=0xe1cae020, spec_klass=0xe1cadfb0, name=0xe3eda160, signature=0xe3edb5f8, args=0xe1cadf68, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/javaCalls.cpp:307
#44 0xf6bf71d5 in JavaCalls::call_virtual (result=..., receiver=..., spec_klass=0xe1cae02c, name=0xe3eda160, signature=0xe3edb5f8, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/javaCalls.cpp:210
#45 0xf6cd3a01 in thread_entry (thread=0xe270c000, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/prims/jvm.cpp:2812
#46 0xf701eda4 in JavaThread::thread_main_inner (this=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/thread.cpp:1699
#47 0xf701f144 in JavaThread::run (this=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/thread.cpp:1679
#48 0xf6ed0619 in java_start (thread=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/os/linux/vm/os_linux.cpp:860
#49 0x4665ba49 in start_thread () from /lib/libpthread.so.0
#50 0x4656ce1e in clone () from /lib/libc.so.6
Method at the top of stack was introduced in following changeset: http://hg.openjdk.java.net/jdk9/hs/hotspot/rev/3aaa4b9966f6
What does the test do: it loads a long chain of classes. Loading of each class makes next class in chain to be loaded. Chain consists of 1000 classes. After loading of all this chain test releases references and expects stuff to be unloaded.
As far as I understand stack trace jvm stucks in compiler flushing dependents of every new loaded class. Looks like jvm doesn't totally hang, rather spends too much time in this place, leaving it and entering again. I conclude it as I observe output of -XX:+PrintGCDetails. So, this bug isn't exactly hang, it's rather extreme performance regression.
RULE gc/gctests/LoadUnloadGC Timeout none
Reproducible: 100%
Steps to reproduce: go to scaaa107 and launch local.rerun.sh.
Core file presents. A lot of core files. I wrote script that takes core each 5 seconds and it revealed that the only alive thread is always stuck in following point:
Thread 21 (Thread 0xe1caeb70 (LWP 15286)):
#0 DependencySignatureBuffer::add_if_missing (this=0xe1ca9594, sig=0xe0560640) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/code/dependencies.hpp:547
#1 0xf6e95f3b in nmethod::check_all_dependencies (changes=0xe1ca9668) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/code/nmethod.cpp:2184
#2 0xf69cbe91 in CodeCache::mark_for_deoptimization (changes=0xe1ca9668) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/code/codeCache.cpp:626
#3 0xf7034393 in Universe::flush_dependents_on (dependee=0xe1ca9698) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/memory/universe.cpp:1189
#4 0xf6fd9cbf in SystemDictionary::add_to_hierarchy (k=0xe1ca975c, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/classfile/systemDictionary.cpp:1654
#5 0xf6fe1fcb in SystemDictionary::define_instance_class (k=0xe1ca9920, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/classfile/systemDictionary.cpp:1443
#6 0xf6fe6053 in SystemDictionary::resolve_from_stream (class_name=0xe16123d8, class_loader=..., protection_domain=..., st=0xe1ca99e4, verify=1, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/classfile/systemDictionary.cpp:1155
#7 0xf6cd57ae in jvm_define_class_common(JNIEnv_*, char const*, _jobject*, signed char const*, int, _jobject*, char const*, unsigned char, Thread*) (func=0xe1ca9fa0) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/prims/jvm.cpp:907
#8 0xf6cd7a4e in JVM_DefineClassWithSource (env=0xe270c15c, name=0xe1ca9eec "nsk/share/gc/Class244", loader=0xe1ca9fa0, buf=0xe1904d20 "\312\376\272\276", len=253, pd=0xe1ca9fcc, source=0x0) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/prims/jvm.cpp:927
#9 0xf66d5086 in Java_java_lang_ClassLoader_defineClass1 () from /scratch/local/repr1/resources/jdk/jre/lib/i386/libjava.so
#10 0xf471b84b in ?? ()
#11 0xf47185f8 in ?? ()
#12 0xf6bf429d in JavaCalls::call_helper (result=0xe1caa598, m=0xe1caa3e8, args=0xe1caa464, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/javaCalls.cpp:402
#13 0xf6ec7899 in os::os_exception_wrapper (f=0xf6bf2ae0 <JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)>, value=0xe1caa598, method=0xe1caa3e8, args=0xe1caa464, thread=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/os/linux/vm/os_linux.cpp:5132
#14 0xf6bf67f0 in JavaCalls::call_virtual (result=0xe1caa598, spec_klass=0xe1caa4ac, name=0xe3eda328, signature=0xe3edbac8, args=0xe1caa464, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/javaCalls.cpp:307
#15 0xf6bf7094 in JavaCalls::call_virtual (result=0xe1caa598, receiver=..., spec_klass=0xe1caa5b4, name=0xe3eda328, signature=0xe3edbac8, arg1=..., __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/javaCalls.cpp:217
#16 0xf6fe2ec3 in SystemDictionary::load_instance_class (class_name=0xe16123d8, class_loader=..., __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/classfile/systemDictionary.cpp:1366
#17 0xf6fe389d in SystemDictionary::resolve_instance_class_or_null (name=0xe16123d8, class_loader=..., protection_domain=..., __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/classfile/systemDictionary.cpp:779
#18 0xf6fe3e9f in SystemDictionary::resolve_or_null (class_name=0xe16123d8, class_loader=..., protection_domain=..., __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/classfile/systemDictionary.cpp:232
#19 0xf6fe511e in SystemDictionary::resolve_or_fail (class_name=0xe16123d8, class_loader=..., protection_domain=..., throw_error=1, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/classfile/systemDictionary.cpp:171
#20 0xf6a55749 in ConstantPool::klass_at_impl(constantPoolHandle, int, Thread*) (this=0xe1caac14, which=2, uncached=-495927296) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/oops/constantPool.cpp:233
#21 0xf691723d in ConstantPool::klass_at(int, Thread*) (this=0xe017acd0) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/handles.inline.hpp:140
#22 0xf6913a1b in Runtime1::patch_code (thread=0xe270c000, stub_id=load_klass_patching_id) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/c1/c1_Runtime1.cpp:874
#23 0xf6914cb6 in Runtime1::move_klass_patching (thread=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/c1/c1_Runtime1.cpp:1193
#24 0xf46622f3 in ?? ()
#25 0xf4bd3f62 in ?? ()
#26 0xf6bf429d in JavaCalls::call_helper (result=0xe1cad848, m=0xe1cad83c, args=0xe1cad78c, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/javaCalls.cpp:402
#27 0xf6ec7899 in os::os_exception_wrapper (f=0xf6bf2ae0 <JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)>, value=0xe1cad848, method=0xe1cad83c, args=0xe1cad78c, thread=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/os/linux/vm/os_linux.cpp:5132
#28 0xf6bf1097 in JavaCalls::call (result=0xe1cad848, method=0xe1cad83c, args=0xe1cad78c, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/javaCalls.cpp:307
#29 0xf6f4daaf in Reflection::invoke (klass=0xe1cad9d0, reflected_method=0xe1cad9b8, receiver=..., override=1, ptypes=..., rtype=T_VOID, args=..., is_method_invoke=0, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/reflection.cpp:1070
#30 0xf6f4ed4d in Reflection::invoke_constructor (constructor_mirror=0xe1cada80, args=..., __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/reflection.cpp:1164
#31 0xf6ce1f2f in JVM_NewInstanceFromConstructor (env=0xe270c15c, c=0xe1cadb20, args0=0x0) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/prims/jvm.cpp:4005
#32 0xf66d641b in Java_sun_reflect_NativeConstructorAccessorImpl_newInstance0 () from /scratch/local/repr1/resources/jdk/jre/lib/i386/libjava.so
#33 0xf450f49d in ?? ()
#34 0xf45040e4 in ?? ()
#35 0xf45040e4 in ?? ()
#36 0xf4504119 in ?? ()
#37 0xf45040e4 in ?? ()
#38 0xf45040e4 in ?? ()
#39 0xf45041b9 in ?? ()
#40 0xf450043d in ?? ()
#41 0xf6bf429d in JavaCalls::call_helper (result=0xe1cae020, m=0xe1cadee8, args=0xe1cadf68, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/javaCalls.cpp:402
#42 0xf6ec7899 in os::os_exception_wrapper (f=0xf6bf2ae0 <JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)>, value=0xe1cae020, method=0xe1cadee8, args=0xe1cadf68, thread=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/os/linux/vm/os_linux.cpp:5132
#43 0xf6bf67f0 in JavaCalls::call_virtual (result=0xe1cae020, spec_klass=0xe1cadfb0, name=0xe3eda160, signature=0xe3edb5f8, args=0xe1cadf68, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/javaCalls.cpp:307
#44 0xf6bf71d5 in JavaCalls::call_virtual (result=..., receiver=..., spec_klass=0xe1cae02c, name=0xe3eda160, signature=0xe3edb5f8, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/javaCalls.cpp:210
#45 0xf6cd3a01 in thread_entry (thread=0xe270c000, __the_thread__=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/prims/jvm.cpp:2812
#46 0xf701eda4 in JavaThread::thread_main_inner (this=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/thread.cpp:1699
#47 0xf701f144 in JavaThread::run (this=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/share/vm/runtime/thread.cpp:1679
#48 0xf6ed0619 in java_start (thread=0xe270c000) at /opt/jprt/T/P1/175911.amurillo/s/hotspot/src/os/linux/vm/os_linux.cpp:860
#49 0x4665ba49 in start_thread () from /lib/libpthread.so.0
#50 0x4656ce1e in clone () from /lib/libc.so.6
Method at the top of stack was introduced in following changeset: http://hg.openjdk.java.net/jdk9/hs/hotspot/rev/3aaa4b9966f6
What does the test do: it loads a long chain of classes. Loading of each class makes next class in chain to be loaded. Chain consists of 1000 classes. After loading of all this chain test releases references and expects stuff to be unloaded.
As far as I understand stack trace jvm stucks in compiler flushing dependents of every new loaded class. Looks like jvm doesn't totally hang, rather spends too much time in this place, leaving it and entering again. I conclude it as I observe output of -XX:+PrintGCDetails. So, this bug isn't exactly hang, it's rather extreme performance regression.