FULL PRODUCT VERSION :
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)


FULL OS VERSION :
Linux sb1-4.sli-systems.net 2.6.32-573.3.1.el6.x86_64 #1 SMP Fri Aug 14 03:56:13 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux


A DESCRIPTION OF THE PROBLEM :
Our application is single threaded, sometimes it will stuck in seemingly impossible places with 100% CPU usage, not making any progress.

Looking at reports generated by perf on Linux, > 85% of time is spent in SharedRuntime::handle_wrong_method -> SharedRuntime::reresolve_call_site -> ....

There is no attachment upload on this page, but you can email me and I can send through the perf data and flame graph.


THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Did not try

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Did not try

ERROR MESSAGES/STACK TRACES THAT OCCUR :
Output of jstack (thread "main" is stuck):

2016-03-15 09:34:17
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode):

"Attach Listener" #23 daemon prio=9 os_prio=0 tid=0x00007f9ccc001000 nid=0x5cc9 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Service Thread" #21 daemon prio=9 os_prio=0 tid=0x00007f9d2474f800 nid=0x9c6a runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread14" #20 daemon prio=9 os_prio=0 tid=0x00007f9d24742000 nid=0x9c69 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread13" #19 daemon prio=9 os_prio=0 tid=0x00007f9d2473f800 nid=0x9c68 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread12" #18 daemon prio=9 os_prio=0 tid=0x00007f9d2473d800 nid=0x9c67 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread11" #17 daemon prio=9 os_prio=0 tid=0x00007f9d2473b800 nid=0x9c66 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread10" #16 daemon prio=9 os_prio=0 tid=0x00007f9d24739800 nid=0x9c64 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread9" #15 daemon prio=9 os_prio=0 tid=0x00007f9d24736800 nid=0x9c63 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread8" #14 daemon prio=9 os_prio=0 tid=0x00007f9d24734800 nid=0x9c62 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread7" #13 daemon prio=9 os_prio=0 tid=0x00007f9d24732800 nid=0x9c60 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread6" #12 daemon prio=9 os_prio=0 tid=0x00007f9d24730000 nid=0x9c5f waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread5" #11 daemon prio=9 os_prio=0 tid=0x00007f9d2472e000 nid=0x9c5e waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread4" #10 daemon prio=9 os_prio=0 tid=0x00007f9d2472b800 nid=0x9c5d waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread3" #9 daemon prio=9 os_prio=0 tid=0x00007f9d24721800 nid=0x9c5c waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f9d2471f000 nid=0x9c5b waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f9d2471d000 nid=0x9c5a waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f9d2471a800 nid=0x9c59 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f9d24718800 nid=0x9c58 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007f9d24716800 nid=0x9c57 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f9d246db000 nid=0x9c44 in Object.wait() [0x00007f9cfd5fc000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000637a7def8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x0000000637a7def8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f9d246d9000 nid=0x9c43 in Object.wait() [0x00007f9cfd6fd000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000637a7e0b0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
- locked <0x0000000637a7e0b0> (a java.lang.ref.Reference$Lock)

"main" #1 prio=5 os_prio=0 tid=0x00007f9d24009000 nid=0x9c21 runnable [0x00007f9d285ca000]
   java.lang.Thread.State: RUNNABLE
at com.ibm.icu.impl.Trie2$Trie2Iterator.next(Trie2.java:910)
at com.ibm.icu.impl.Trie2$Trie2Iterator.next(Trie2.java:858)
at com.ibm.icu.impl.Normalizer2Impl.addLcccChars(Normalizer2Impl.java:539)
at com.ibm.icu.impl.coll.CollationDataReader.read(CollationDataReader.java:321)
at com.ibm.icu.impl.coll.CollationRoot.<clinit>(CollationRoot.java:49)
at com.ibm.icu.impl.coll.CollationLoader.loadTailoring(CollationLoader.java:78)
at com.ibm.icu.text.CollatorServiceShim.makeInstance(CollatorServiceShim.java:154)
at com.ibm.icu.text.CollatorServiceShim.access$000(CollatorServiceShim.java:26)
at com.ibm.icu.text.CollatorServiceShim$CService$1CollatorFactory.handleCreate(CollatorServiceShim.java:128)
at com.ibm.icu.impl.ICULocaleService$LocaleKeyFactory.create(ICULocaleService.java:376)
at com.ibm.icu.impl.ICUService.getKey(ICUService.java:461)
at com.ibm.icu.impl.ICUService.getKey(ICUService.java:381)
at com.ibm.icu.impl.ICULocaleService.get(ICULocaleService.java:71)
at com.ibm.icu.impl.ICULocaleService.get(ICULocaleService.java:55)
at com.ibm.icu.text.CollatorServiceShim.getInstance(CollatorServiceShim.java:35)
at com.ibm.icu.text.Collator.getInstance(Collator.java:615)
at org.apache.lucene.collation.ICUCollationKeyFilterFactory.createFromLocale(ICUCollationKeyFilterFactory.java:188)
at org.apache.lucene.collation.ICUCollationKeyFilterFactory.inform(ICUCollationKeyFilterFactory.java:116)
at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:681)
at org.apache.solr.schema.IndexSchema.<init>(IndexSchema.java:167)
at com.example.phraseNormaliser.PhraseNormaliserResourceLoader.initIndex(PhraseNormaliserResourceLoader.java:163)
at com.example.phraseNormaliser.IndexPhraseNormaliser.init(IndexPhraseNormaliser.java:25)
at com.example.phraseNormaliser.IndexPhraseNormaliser.<init>(IndexPhraseNormaliser.java:21)
at com.example.index.moby.MobyIndexer.create(MobyIndexer.java:193)
at com.example.main.Main.main(Main.java:138)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.example.main.Starter.main(Starter.java:60)

"VM Thread" os_prio=0 tid=0x00007f9d246d4000 nid=0x9c40 runnable

"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f9d2401f800 nid=0x9c24 runnable

"Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f9d24021000 nid=0x9c25 runnable

"G1 Main Concurrent Mark GC Thread" os_prio=0 tid=0x00007f9d24039800 nid=0x9c29 runnable

"Gang worker#0 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007f9d2403b800 nid=0x9c2a runnable

"G1 Concurrent Refinement Thread#0" os_prio=0 tid=0x00007f9d24027800 nid=0x9c28 runnable

"G1 Concurrent Refinement Thread#1" os_prio=0 tid=0x00007f9d24025800 nid=0x9c27 runnable

"G1 Concurrent Refinement Thread#2" os_prio=0 tid=0x00007f9d24023800 nid=0x9c26 runnable

"String Deduplication Thread" os_prio=0 tid=0x00007f9d2465a800 nid=0x9c33 runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007f9d24752000 nid=0x9c6c waiting on condition

JNI global references: 264







Output of jstack -m:

Attaching to process ID 39966, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.45-b02
Deadlock Detection:

No deadlocks found.

----------------- 39969 -----------------
0x00007f9d28b51f72 _ZNK5frame21oopmapreg_to_locationEP9VMRegImplPK11RegisterMap + 0x32
0x00007f9d28ecd4fa _ZN9OopMapSet19update_register_mapEPK5frameP11RegisterMap + 0x15a
0x00007f9d28b562b5 _ZNK5frame25sender_for_compiled_frameEP11RegisterMap + 0x135
0x00007f9d28b565bc _ZNK5frame6senderEP11RegisterMap + 0x4c
0x00007f9d28f7d957 _ZN13SharedRuntime23find_callee_info_helperEP10JavaThreadR12vframeStreamRN9Bytecodes4CodeER8CallInfoP6Thread + 0x2e7
0x00007f9d28f7e1a9 _ZN13SharedRuntime18find_callee_methodEP10JavaThreadP6Thread + 0xf9
0x00007f9d28f7e943 _ZN13SharedRuntime19reresolve_call_siteEP10JavaThreadP6Thread + 0xb3
0x00007f9d28f7eda8 _ZN13SharedRuntime19handle_wrong_methodEP10JavaThread + 0x158
0x00007f9d149b039a <RuntimeStub>
0x00007f9d15199ff4 * com.ibm.icu.impl.Trie2$Trie2Iterator.next() bci:110 line:910 (Compiled frame)
* com.ibm.icu.impl.Trie2$Trie2Iterator.next() bci:1 line:858 (Compiled frame)
* com.ibm.icu.impl.Normalizer2Impl.addLcccChars(com.ibm.icu.text.UnicodeSet) bci:18 line:539 (Interpreted frame)
0x00007f9d149b2c4d * com.ibm.icu.impl.coll.CollationDataReader.read(com.ibm.icu.impl.coll.CollationTailoring, java.io.InputStream, com.ibm.icu.impl.coll.CollationTailoring) bci:1278 line:321 (Interpreted frame)
0x00007f9d149b2c4d * com.ibm.icu.impl.coll.CollationRoot.<clinit>() bci:22 line:49 (Interpreted frame)
0x00007f9d149ab7a7 <StubRoutines>
0x00007f9d28c4fa26 _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x1056
0x00007f9d28c02fd7 _ZN13InstanceKlass27call_class_initializer_implE19instanceKlassHandleP6Thread + 0xd7
0x00007f9d28c04c2c _ZN13InstanceKlass15initialize_implE19instanceKlassHandleP6Thread + 0x1ac
0x00007f9d28c04fe1 _ZN13InstanceKlass10initializeEP6Thread + 0x41
0x00007f9d28db7756 _ZN12LinkResolver19resolve_static_callER8CallInfoR11KlassHandleP6SymbolS5_S2_bbP6Thread + 0x246
0x00007f9d28db79df _ZN12LinkResolver20resolve_invokestaticER8CallInfo18constantPoolHandleiP6Thread + 0x23f
0x00007f9d28db7f11 _ZN12LinkResolver14resolve_invokeER8CallInfo6Handle18constantPoolHandleiN9Bytecodes4CodeEP6Thread + 0x4f1
0x00007f9d28c48972 _ZN18InterpreterRuntime14resolve_invokeEP10JavaThreadN9Bytecodes4CodeE + 0x1b2
0x00007f9d149cccda * com.ibm.icu.impl.coll.CollationLoader.loadTailoring(com.ibm.icu.util.ULocale, com.ibm.icu.util.Output) bci:0 line:78 (Interpreted frame)
0x00007f9d149b29d0 * com.ibm.icu.text.CollatorServiceShim.makeInstance(com.ibm.icu.util.ULocale) bci:13 line:154 (Interpreted frame)
0x00007f9d149b29d0 * com.ibm.icu.text.CollatorServiceShim.access$000(com.ibm.icu.util.ULocale) bci:1 line:26 (Interpreted frame)
0x00007f9d149b29d0 * com.ibm.icu.text.CollatorServiceShim$CService$1CollatorFactory.handleCreate(com.ibm.icu.util.ULocale, int, com.ibm.icu.impl.ICUService) bci:1 line:128 (Interpreted frame)
0x00007f9d149b29d0 * com.ibm.icu.impl.ICULocaleService$LocaleKeyFactory.create(com.ibm.icu.impl.ICUService$Key, com.ibm.icu.impl.ICUService) bci:31 line:376 (Interpreted frame)
0x00007f9d149b2aa4 * com.ibm.icu.impl.ICUService.getKey(com.ibm.icu.impl.ICUService$Key, java.lang.String[], com.ibm.icu.impl.ICUService$Factory) bci:567 line:461 (Interpreted frame)
0x00007f9d149b29d0 * com.ibm.icu.impl.ICUService.getKey(com.ibm.icu.impl.ICUService$Key, java.lang.String[]) bci:4 line:381 (Interpreted frame)
0x00007f9d149b29d0 * com.ibm.icu.impl.ICULocaleService.get(com.ibm.icu.util.ULocale, int, com.ibm.icu.util.ULocale[]) bci:30 line:71 (Interpreted frame)
0x00007f9d149b29d0 * com.ibm.icu.impl.ICULocaleService.get(com.ibm.icu.util.ULocale, com.ibm.icu.util.ULocale[]) bci:4 line:55 (Interpreted frame)
0x00007f9d149b29d0 * com.ibm.icu.text.CollatorServiceShim.getInstance(com.ibm.icu.util.ULocale) bci:10 line:35 (Interpreted frame)
0x00007f9d149b29d0 * com.ibm.icu.text.Collator.getInstance(com.ibm.icu.util.ULocale) bci:4 line:615 (Interpreted frame)
0x00007f9d149b29d0 * org.apache.lucene.collation.ICUCollationKeyFilterFactory.createFromLocale(java.lang.String) bci:8 line:188 (Interpreted frame)
0x00007f9d149b29d0 * org.apache.lucene.collation.ICUCollationKeyFilterFactory.inform(org.apache.lucene.analysis.util.ResourceLoader) bci:13 line:116 (Interpreted frame)
0x00007f9d149b2c92 * org.apache.solr.core.SolrResourceLoader.inform(org.apache.solr.common.ResourceLoader) bci:90 line:681 (Interpreted frame)
0x00007f9d149b2c4d * org.apache.solr.schema.IndexSchema.<init>(org.apache.solr.core.SolrConfig, java.lang.String, org.xml.sax.InputSource) bci:196 line:167 (Interpreted frame)
0x00007f9d149b2c4d * com.example.phraseNormaliser.PhraseNormaliserResourceLoader.initIndex() bci:83 line:163 (Interpreted frame)
0x00007f9d149b29d0 * com.example.phraseNormaliser.IndexPhraseNormaliser.init() bci:5 line:25 (Interpreted frame)
0x00007f9d149b2c4d * com.example.phraseNormaliser.IndexPhraseNormaliser.<init>(java.lang.String) bci:17 line:21 (Interpreted frame)
0x00007f9d149b2c4d * com.example.index.moby.MobyIndexer.create(com.example.index.moby.MobyIndexType) bci:189 line:193 (Interpreted frame)
0x00007f9d149b2c4d * com.example.main.Main.main(java.lang.String[]) bci:551 line:138 (Interpreted frame)
0x00007f9d149ab7a7 <StubRoutines>
0x00007f9d28c4fa26 _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x1056
0x00007f9d28f52f17 _ZN10Reflection6invokeE19instanceKlassHandle12methodHandle6Handleb14objArrayHandle9BasicTypeS3_bP6Thread + 0x5d7
0x00007f9d28f565e7 _ZN10Reflection13invoke_methodEP7oopDesc6Handle14objArrayHandleP6Thread + 0x147
0x00007f9d28cdd5ab JVM_InvokeMethod + 0x26b
0x00007f9d149c0a14 * sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) bci:0 (Interpreted frame)
0x00007f9d149b29d0 * sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) bci:100 line:62 (Interpreted frame)
0x00007f9d149b29d0 * sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) bci:6 line:43 (Interpreted frame)
0x00007f9d149b2aa4 * java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) bci:56 line:497 (Interpreted frame)
0x00007f9d149b29d0 * com.example.main.Starter.main(java.lang.String[]) bci:238 line:60 (Interpreted frame)
0x00007f9d149ab7a7 <StubRoutines>
0x00007f9d28c4fa26 _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x1056
0x00007f9d28c91692 _ZL17jni_invoke_staticP7JNIEnv_P9JavaValueP8_jobject11JNICallTypeP10_jmethodIDP18JNI_ArgumentPusherP6Thread + 0x362
0x00007f9d28cae09a jni_CallStaticVoidMethod + 0x17a
0x00000033ade07bcc JavaMain + 0x80c
----------------- 39972 -----------------
0x00000033ad60b68c __pthread_cond_wait + 0xcc
0x00007f9d28e98917 _ZN7Monitor5IWaitEP6Threadl + 0xf7
0x00007f9d28e99236 _ZN7Monitor4waitEblb + 0x256
0x00007f9d2909a10b _ZN10GangWorker4loopEv + 0x5b
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 39973 -----------------
0x00000033ad60b68c __pthread_cond_wait + 0xcc
0x00007f9d28e98917 _ZN7Monitor5IWaitEP6Threadl + 0xf7
0x00007f9d28e99236 _ZN7Monitor4waitEblb + 0x256
0x00007f9d2909a10b _ZN10GangWorker4loopEv + 0x5b
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 39974 -----------------
0x00000033ad60ba5e __pthread_cond_timedwait + 0x13e
0x00007f9d28e98c6e _ZN7Monitor5IWaitEP6Threadl + 0x44e
0x00007f9d28e99236 _ZN7Monitor4waitEblb + 0x256
0x00007f9d28a838eb _ZN24ConcurrentG1RefineThread21run_young_rs_samplingEv + 0x7b
0x00007f9d28a84008 _ZN24ConcurrentG1RefineThread3runEv + 0x3f8
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 39975 -----------------
0x00000033ad60b68c __pthread_cond_wait + 0xcc
0x00007f9d28e98917 _ZN7Monitor5IWaitEP6Threadl + 0xf7
0x00007f9d28e99236 _ZN7Monitor4waitEblb + 0x256
0x00007f9d28a83e1f _ZN24ConcurrentG1RefineThread3runEv + 0x20f
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 39976 -----------------
0x00000033ad60b68c __pthread_cond_wait + 0xcc
0x00007f9d28e98917 _ZN7Monitor5IWaitEP6Threadl + 0xf7
0x00007f9d28e99236 _ZN7Monitor4waitEblb + 0x256
0x00007f9d28a83e1f _ZN24ConcurrentG1RefineThread3runEv + 0x20f
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 39977 -----------------
0x00000033ad60b68c __pthread_cond_wait + 0xcc
0x00007f9d28e98917 _ZN7Monitor5IWaitEP6Threadl + 0xf7
0x00007f9d28e99236 _ZN7Monitor4waitEblb + 0x256
0x00007f9d28aaf727 _ZN20ConcurrentMarkThread3runEv + 0xc7
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 39978 -----------------
0x00000033ad60b68c __pthread_cond_wait + 0xcc
0x00007f9d28e98917 _ZN7Monitor5IWaitEP6Threadl + 0xf7
0x00007f9d28e99236 _ZN7Monitor4waitEblb + 0x256
0x00007f9d2909a10b _ZN10GangWorker4loopEv + 0x5b
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 39987 -----------------
0x00000033ad60b68c __pthread_cond_wait + 0xcc
0x00007f9d28e98917 _ZN7Monitor5IWaitEP6Threadl + 0xf7
0x00007f9d28e99236 _ZN7Monitor4waitEblb + 0x256
0x00007f9d28b8e6cb _ZN18G1StringDedupQueue4waitEv + 0x5b
0x00007f9d28b92252 _ZN19G1StringDedupThread3runEv + 0x62
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 40000 -----------------
0x00000033ad60ba5e __pthread_cond_timedwait + 0x13e
0x00007f9d28e98c6e _ZN7Monitor5IWaitEP6Threadl + 0x44e
0x00007f9d28e99236 _ZN7Monitor4waitEblb + 0x256
0x00007f9d29082289 _ZN8VMThread4loopEv + 0x339
0x00007f9d29082590 _ZN8VMThread3runEv + 0x70
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 40003 -----------------
0x00000033ad60b68c __pthread_cond_wait + 0xcc
0x00007f9d28ec9b05 _ZN13ObjectMonitor4waitElbP6Thread + 0xa65
0x00007f9d28cea345 JVM_MonitorWait + 0x1a5
0x00007f9d149c0a14 * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0x00007f9d149b2c4d * java.lang.Object.wait() bci:2 line:502 (Interpreted frame)
0x00007f9d149b2c4d * java.lang.ref.Reference$ReferenceHandler.run() bci:36 line:157 (Interpreted frame)
0x00007f9d149ab7a7 <StubRoutines>
0x00007f9d28c4fa26 _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x1056
0x00007f9d28c4ff31 _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_P17JavaCallArgumentsP6Thread + 0x321
0x00007f9d28c503d7 _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandleP6SymbolS5_P6Thread + 0x47
0x00007f9d28ce6280 _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
0x00007f9d2902a86f _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007f9d2902a99c _ZN10JavaThread3runEv + 0x11c
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 40004 -----------------
0x00000033ad60b68c __pthread_cond_wait + 0xcc
0x00007f9d28ec9b05 _ZN13ObjectMonitor4waitElbP6Thread + 0xa65
0x00007f9d28cea345 JVM_MonitorWait + 0x1a5
0x00007f9d149c0a14 * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0x00007f9d151037f4 * java.lang.ref.ReferenceQueue.remove(long) bci:59 line:143 (Compiled frame)
* java.lang.ref.ReferenceQueue.remove() bci:2 line:164 (Compiled frame)
* java.lang.ref.Finalizer$FinalizerThread.run() bci:36 line:209 (Interpreted frame)
0x00007f9d149ab7a7 <StubRoutines>
0x00007f9d28c4fa26 _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x1056
0x00007f9d28c4ff31 _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_P17JavaCallArgumentsP6Thread + 0x321
0x00007f9d28c503d7 _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandleP6SymbolS5_P6Thread + 0x47
0x00007f9d28ce6280 _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
0x00007f9d2902a86f _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007f9d2902a99c _ZN10JavaThread3runEv + 0x11c
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 40023 -----------------
0x00000033ad60b68c __pthread_cond_wait + 0xcc
0x00007f9d28e98917 _ZN7Monitor5IWaitEP6Threadl + 0xf7
0x00007f9d28e992b6 _ZN7Monitor4waitEblb + 0x2d6
0x00007f9d28a841de _ZN21SurrogateLockerThread4loopEv + 0x4e
0x00007f9d2902a86f _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007f9d2902a99c _ZN10JavaThread3runEv + 0x11c
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 40024 -----------------
0x00000033ad60da00 sem_wait + 0x30
0x00007f9d28ed4ae3 _ZL19signal_thread_entryP10JavaThreadP6Thread + 0x73
0x00007f9d2902a86f _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007f9d2902a99c _ZN10JavaThread3runEv + 0x11c
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 40025 -----------------
0x00000033ad60ba5e __pthread_cond_timedwait + 0x13e
0x00007f9d28e98c6e _ZN7Monitor5IWaitEP6Threadl + 0x44e
0x00007f9d28e992b6 _ZN7Monitor4waitEblb + 0x2d6
0x00007f9d28a74d4b _ZN12CompileQueue3getEv + 0x13b
0x00007f9d28a76f9b _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
0x00007f9d2902a86f _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007f9d2902a99c _ZN10JavaThread3runEv + 0x11c
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 40026 -----------------
0x00000033ad60ba5e __pthread_cond_timedwait + 0x13e
0x00007f9d28e98c6e _ZN7Monitor5IWaitEP6Threadl + 0x44e
0x00007f9d28e992b6 _ZN7Monitor4waitEblb + 0x2d6
0x00007f9d28a74d4b _ZN12CompileQueue3getEv + 0x13b
0x00007f9d28a76f9b _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
0x00007f9d2902a86f _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007f9d2902a99c _ZN10JavaThread3runEv + 0x11c
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 40027 -----------------
0x00000033ad60ba5e __pthread_cond_timedwait + 0x13e
0x00007f9d28e98c6e _ZN7Monitor5IWaitEP6Threadl + 0x44e
0x00007f9d28e992b6 _ZN7Monitor4waitEblb + 0x2d6
0x00007f9d28a74d4b _ZN12CompileQueue3getEv + 0x13b
0x00007f9d28a76f9b _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
0x00007f9d2902a86f _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007f9d2902a99c _ZN10JavaThread3runEv + 0x11c
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 40028 -----------------
0x00000033ad60ba5e __pthread_cond_timedwait + 0x13e
0x00007f9d28e98c6e _ZN7Monitor5IWaitEP6Threadl + 0x44e
0x00007f9d28e992b6 _ZN7Monitor4waitEblb + 0x2d6
0x00007f9d28a74d4b _ZN12CompileQueue3getEv + 0x13b
0x00007f9d28a76f9b _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
0x00007f9d2902a86f _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007f9d2902a99c _ZN10JavaThread3runEv + 0x11c
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 40029 -----------------
0x00000033ad60ba5e __pthread_cond_timedwait + 0x13e
0x00007f9d28e98c6e _ZN7Monitor5IWaitEP6Threadl + 0x44e
0x00007f9d28e992b6 _ZN7Monitor4waitEblb + 0x2d6
0x00007f9d28a74d4b _ZN12CompileQueue3getEv + 0x13b
0x00007f9d28a76f9b _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
0x00007f9d2902a86f _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007f9d2902a99c _ZN10JavaThread3runEv + 0x11c
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 40030 -----------------
0x00000033ad60ba5e __pthread_cond_timedwait + 0x13e
0x00007f9d28e98c6e _ZN7Monitor5IWaitEP6Threadl + 0x44e
0x00007f9d28e992b6 _ZN7Monitor4waitEblb + 0x2d6
0x00007f9d28a74d4b _ZN12CompileQueue3getEv + 0x13b
0x00007f9d28a76f9b _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
0x00007f9d2902a86f _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007f9d2902a99c _ZN10JavaThread3runEv + 0x11c
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 40031 -----------------
0x00000033ad60ba5e __pthread_cond_timedwait + 0x13e
0x00007f9d28e98c6e _ZN7Monitor5IWaitEP6Threadl + 0x44e
0x00007f9d28e992b6 _ZN7Monitor4waitEblb + 0x2d6
0x00007f9d28a74d4b _ZN12CompileQueue3getEv + 0x13b
0x00007f9d28a76f9b _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
0x00007f9d2902a86f _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007f9d2902a99c _ZN10JavaThread3runEv + 0x11c
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 40032 -----------------
0x00000033ad60ba5e __pthread_cond_timedwait + 0x13e
0x00007f9d28e98c6e _ZN7Monitor5IWaitEP6Threadl + 0x44e
0x00007f9d28e992b6 _ZN7Monitor4waitEblb + 0x2d6
0x00007f9d28a74d4b _ZN12CompileQueue3getEv + 0x13b
0x00007f9d28a76f9b _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
0x00007f9d2902a86f _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007f9d2902a99c _ZN10JavaThread3runEv + 0x11c
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 40034 -----------------
0x00000033ad60ba5e __pthread_cond_timedwait + 0x13e
0x00007f9d28e98c6e _ZN7Monitor5IWaitEP6Threadl + 0x44e
0x00007f9d28e992b6 _ZN7Monitor4waitEblb + 0x2d6
0x00007f9d28a74d4b _ZN12CompileQueue3getEv + 0x13b
0x00007f9d28a76f9b _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
0x00007f9d2902a86f _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007f9d2902a99c _ZN10JavaThread3runEv + 0x11c
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 40035 -----------------
0x00000033ad60ba5e __pthread_cond_timedwait + 0x13e
0x00007f9d28e98c6e _ZN7Monitor5IWaitEP6Threadl + 0x44e
0x00007f9d28e992b6 _ZN7Monitor4waitEblb + 0x2d6
0x00007f9d28a74d4b _ZN12CompileQueue3getEv + 0x13b
0x00007f9d28a76f9b _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
0x00007f9d2902a86f _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007f9d2902a99c _ZN10JavaThread3runEv + 0x11c
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 40036 -----------------
0x00000033ad60ba5e __pthread_cond_timedwait + 0x13e
0x00007f9d28e98c6e _ZN7Monitor5IWaitEP6Threadl + 0x44e
0x00007f9d28e992b6 _ZN7Monitor4waitEblb + 0x2d6
0x00007f9d28a74d4b _ZN12CompileQueue3getEv + 0x13b
0x00007f9d28a76f9b _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
0x00007f9d2902a86f _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007f9d2902a99c _ZN10JavaThread3runEv + 0x11c
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 40038 -----------------
0x00000033ad60ba5e __pthread_cond_timedwait + 0x13e
0x00007f9d28e98c6e _ZN7Monitor5IWaitEP6Threadl + 0x44e
0x00007f9d28e992b6 _ZN7Monitor4waitEblb + 0x2d6
0x00007f9d28a74d4b _ZN12CompileQueue3getEv + 0x13b
0x00007f9d28a76f9b _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
0x00007f9d2902a86f _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007f9d2902a99c _ZN10JavaThread3runEv + 0x11c
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 40039 -----------------
0x00000033ad60ba5e __pthread_cond_timedwait + 0x13e
0x00007f9d28e98c6e _ZN7Monitor5IWaitEP6Threadl + 0x44e
0x00007f9d28e992b6 _ZN7Monitor4waitEblb + 0x2d6
0x00007f9d28a74d4b _ZN12CompileQueue3getEv + 0x13b
0x00007f9d28a76f9b _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
0x00007f9d2902a86f _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007f9d2902a99c _ZN10JavaThread3runEv + 0x11c
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 40040 -----------------
0x00000033ad60ba5e __pthread_cond_timedwait + 0x13e
0x00007f9d28e98c6e _ZN7Monitor5IWaitEP6Threadl + 0x44e
0x00007f9d28e992b6 _ZN7Monitor4waitEblb + 0x2d6
0x00007f9d28a74d4b _ZN12CompileQueue3getEv + 0x13b
0x00007f9d28a76f9b _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
0x00007f9d2902a86f _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007f9d2902a99c _ZN10JavaThread3runEv + 0x11c
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 40041 -----------------
0x00000033ad60ba5e __pthread_cond_timedwait + 0x13e
0x00007f9d28e98c6e _ZN7Monitor5IWaitEP6Threadl + 0x44e
0x00007f9d28e992b6 _ZN7Monitor4waitEblb + 0x2d6
0x00007f9d28a74d4b _ZN12CompileQueue3getEv + 0x13b
0x00007f9d28a76f9b _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
0x00007f9d2902a86f _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007f9d2902a99c _ZN10JavaThread3runEv + 0x11c
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 40042 -----------------
0x00000033ad60b68c __pthread_cond_wait + 0xcc
0x00007f9d28e98917 _ZN7Monitor5IWaitEP6Threadl + 0xf7
0x00007f9d28e99236 _ZN7Monitor4waitEblb + 0x256
0x00007f9d28f76028 _ZN13ServiceThread20service_thread_entryEP10JavaThreadP6Thread + 0x398
0x00007f9d2902a86f _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007f9d2902a99c _ZN10JavaThread3runEv + 0x11c
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 40044 -----------------
0x00000033ad60ba5e __pthread_cond_timedwait + 0x13e
0x00007f9d28e98c6e _ZN7Monitor5IWaitEP6Threadl + 0x44e
0x00007f9d28e99236 _ZN7Monitor4waitEblb + 0x256
0x00007f9d29025121 _ZNK13WatcherThread5sleepEv + 0x71
0x00007f9d290255ce _ZN13WatcherThread3runEv + 0x5e
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 23753 -----------------
0x00000033ad60eb2d __libc_accept + 0x2d
0x00007f9d288e695e _ZN14AttachListener7dequeueEv + 0x9e
0x00007f9d288e586a _ZL28attach_listener_thread_entryP10JavaThreadP6Thread + 0x19a
0x00007f9d2902a86f _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007f9d2902a99c _ZN10JavaThread3runEv + 0x11c
0x00007f9d28edede8 _ZL10java_startP6Thread + 0x108
----------------- 39966 -----------------
0x00000033ad6082fd pthread_join + 0x10d



REPRODUCIBILITY :
This bug can be reproduced rarely.