FULL PRODUCT VERSION :
java version " 1.6.0_41 "
Java(TM) SE Runtime Environment (build 1.6.0_41-b02)
Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Linux balg1706 2.6.18-308.0.0.0.1.el5 #1 SMP Sat Feb 25 16:16:23 EST 2012 x86_64 x86_64 x86_64 GNU/Linux


A DESCRIPTION OF THE PROBLEM :
One thread entered in to a synchronized method of JAVA API code and trying to acquire a monitor which doesn?t exists. The other thread called same synchronized method and got the monitor. Hence they got stuck in there and never returned back to processing.

Both the threads are holding lock on the below object

private static final synchronized BaseCalendar getJulianCalendarSystem()
{
if (jcal == null) {
jcal = (JulianCalendar)CalendarSystem.forName( " julian " );
jeras = jcal.getEras();
}
return jcal;
}

The Java Doc, multiple threads cannot acquire lock on synchronized method but the below thread dumps shows the evidence of aquireing locks by two threads on the same monitor/object. The thread dumps for the locks are given below.


 " DefaultTaskProcessor-TPSK:{Identifier{CycleRunnableTaskProcessor},0,0x0A8C900F0000013FA3FF87C62AD494A0}:Thread-32 " daemon prio=10 tid=0x000000004e348000 nid=0x9c7 waiting for monitor entry [0x0000000053833000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.util.GregorianCalendar.getJulianCalendarSystem(GregorianCalendar.java:2642)
        - locked <0x0000000730531988> (a java.lang.Class for java.util.GregorianCalendar)
        at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2095)
        at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:1996)
        at java.util.Calendar.setTimeInMillis(Calendar.java:1110)
        at java.util.GregorianCalendar.add(GregorianCalendar.java:999)
        at com.adminserver.utl.DateMathHelperUtl.differenceByDays(DateMathHelperUtl.java:64)
        at com.adminserver.math.engine.CalculationFunctions.DaysDiffOf(CalculationFunctions.java:274)
        at com.adminserver.sre.bll.generatedClasses.MathEngine_25f23d40b37a7df3683fa8a69430c4b6.executeStatement_4$(MathEngine_25f23d40b37a7df3683fa8a69430c4b6:188)
        at com.adminserver.sre.bll.generatedClasses.MathEngine_25f23d40b37a7df3683fa8a69430c4b6.execute(MathEngine_25f23d40b37a7df3683fa8a69430c4b6:98)
        at com.adminserver.pas.bll.FundValuationBll.calculateRedemptionAmounts(FundValuationBll.java:427)
        at com.adminserver.pas.bll.PolicyValueBll.processRedemptionFee(PolicyValueBll.java:689)
        at com.adminserver.pas.bll.PolicyValueBll.processValuationCalculationOrder(PolicyValueBll.java:375)
        at com.adminserver.pas.bll.PolicyValueBll.value(PolicyValueBll.java:254)
        at com.adminserver.pas.bll.AbstractPolicyValuationActivityBll.value(AbstractPolicyValuationActivityBll.java:222)
        at com.adminserver.sre.bll.PasTransactionBll.value_aroundBody4(PasTransactionBll.java:653)
        at com.adminserver.sre.bll.PasTransactionBll$AjcClosure5.run(PasTransactionBll.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.adminserver.sre.aspects.BaseSreExtensibilityAspect.executeExtension(BaseSreExtensibilityAspect.java:112)
        at com.adminserver.sre.aspects.SreForwardProcessingExtensibilityAspect.ajc$superDispatch$com_adminserver_sre_aspects_SreForwardProcessingExtensibilityAspect$executeExtension(SreForwardProcessingExtensibilityAspect.java:1)
        at com.adminserver.sre.aspects.SreForwardProcessingExtensibilityAspect.instrumentPolicyValuationBll_value(SreForwardProcessingExtensibilityAspect.java:81)
        at com.adminserver.sre.bll.PasTransactionBll.processValuation(PasTransactionBll.java:653)
        at com.adminserver.sre.bll.PasTransactionBll.doValuation(PasTransactionBll.java:627)
        at com.adminserver.sre.bll.PolicyFinancialProcessorBll.doValuation_aroundBody0(PolicyFinancialProcessorBll.java:157)
        at com.adminserver.sre.bll.PolicyFinancialProcessorBll$AjcClosure1.run(PolicyFinancialProcessorBll.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.adminserver.sre.aspects.BaseSreExtensibilityAspect.executeExtension(BaseSreExtensibilityAspect.java:112)
        at com.adminserver.sre.aspects.SreForwardProcessingExtensibilityAspect.ajc$superDispatch$com_adminserver_sre_aspects_SreForwardProcessingExtensibilityAspect$executeExtension(SreForwardProcessingExtensibilityAspect.java:1)
        at com.adminserver.sre.aspects.SreForwardProcessingExtensibilityAspect.instrumentPolicyFinancialProcessorBll_doValuation(SreForwardProcessingExtensibilityAspect.java:75)
        at com.adminserver.sre.bll.PolicyFinancialProcessorBll.doValuation(PolicyFinancialProcessorBll.java:154)
        at com.adminserver.sre.bll.PolicyFinancialProcessorBll.process(PolicyFinancialProcessorBll.java:65)
        at com.adminserver.sre.bll.AbstractActivityProcessorBll.process(AbstractActivityProcessorBll.java:99)
        at com.adminserver.sre.bll.ActivityProcessorBll.processWithoutUndoRedoGeneration(ActivityProcessorBll.java:215)
        at com.adminserver.sre.bll.ActivityProcessorBll.process_aroundBody0(ActivityProcessorBll.java:98)
        at com.adminserver.sre.bll.ActivityProcessorBll$AjcClosure1.run(ActivityProcessorBll.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.adminserver.sre.aspects.BaseSreExtensibilityAspect.executeReturningExtension(BaseSreExtensibilityAspect.java:137)
        at com.adminserver.sre.aspects.SreForwardProcessingExtensibilityAspect.ajc$superDispatch$com_adminserver_sre_aspects_SreForwardProcessingExtensibilityAspect$executeReturningExtension(SreForwardProcessingExtensibilityAspect.java:1)
        at com.adminserver.sre.aspects.SreForwardProcessingExtensibilityAspect.instrumentActivityProcessorBll_process(SreForwardProcessingExtensibilityAspect.java:51)
        at com.adminserver.sre.bll.ActivityProcessorBll.process(ActivityProcessorBll.java:91)
        at com.adminserver.pas.bll.ActivityProcessHelper.processActivity(ActivityProcessHelper.java:131)
        at com.adminserver.pas.bll.ActivityBll.process_aroundBody0(ActivityBll.java:2415)
        at com.adminserver.pas.bll.ActivityBll$AjcClosure1.run(ActivityBll.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.adminserver.sre.aspects.BaseSreExtensibilityAspect.executeReturningExtension(BaseSreExtensibilityAspect.java:137)
        at com.adminserver.pas.bll.aspects.SreForwardProcessingExtensibilityAspect.ajc$superDispatch$com_adminserver_pas_bll_aspects_SreForwardProcessingExtensibilityAspect$executeReturningExtension(SreForwardProcessingExtensibilityAspect.java:1)
        at com.adminserver.pas.bll.aspects.SreForwardProcessingExtensibilityAspect.instrumentActivityBll_process(SreForwardProcessingExtensibilityAspect.java:30)
        at com.adminserver.pas.bll.ActivityBll.process(ActivityBll.java:2408)
        at com.adminserver.pas.bll.ActivityTaskExecutorBll.process(ActivityTaskExecutorBll.java:79)
        at com.adminserver.pas.bll.ActivityProcessHelper.processPendingActivitiesForCycle(ActivityProcessHelper.java:975)
        at com.adminserver.pas.bll.PolicyBll.processPendingActivitiesForCycle(PolicyBll.java:686)
        at com.adminserver.pas.cycle.bll.PasCycleBll.processCycleTask(PasCycleBll.java:162)
        at com.adminserver.cycle.agent.bll.tasks.CycleTask.run(CycleTask.java:50)
        at com.adminserver.cycle.agent.bll.tasks.CycleGroupTask.run(CycleGroupTask.java:62)
        at com.adminserver.global.processing.coherence.RunnableTaskWrapper.run(RunnableTaskWrapper.java:39)
        at com.oracle.coherence.patterns.processing.taskprocessor.TaskRunner.doExecuteTask(TaskRunner.java:301)
        at com.oracle.coherence.patterns.processing.taskprocessor.TaskRunner.run(TaskRunner.java:209)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:662)



 " DefaultTaskProcessor-TPSK:{Identifier{CycleRunnableTaskProcessor},0,0x0A8C900F0000013FA3FF87C62AD494A0}:Thread-29 " daemon prio=10 tid=0x000000004d66b800 nid=0x9bb runnable [0x0000000053530000]
   java.lang.Thread.State: RUNNABLE
        at java.util.GregorianCalendar.getJulianCalendarSystem(GregorianCalendar.java:2642)
        - locked <0x0000000730531988> (a java.lang.Class for java.util.GregorianCalendar)
        at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2095)
        at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:1996)
        at java.util.Calendar.setTimeInMillis(Calendar.java:1110)
        at java.util.GregorianCalendar.add(GregorianCalendar.java:999)
        at com.adminserver.utl.DateMathHelperUtl.differenceByDays(DateMathHelperUtl.java:64)
        at com.adminserver.math.engine.CalculationFunctions.DaysDiffOf(CalculationFunctions.java:274)
        at com.adminserver.sre.bll.generatedClasses.MathEngine_25f23d40b37a7df3683fa8a69430c4b6.executeStatement_4$(MathEngine_25f23d40b37a7df3683fa8a69430c4b6:188)
        at com.adminserver.sre.bll.generatedClasses.MathEngine_25f23d40b37a7df3683fa8a69430c4b6.execute(MathEngine_25f23d40b37a7df3683fa8a69430c4b6:98)
        at com.adminserver.pas.bll.FundValuationBll.calculateRedemptionAmounts(FundValuationBll.java:427)
        at com.adminserver.pas.bll.PolicyValueBll.processRedemptionFee(PolicyValueBll.java:689)
        at com.adminserver.pas.bll.PolicyValueBll.processValuationCalculationOrder(PolicyValueBll.java:375)
        at com.adminserver.pas.bll.PolicyValueBll.value(PolicyValueBll.java:254)
        at com.adminserver.pas.bll.AbstractPolicyValuationActivityBll.value(AbstractPolicyValuationActivityBll.java:222)
        at com.adminserver.sre.bll.PasTransactionBll.value_aroundBody4(PasTransactionBll.java:653)
        at com.adminserver.sre.bll.PasTransactionBll$AjcClosure5.run(PasTransactionBll.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.adminserver.sre.aspects.BaseSreExtensibilityAspect.executeExtension(BaseSreExtensibilityAspect.java:112)
        at com.adminserver.sre.aspects.SreForwardProcessingExtensibilityAspect.ajc$superDispatch$com_adminserver_sre_aspects_SreForwardProcessingExtensibilityAspect$executeExtension(SreForwardProcessingExtensibilityAspect.java:1)
        at com.adminserver.sre.aspects.SreForwardProcessingExtensibilityAspect.instrumentPolicyValuationBll_value(SreForwardProcessingExtensibilityAspect.java:81)
        at com.adminserver.sre.bll.PasTransactionBll.processValuation(PasTransactionBll.java:653)
        at com.adminserver.sre.bll.PasTransactionBll.doValuation(PasTransactionBll.java:627)
        at com.adminserver.sre.bll.PolicyFinancialProcessorBll.doValuation_aroundBody0(PolicyFinancialProcessorBll.java:157)
        at com.adminserver.sre.bll.PolicyFinancialProcessorBll$AjcClosure1.run(PolicyFinancialProcessorBll.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.adminserver.sre.aspects.BaseSreExtensibilityAspect.executeExtension(BaseSreExtensibilityAspect.java:112)
        at com.adminserver.sre.aspects.SreForwardProcessingExtensibilityAspect.ajc$superDispatch$com_adminserver_sre_aspects_SreForwardProcessingExtensibilityAspect$executeExtension(SreForwardProcessingExtensibilityAspect.java:1)
        at com.adminserver.sre.aspects.SreForwardProcessingExtensibilityAspect.instrumentPolicyFinancialProcessorBll_doValuation(SreForwardProcessingExtensibilityAspect.java:75)
        at com.adminserver.sre.bll.PolicyFinancialProcessorBll.doValuation(PolicyFinancialProcessorBll.java:154)
        at com.adminserver.sre.bll.PolicyFinancialProcessorBll.process(PolicyFinancialProcessorBll.java:65)
        at com.adminserver.sre.bll.AbstractActivityProcessorBll.process(AbstractActivityProcessorBll.java:99)
        at com.adminserver.sre.bll.ActivityProcessorBll.processWithoutUndoRedoGeneration(ActivityProcessorBll.java:215)
        at com.adminserver.sre.bll.ActivityProcessorBll.process_aroundBody0(ActivityProcessorBll.java:98)
        at com.adminserver.sre.bll.ActivityProcessorBll$AjcClosure1.run(ActivityProcessorBll.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.adminserver.sre.aspects.BaseSreExtensibilityAspect.executeReturningExtension(BaseSreExtensibilityAspect.java:137)
        at com.adminserver.sre.aspects.SreForwardProcessingExtensibilityAspect.ajc$superDispatch$com_adminserver_sre_aspects_SreForwardProcessingExtensibilityAspect$executeReturningExtension(SreForwardProcessingExtensibilityAspect.java:1)
        at com.adminserver.sre.aspects.SreForwardProcessingExtensibilityAspect.instrumentActivityProcessorBll_process(SreForwardProcessingExtensibilityAspect.java:51)
        at com.adminserver.sre.bll.ActivityProcessorBll.process(ActivityProcessorBll.java:91)
        at com.adminserver.pas.bll.ActivityProcessHelper.processActivity(ActivityProcessHelper.java:131)
        at com.adminserver.pas.bll.ActivityBll.process_aroundBody0(ActivityBll.java:2415)
        at com.adminserver.pas.bll.ActivityBll$AjcClosure1.run(ActivityBll.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.adminserver.sre.aspects.BaseSreExtensibilityAspect.executeReturningExtension(BaseSreExtensibilityAspect.java:137)
        at com.adminserver.pas.bll.aspects.SreForwardProcessingExtensibilityAspect.ajc$superDispatch$com_adminserver_pas_bll_aspects_SreForwardProcessingExtensibilityAspect$executeReturningExtension(SreForwardProcessingExtensibilityAspect.java:1)
        at com.adminserver.pas.bll.aspects.SreForwardProcessingExtensibilityAspect.instrumentActivityBll_process(SreForwardProcessingExtensibilityAspect.java:30)
        at com.adminserver.pas.bll.ActivityBll.process(ActivityBll.java:2408)
        at com.adminserver.pas.bll.ActivityTaskExecutorBll.process(ActivityTaskExecutorBll.java:79)
        at com.adminserver.pas.bll.ActivityProcessHelper.processPendingActivitiesForCycle(ActivityProcessHelper.java:975)
        at com.adminserver.pas.bll.PolicyBll.processPendingActivitiesForCycle(PolicyBll.java:686)
        at com.adminserver.pas.cycle.bll.PasCycleBll.processCycleTask(PasCycleBll.java:162)
        at com.adminserver.cycle.agent.bll.tasks.CycleTask.run(CycleTask.java:50)
        at com.adminserver.cycle.agent.bll.tasks.CycleGroupTask.run(CycleGroupTask.java:62)
        at com.adminserver.global.processing.coherence.RunnableTaskWrapper.run(RunnableTaskWrapper.java:39)
        at com.oracle.coherence.patterns.processing.taskprocessor.TaskRunner.doExecuteTask(TaskRunner.java:301)
        at com.oracle.coherence.patterns.processing.taskprocessor.TaskRunner.run(TaskRunner.java:209)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:662)



STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
with heavy load we can reproduce this bug in the our test environment.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The business transactions should be processed and no hanging or thread getting stuck should happen. Mulitple methods should not lock on a synchronized method which has been seen in the thread dumps given.
ACTUAL -
Threads are hanging and our business transactions are not processing associated with the thread.

REPRODUCIBILITY :
This bug can be reproduced often.

SUPPORT :
YES