Details
-
Bug
-
Resolution: Cannot Reproduce
-
P4
-
None
-
6u25
-
x86
-
linux_redhat_5.0
Description
FULL PRODUCT VERSION :
1.6.0_25-b06
ADDITIONAL OS VERSION INFORMATION :
Linux x86-64
2.6.18-194.32.1.el5
A DESCRIPTION OF THE PROBLEM :
I am raising this bug as advised by an Oracle Support Engineer in SR 3-6014003521.
We have only seen once incidence of this problem but, based on our heap dump analysis, it definitely looks like a bug. I accept that it would be hard to fix this problem based on the details below, but I think it is worth raising it anyway for the sake of visibility.
We had a deadlock-like failure of our application recently. We are trying to find the root cause.
I initially reported it on the BDB JE forum (https://forums.oracle.com/forums/thread.jspa?messageID=10480988) but further analysis of the heap and thread dumps has exposed a problem that looks like a Java bug.
We’re using Oracle JVM 1.6.0_25-b06, running on Linux version: 2.6.18-194.32.1.el5.
Thread t@41101 was blocked indefinitely in ReentrantReadWriteLock.writeLock().lock().
We know from code inspection that nothing ever takes a read lock on this ReentrantReadWriteLock, so started trying to find out what has got its write lock.
The output of "jstack -l" should list which thread holds this exclusive lock in the "locked ownable synchronizers" section but does not.
Our first theory was that the owning thread might have terminated.
We wrote a simple test program to explore this. We found from heap dump analysis that even if the owning thread terminates, the lock itself still refers to it via the ReentrantReadWriteLock.WriteLock.sync.exclusiveOwnerThread field. Looking in the java.util.concurrent source code, it seems that this field only gets null'ed when the lock is released.
However, looking in the heap dump taken following our "deadlock", we were surprised to find that the lock in question has a null sync.exclusiveOwnerThread field.
Surely a write lock should be in one of two states (except possibly for a tiny instant when its state is being non-atomically switched):
1) The lock is available, and sync.exclusiveOwnerThread is null
2) The lock is unavailable, and sync.exclusiveOwnerThread is populated
But our lock was indefinitely in this state:
3) The lock is unavailable and sync.exclusiveOwnerThread is null
Note that I also asked this question on the Concurrency Interest list, and got a response from Doug Lea confirming that it looks like a VM, OS, or processor bug:
http://cs.oswego.edu/pipermail/concurrency-interest/2012-August/009635.html
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Not known
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
1) The lock is available, and sync.exclusiveOwnerThread is null
2) The lock is unavailable, and sync.exclusiveOwnerThread is populated
ACTUAL -
The lock is unavailable and sync.exclusiveOwnerThread is null
REPRODUCIBILITY :
This bug can be reproduced rarely.
---------- BEGIN SOURCE ----------
not available
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
not known
1.6.0_25-b06
ADDITIONAL OS VERSION INFORMATION :
Linux x86-64
2.6.18-194.32.1.el5
A DESCRIPTION OF THE PROBLEM :
I am raising this bug as advised by an Oracle Support Engineer in SR 3-6014003521.
We have only seen once incidence of this problem but, based on our heap dump analysis, it definitely looks like a bug. I accept that it would be hard to fix this problem based on the details below, but I think it is worth raising it anyway for the sake of visibility.
We had a deadlock-like failure of our application recently. We are trying to find the root cause.
I initially reported it on the BDB JE forum (https://forums.oracle.com/forums/thread.jspa?messageID=10480988) but further analysis of the heap and thread dumps has exposed a problem that looks like a Java bug.
We’re using Oracle JVM 1.6.0_25-b06, running on Linux version: 2.6.18-194.32.1.el5.
Thread t@41101 was blocked indefinitely in ReentrantReadWriteLock.writeLock().lock().
We know from code inspection that nothing ever takes a read lock on this ReentrantReadWriteLock, so started trying to find out what has got its write lock.
The output of "jstack -l" should list which thread holds this exclusive lock in the "locked ownable synchronizers" section but does not.
Our first theory was that the owning thread might have terminated.
We wrote a simple test program to explore this. We found from heap dump analysis that even if the owning thread terminates, the lock itself still refers to it via the ReentrantReadWriteLock.WriteLock.sync.exclusiveOwnerThread field. Looking in the java.util.concurrent source code, it seems that this field only gets null'ed when the lock is released.
However, looking in the heap dump taken following our "deadlock", we were surprised to find that the lock in question has a null sync.exclusiveOwnerThread field.
Surely a write lock should be in one of two states (except possibly for a tiny instant when its state is being non-atomically switched):
1) The lock is available, and sync.exclusiveOwnerThread is null
2) The lock is unavailable, and sync.exclusiveOwnerThread is populated
But our lock was indefinitely in this state:
3) The lock is unavailable and sync.exclusiveOwnerThread is null
Note that I also asked this question on the Concurrency Interest list, and got a response from Doug Lea confirming that it looks like a VM, OS, or processor bug:
http://cs.oswego.edu/pipermail/concurrency-interest/2012-August/009635.html
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Not known
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
1) The lock is available, and sync.exclusiveOwnerThread is null
2) The lock is unavailable, and sync.exclusiveOwnerThread is populated
ACTUAL -
The lock is unavailable and sync.exclusiveOwnerThread is null
REPRODUCIBILITY :
This bug can be reproduced rarely.
---------- BEGIN SOURCE ----------
not available
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
not known
Attachments
Issue Links
- relates to
-
JDK-6822370 ReentrantReadWriteLock: threads hung when there are no threads holding onto the lock (Netra x4450)
- Closed