ReentrantReadWriteLock: threads hung when there are no threads holding onto the lock (Netra x4450)

Only happen on Hardware: x4450
OS = Solaris 10 (does not happen with Linux)
JDK = JDK 6.0 upd 10 and 12

Problem Description

Java threads are blocked waiting for a lock that is not held by
any threads ie. a lock that has already been released. thread is
seen waiting on this synchronizer -
- parking to wait for <0xfffffd72f0e32118> (a java.util.concurrent.lock
s.ReentrantReadWriteLock$NonfairSync)

Troubleshooting results are available to show that
the lock is not being held by any thread. See details in comments.

This hung issue is only happening on Netra x4450 but not
any other platforms such as x2200s, x4440s, x4600s, and Niagara-based systems like a 5440.

A testcase and more details are available in comments.
A similar problem was reported via the concurrency-interest mailing list and then on the core-libs dev list.

http://mail.openjdk.java.net/pipermail/core-libs-dev/2009-July/002040.html

The problem is not restricted to x4450 but shows up on 4-8way Intel systems. Same basic scenario - the application hangs with a bunch of threads all inside a LinkedBlockingQueue trying to acquire the internal ReentrantLock that nobody seems to own. Again -XX:+UseMembar avoids the problem.

I used the attached test program to reproduce the problem on an 8-way Intel machine that I have access to. However when I tried to probe deeper by using modified classes that allowed me to examine the internal lock state when the hang occurs, it ceased to occur.

Two different hangs are possible with the test program:
a) use of the application LinkedBlockingDeque
b) use of the ThreadPoolExecutors LinkedBlockingQueue

Thanks to Ariel Weisburg and Ryan Betts for reporting the problem and working to get a small reproducible test case.