We have a customer who is looking to deploy an application on several
hundred T2000 machines. Unfortunately, during load testing we found the
performance of the WebLogic SIP Server to be pathetic on that hardware.
We traced the issue down to a bottleneck that occurs in the JDK 1.5.0_6
through _11. A summary of the issue is below. The bottleneck was not an
issue on x86 hardware or other non-Niagra based Sparc architectures.

The issue we have run into is heavy contention on
java.lang.ref.ReferenceQueue.lock during object serialization on Solaris
10 SunOS 5.10 Generic_118833-17 sun4v sparc SUNW,Sun-Fire-T200, Java(TM)
2 Runtime Environment, Standard Edition (build 1.5.0_06-b05) Java
HotSpot(TM) Server VM (build 1.5.0_06-b05, mixed mode) and Sun T2000.
The following stack trace illustrates the contention; in our tests we
observe many threads waiting on the same semaphore :
 
"pool-1-thread-23" prio=10 tid=0x00282d18 nid=0x41 waiting for monitor
entry [0xb07fe000..0xb07ffaf0]
        at java.lang.ref.ReferenceQueue.poll(ReferenceQueue.java:82)
        - waiting to lock <0xe7569aa0> (a
java.lang.ref.ReferenceQueue$Lock)
        at
java.io.ObjectStreamClass.processQueue(ObjectStreamClass.java:2206)
        at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:253)
        at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1035)
        at
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
 
"pool-1-thread-29" prio=10 tid=0x002895a0 nid=0x47 waiting for monitor
entry [0xb01fe000..0xb01ffbf0]
        at java.lang.ref.ReferenceQueue.poll(ReferenceQueue.java:82)
        - waiting to lock <0xe7569aa0> (a
java.lang.ref.ReferenceQueue$Lock)
        at
java.io.ObjectStreamClass.processQueue(ObjectStreamClass.java:2206)
        at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:253)
        at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1035)
        at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:13
75)
        at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
        at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1
290)
        at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
        at
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)

Our analysis of the root cause revealed that the ObjectStreamClass
static Cache data members are the bottleneck.
 
  To help with troubleshooting the problem, we have created a reproducer
(attached) and a patch to ReferenceQueue.java (attached). While we are
not proposing the patch as a final solution, it helps illustrate the
problem. The reproducer requires WLS 9.2 to compile and run. The results
without the patch are:

bash-3.00$ ~/bea.sparc.30/jdk1.5.0_11/bin/java -server Serialize 32
120000 1000
Warming up with : [32] threads, for : [30000] ms, [1000] requests at a
time
Running with : [32] threads, for : [120000] ms, [1000] requests at a
time
Throghuput : [6.0] (ser-deSer / ms)

With the patch we can fully utilize the CPU and we see over 3x
improvement in throughput :

bash-3.00$ ~/bea.sparc.30/jdk1.5.0_11/bin/java
-Xbootclasspath/p:/home/jwilkiew/temp/relock.jar -server Serialize 32
120000 1000
Warming up with : [32] threads, for : [30000] ms, [1000] requests at a
time
Running with : [32] threads, for : [120000] ms, [1000] requests at a
time
Throghuput : [21.0] (ser-deSer / ms)
workaround: To help with troubleshooting the problem, we have created a
reproducer (attached) and a patch to ReferenceQueue.java (attached).
While we are not proposing the patch as a final solution, it helps
illustrate the problem.