FULL PRODUCT VERSION :
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

java version "1.6.0_30"
Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)

Confirmed that it exists in several other versions by examining Java source code, including source from http://download.java.net/openjdk/jdk7u4/promoted/b22/openjdk-7u4-fcs-src-b22-02_may_2012.zip


ADDITIONAL OS VERSION INFORMATION :
Unlikely to matter, but:
Linux kgrittn-workstation 2.6.38-15-generic #65-Ubuntu SMP Thu Jul 26 20:13:19 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux


A DESCRIPTION OF THE PROBLEM :
Apparently endless loop (has gone on for days at time) with stack trace stuck on this and not changing:

State: RUNNABLE
  Total blocked: 4,644 Total waited: 8,542

Stack trace:
 java.io.ObjectOutputStream$HandleTable.lookup(ObjectOutputStream.java:2264)
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1096)
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518)
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483)
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400)
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
us.wi.state.courts.jade.jms.RemoteQueueServiceConnection.processOutput(RemoteQueueServiceConnection.java:412)
us.wi.state.courts.jade.jms.RemoteQueueServiceConnection$SocketOutputRunner.run(RemoteQueueServiceConnection.java:42)
java.lang.Thread.run(Thread.java:662)

Examination of the method shows that it is vulnerable to endless looping based on object hash values. This could be fixed by filling unused "next" array entries with -1 in clear() and growEntries() methods. There are probably other ways to fix it, but this seems like the most simple and straightforward.


REGRESSION. Last worked in version 5.0

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Send a lot of complex objects over an ObjectOutputStream, invoking the reset() method after sending each top-level object. It probably happens with a frequency of less that once each million top level objects, but with our volume that makes it a recurring problem, disrupting production applications.


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
java.io.ObjectOutputStream.HandleTable.lookup() eventually returns from each call.

ACTUAL -
java.io.ObjectOutputStream.HandleTable.lookup() sometimes goes into a hard loop, consuming all CPU time for one core as "user" time, never returning.


REPRODUCIBILITY :
This bug can be reproduced rarely.

CUSTOMER SUBMITTED WORKAROUND :
We are using a modified class which initializes "next" array entries.