FULL PRODUCT VERSION :
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)

(but also existed in 1.8.0_77)

ADDITIONAL OS VERSION INFORMATION :
on my production node: 2.6.18-274.el5 #1 SMP Fri Jul 8 17:36:59 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
on my laptop: 16.0.0 Darwin Kernel Version 16.0.0: Mon Aug 29 17:56:20 PDT 2016; root:xnu-3789.1.32~3/RELEASE_X86_64 x86_64

EXTRA RELEVANT SYSTEM CONFIGURATION :
Not related to this issue.

A DESCRIPTION OF THE PROBLEM :
Hi:

My issue's phenomenon is: on one of me production node, all of the serialization and deserialization procedures are failed due to an "java.lang.OutOfMemoryError: GC overhead limit exceeded" error, while at that time of my system, no Full GC occurred at that time and have plenty of memory on all generations, see the output below using 'jstat -gcutil' command:

  S0 S1 E O P YGC YGCT FGC FGCT GCT
  0.00 5.36 50.07 62.14 18.25 1496 50.143 147 1048.493 1098.636
  0.00 5.36 50.57 62.14 18.25 1496 50.143 147 1048.493 1098.636

Another wired thing is this error is ONLY triggered while doing serialization/deserialization for class "java.util.LinkedList", and the error is came from method java.io.ObjectStreamClass.lookup(Class<?> cl, boolean all), after have a glance on the code, I found ObjectStreamClass is cached the previous throwable and just "re-throw" it on next lookup, see the code below:

    static ObjectStreamClass lookup(Class<?> cl, boolean all) {
....
// line 322 - 327
        WeakClassKey key = new WeakClassKey(cl, Caches.localDescsQueue);
        Reference<?> ref = Caches.localDescs.get(key);
        Object entry = null;
        if (ref != null) {
            entry = ref.get();
        }
....
// line 346 - 348
        if (entry instanceof ObjectStreamClass) { // will return on if the entry is an instance of ObjectStreamClass
            return (ObjectStreamClass) entry;
        }
.....
// line 363 - 375
        if (entry == null) { // only re-init an instance if the entry is null
            try {
                entry = new ObjectStreamClass(cl); // if an error happened here
            } catch (Throwable th) {
                entry = th; // treat the throwable as an normal "entry"
            }
            if (future.set(entry)) { // and cache it
                Caches.localDescs.put(key, new SoftReference<Object>(entry));
            } else {
                // nested lookup call already set future
                entry = future.get();
            }
        }

// line 377 - ...
        if (entry instanceof ObjectStreamClass) {
            return (ObjectStreamClass) entry;
        } else if (entry instanceof RuntimeException) {
            throw (RuntimeException) entry; // re-throw any RuntimeException
        } else if (entry instanceof Error) {
            throw (Error) entry; // re-throw any Error, including OutOfMemoryError
        } else {
            throw new InternalError("unexpected entry: " + entry);
        }
    }


This could be rarely happened, because the Error could happened exactly while initiating the ObjectStreamClass, but the only workaround is rebooting the JVM once it happened, although from the logic above that could recovered while all the soft-reference are collected during next FullGC, but that's an unpredictable JVM behavior.

Please advice ... Thank you

  - Thomas Li

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
According to the code I pasted, I didn't found the way to re-produce that at every time I run it, but the code will cache all other RuntimeException as well, I can reproduce it during debugging mode by throw an exception manually while constructing an instance of ObjectStreamClass, and after that, the exception will be cached and just re-throw it while doing the same class's lookup, instead of re-initiating a newly legal instance.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
OutOfMemoryError should not be thrown while there is enough memory.
ACTUAL -
The JVM is always re-throw the cached OutOfMemoryError

REPRODUCIBILITY :
This bug can be reproduced rarely.

CUSTOMER SUBMITTED WORKAROUND :
Rebooting the JVM