Customer's app is not scaling well with double load causing about a 6 fold
increase in cpu as well as high thread contention which we can see in the
Java Flight Recorder(JFR)recording. See attached. Contention is due to
threads competing for the classloader.

The recording shows numerous blocked threads on classloader. We do observe
weblogic.utils.classloaders.ChangeAwareClassLoader (21K count) in the JFR
recording.
One of the highest caller is from com.sun.org.apache.xalan.internal.utils.

The customer Note we has already ruled out things like GC performance, heap
size, physical memory, etc. The GC logs, CA Introscope monitoring, and the
JFR recording all show GCs are not a concern. We also have the same issue
with a 10g heap in our labs (the attached JFR is from production with a 5g
heap).

The application is not scaling as it should with increased load. A doubling
of load causes several times more cpu usage, several times more threads, very
high amounts of contention (lots of blocked threads), and significant delays
in response times. Doubling the load has caused response times to increase
exponentially.

The attached JFR shows very high contention among threads. We are trying to
understand why this contention is happening.

We have 16 instances per server so 4% cpu per instance is a lot in this
scenario. We normally run at about 0 - 1% and don't expect a small increase
in load to put us at 4% cpu per instance. The issue gets far worse if load
gets higher and has caused a production outage before with 100% system cpu.

Our biggest issue is that response times for the app increase exponentially.
This is unacceptable and we cannot scale the application to handle upcoming
production load increases.

Check out at all the blocking in the JFR. It is really apparent when you look
at the Graph tab for Events. We are trying to understand why there is so much
contention as we do not see this behavior with lower load. It appears to be
related to the classloader but we don't understand why this is happening.