the constantPoolCacheOopDesc::adjust_method_entries() used in RedefineClasses does not scale

The issue was reported by the HP.

Please, see the email from the support engineer Kevin Brown:

Hi Serguei,

I have an issue from our source licensee HP, and while it's not reproducible on Solaris/Linux I'm trying to advise HP on how to further debug it on HPUX/PaRISC.

Vladimir indicated the method in question "adjust_cpool_cache_and_vtable()" was a JVMTI method and I hear you are an expert in this area. Could you review the data below and let me know if you have any advise?

Thanks,
Kevin

-------- Original Message --------
Subject: Re: HP's high CPU utilization investigation...
Date: Thu, 08 May 2014 17:52:59 -0700
From: Vladimir Kozlov <vladimir.kozlov@oracle.com>
To: Kevin Brown <Kevin.L.Brown@oracle.com>


Hi Kevin,

The bug I mentioned to you:

https://bugs.openjdk.java.net/browse/JDK-8041984

but it looks like it is not related to your problem.

> Question: By design, is re-compilation(i.e. compilation while reloading
> instrumented classes) so very CPU intensive (~100%) Or is it expected
> for large no. of classes ?

When class is redefined we throw away (deoptimize) all related compiled
methods. If new methods are hot they will be recompiled again after
executed in Interpreter.

The method adjust_cpool_cache_and_vtable() is not JIT compiler related.
This method is in jvmti code. Ask serviceability group about it.

That code was significantly changed since jdk7.

Regards,
Vladimir

On 5/8/14 5:20 PM, Kevin Brown wrote:
> Hi Vladimir,
>
> Thanks for the quick chat and willingness to look at this. I was hoping
> you might have some insight or could at least better direct me on
> investigating a high CPU usage issue that HP is reporting. Their
> customer reported the issue initially with Weblogic on HPUX, and HP's
> testcase without Weblogic (attached) successfully reproduces the problem.
>
> *But* it is only reproduciable on HPUX. When I run the test on Solaris
> or Linux, it does not show the same high CPU usage. (never gets above
> 12-13%) I cannot formally file a bug since we don't see it our our
> platforms, but HP is hoping we can provide guidance.
>
> Do you have any suggestions on further debugging, VM options to tune
> with, or areas to look into?
>
> Here is my full analysis thus far...
>
> Problem statement:
> 100% CPU utilization is seen when the number of classes need to be
> compiled are of the order of a thousand or more. Below are the numbers
> that could be of interest to you when tested on HPUX.
>
> o High CPU time: 8-10 mins
> o No. of loaded classes : ~1600
> o No. of redefined classes: 443
> o Calls to evaluate_operation(): 658
>
> What HP did on HP-UX:
> HP could produce the above numbers through dynamically instrumenting an
> in-house application that they generally use for testing. (the attached
> testcase)
>
> JLE testing on Solaris and Linux:
> I (Kevin) was unable to reproduce the results on Solaris or Linux. I
> tested with JDK 7u51 on Linux x64, connecting JConsole to badapp.jar
> file. I do not see similar results, after 30 minutes of testing, the
> CPU usage never goes over 12.9%
>
> I suggested HP to try setting -XX:CompileThreshold to a lower setting
> but they claim this delays the high CPU usage since compilation
> eventually happens and goes to 100%.
>
> JVM Call hierarchy:
> Here are the functions called from jvm, the
> adjust_cpool_cache_and_vtable() is the busiest method consuming the
> highest CPU.
>
> VMThread::run()
> ….
> VMThread::loop()
> ...
> evaluate_operations()
> ..
> evaluate()
> ...
> doit()
> ...
> redefine_single_Class()
> ....
> classes_do()
> ....
> adjust_cpool_cache_and_vtable()
> ….
>
> Question: By design, is re-compilation(i.e. compilation while reloading
> instrumented classes) so very CPU intensive (~100%) Or is it expected
> for large no. of classes ?
>
> Thanks,
> Kevin

As we can see the HP reported a performance problem in the VM_RedefineClasses::adjust_cpool_cache_and_vtable().
Also, the performance profiling information (see attached file HP-high_cpu_caliper-report.zip received from HP)
shows the most overhead is in the constantPoolCacheOopDesc::adjust_method_entries().

Look at the following fragment:
Function Summary
---------------------------------------------------------------------------------------------------------
% Total Cumulat Secs Msecs
   IP % of IP in Call per
Samples Total Samples Func Count Call Function File
---------------------------------------------------------------------------------------------------------
  19.52 19.52 65739 20.59 62723 0.33 libjvm.so::VM_RedefineClasses::adjust_cpool_cache_and_vtable(klassOopDesc*,oopDesc*,Thread*) jvmtiRedefineClasses.cpp
  15.21 34.73 51230 16.05 27027 0.59 libjvm.so::RelocIterator::initialize(nmethod*,unsigned char*,unsigned char*) relocInfo.cpp
  14.52 49.25 48907 15.32 44635 0.34 libjvm.so::constantPoolCacheOopDesc::adjust_method_entries(methodOopDesc**,methodOopDesc**,int,bool*) cpCacheOop.cpp