Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8046246

the constantPoolCacheOopDesc::adjust_method_entries() used in RedefineClasses does not scale



    • Bug
    • Resolution: Fixed
    • P4
    • 9
    • 7, 8u60, 9
    • hotspot
    • b55



        The issue was reported by the HP.

        Please, see the email from the support engineer Kevin Brown:

        Hi Serguei,

        I have an issue from our source licensee HP, and while it's not reproducible on Solaris/Linux I'm trying to advise HP on how to further debug it on HPUX/PaRISC.

        Vladimir indicated the method in question "adjust_cpool_cache_and_vtable()" was a JVMTI method and I hear you are an expert in this area. Could you review the data below and let me know if you have any advise?


        -------- Original Message --------
        Subject: Re: HP's high CPU utilization investigation...
        Date: Thu, 08 May 2014 17:52:59 -0700
        From: Vladimir Kozlov <vladimir.kozlov@oracle.com>
        To: Kevin Brown <Kevin.L.Brown@oracle.com>

        Hi Kevin,

        The bug I mentioned to you:


        but it looks like it is not related to your problem.

        > Question: By design, is re-compilation(i.e. compilation while reloading
        > instrumented classes) so very CPU intensive (~100%) Or is it expected
        > for large no. of classes ?

        When class is redefined we throw away (deoptimize) all related compiled
        methods. If new methods are hot they will be recompiled again after
        executed in Interpreter.

        The method adjust_cpool_cache_and_vtable() is not JIT compiler related.
        This method is in jvmti code. Ask serviceability group about it.

        That code was significantly changed since jdk7.


        On 5/8/14 5:20 PM, Kevin Brown wrote:
        > Hi Vladimir,
        > Thanks for the quick chat and willingness to look at this. I was hoping
        > you might have some insight or could at least better direct me on
        > investigating a high CPU usage issue that HP is reporting. Their
        > customer reported the issue initially with Weblogic on HPUX, and HP's
        > testcase without Weblogic (attached) successfully reproduces the problem.
        > *But* it is only reproduciable on HPUX. When I run the test on Solaris
        > or Linux, it does not show the same high CPU usage. (never gets above
        > 12-13%) I cannot formally file a bug since we don't see it our our
        > platforms, but HP is hoping we can provide guidance.
        > Do you have any suggestions on further debugging, VM options to tune
        > with, or areas to look into?
        > Here is my full analysis thus far...
        > Problem statement:
        > 100% CPU utilization is seen when the number of classes need to be
        > compiled are of the order of a thousand or more. Below are the numbers
        > that could be of interest to you when tested on HPUX.
        > o High CPU time: 8-10 mins
        > o No. of loaded classes : ~1600
        > o No. of redefined classes: 443
        > o Calls to evaluate_operation(): 658
        > What HP did on HP-UX:
        > HP could produce the above numbers through dynamically instrumenting an
        > in-house application that they generally use for testing. (the attached
        > testcase)
        > JLE testing on Solaris and Linux:
        > I (Kevin) was unable to reproduce the results on Solaris or Linux. I
        > tested with JDK 7u51 on Linux x64, connecting JConsole to badapp.jar
        > file. I do not see similar results, after 30 minutes of testing, the
        > CPU usage never goes over 12.9%
        > I suggested HP to try setting -XX:CompileThreshold to a lower setting
        > but they claim this delays the high CPU usage since compilation
        > eventually happens and goes to 100%.
        > JVM Call hierarchy:
        > Here are the functions called from jvm, the
        > adjust_cpool_cache_and_vtable() is the busiest method consuming the
        > highest CPU.
        > VMThread::run()
        > ….
        > VMThread::loop()
        > ...
        > evaluate_operations()
        > ..
        > evaluate()
        > ...
        > doit()
        > ...
        > redefine_single_Class()
        > ....
        > classes_do()
        > ....
        > adjust_cpool_cache_and_vtable()
        > ….
        > Question: By design, is re-compilation(i.e. compilation while reloading
        > instrumented classes) so very CPU intensive (~100%) Or is it expected
        > for large no. of classes ?
        > Thanks,
        > Kevin

        As we can see the HP reported a performance problem in the VM_RedefineClasses::adjust_cpool_cache_and_vtable().
        Also, the performance profiling information (see attached file HP-high_cpu_caliper-report.zip received from HP)
        shows the most overhead is in the constantPoolCacheOopDesc::adjust_method_entries().

        Look at the following fragment:
        Function Summary
        % Total Cumulat Secs Msecs
           IP % of IP in Call per
        Samples Total Samples Func Count Call Function File
          19.52 19.52 65739 20.59 62723 0.33 libjvm.so::VM_RedefineClasses::adjust_cpool_cache_and_vtable(klassOopDesc*,oopDesc*,Thread*) jvmtiRedefineClasses.cpp
          15.21 34.73 51230 16.05 27027 0.59 libjvm.so::RelocIterator::initialize(nmethod*,unsigned char*,unsigned char*) relocInfo.cpp
          14.52 49.25 48907 15.32 44635 0.34 libjvm.so::constantPoolCacheOopDesc::adjust_method_entries(methodOopDesc**,methodOopDesc**,int,bool*) cpCacheOop.cpp


          Issue Links



                sspitsyn Serguei Spitsyn
                sspitsyn Serguei Spitsyn
                0 Vote for this issue
                5 Start watching this issue