JVMTI PopFrame capability must instruct compilers not to prune locals

Short summary:

Compiled frames/methods might need be deoptimized - in the deoptimization process, the local variables (and, maybe most importantly, the previous caller/sender frame's outgoing parameters) is to be restored in the deoptimized interpreter frame. If method liveness and other compilation steps determines dead locals, deoptimization will not restore these in the interpreter frame, including the outgoing parameters of the previous frame. Instead it will NULL dead local array index slots. A deoptimized interpreter frame which is subsequently popped using PopFrame might therefore induce NPE's and other exceptions since objectref, aka "this" has been pruned.

JVMTI PopFrame capability:

http://docs.oracle.com/javase/7/docs/platform/jvmti/jvmti.html#PopFrame

Most importantly for this discussion:

•the current frame is discarded as the previous frame becomes the current one
•the operand stack is restored--the argument values are added back and if the invoke was not invokestatic, objectref is added back as well
•the Java virtual machine PC is restored to the opcode of the invoke instruction

This is the current problem with the interaction between JVMTI PopFrame and the deoptimization process.

Just like the compiler has been instructed not to prune locals for when other JVMTI capabilities are requested, for example, can_access_local_variables, this should be done for can_pop_frame as well.

Longer discussion:

Currently there are several JVMTI tests that fail intermittently in nightly testing, most prevalent with NPE's. Since many of the tests are intertwined with several technologies (Hotswap, RedefineClasses, others) i have created this bug to factor out this particular aspect.

Describing this issue in detail is perhaps best done in light of a particular instance of the issue, for example https://bugs.openjdk.java.net/browse/JDK-6890958


In JDK-6890958 "Intermittent NPE when calling a method after class redefinition", the code looks somewhat like the following:

The main launcher thread:
...

   boolean passed = false;
        
        MyThread myThread = new MyThread();
        
        try {
            myThread.start();
            
            while (!isRedefined()) { // RedefineClass of MyThread class
                Thread.yield();
            }
            
            suspendThread(myThread);
            
            popThreadFrame(myThread);
            
            resumeThread(myThread);
            
            MyThread.stop = false;
            myThread.join();
}


// MyThread class

public static volatile boolean stop = true;

public void run() {
        doThisFunction();
        doTask3();
    }

    public void doThisFunction() {
        System.out.println(" MyThread.doThisFunction().");

        for (int i = 0; i < run_for; i++) {
            doTask2();
        }

        // it would wait for some time.
        while (stop);
        
        System.out.println(" End of doThisFunction.");
    }

    public void doTask2() {
        int p = 0;
        for (int i = 0; i < 1; i++) {
            threadState++;
            p++;
        }
    }

    public void doTask3() {
        // take some time
        for (int i = 0; i < run_for; i++) {
            doTask2();
        }
    }

(some details omitted for breivity)

The RedefineClasses process for MyThread is triggered after receiving a CompiledMethodLoad event for compilation of doTask2() (this compile is normal invocation hotspot detected with the loop targeting 1000 invocations and having set the -XX:CompileThreshold=1000)

Then comes the interesting part:

Just after the invocation loop, the thread spins on the condition variable "stop", waiting for it to be set by the first thread after having issued the PopFrame call (this means the frame to be popped is doThisFunction())

while(stop) looks like this:
...
27: getstatic #11 // Field stop:Z
30: ifeq 36
33: goto 27

This loop *might* cause the backedge counter to overflow and target the frame for OSR compilation.

A qualification on "might" is because this thread is racing against the RedefineClasses call - if RedefineClasses of MyThread manages to issue dependency invalidations before this backedge overflows (and the method is targeted for OSR compilation) - then the testcase is safe. The existing interpreter frame, and it's associated locals array, will be left as is.

However, if the method manages to move to OSR compilation before the RedefineClasses dependency invalidations, dependency invalidation will now have to deoptimize the just recently OSR compiled method back to an interpreted frame. Since the BCI at the location of the OSR is past any active uses of the locals, they are naturally, and correctly, eliminated by liveness analysis.

However, after the interpreter frame is restored, the PopFrame kicks in which pops doThisFunction() off the stack, it then rewinds the BCP to (0xb6 invokevirtual) and retries the invocation of doThisFunction() - unfortunately the objectref, "this" is now NULL on the stack - NPE ensues.