FULL PRODUCT VERSION :
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)

FULL OS VERSION :
OS X Yosemite 10.10.5

A DESCRIPTION OF THE PROBLEM :
While debugging a JVMTI agent (whose code I unfortunately cannot share with you at the moment), I discovered that GetLocalInstance behaves different than documented.

The documentation [1] states that it returns JVMTI_ERROR_INVALID_SLOT "if the specified frame is a static method frame."

When the specified frame referred to a method "public static void method(int i)", however, I have observed a return value of JVMTI_ERROR_TYPE_MISMATCH.

Moreover, if the method was "public static void method(Object o)" instead, with an Object in the first slot, I even got JVMTI_ERROR_NONE, with the Object o as receiving instance.

Cursory inspection of VM_GetOrSetLocal::check_slot_type (src/share/vm/prims/jvmtiImpl.cpp) suggest that this method, which produces the error code returned by GetLocalInstance, is indeed oblivious as to whether the receiver (which a static method doesn't have) or the object in slot 0 is requested; this would explain the behavior described above.

I hope this is enough information to reproduce the error.

[1] <http://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html#GetLocalInstance>

THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Yes

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
- Call a native method from within a static method of signature (of varying signature, see "expected and actual results").

- Let the native method attempt to use JVMTI's GetLocalInstance function to access the receiver of the stack method.

- Observe the error code.

EXPECTED VERSUS ACTUAL BEHAVIOR :
static void method(int i):

  expected: JVMTI_ERROR_INVALID_SLOT
  actual: JVMTI_ERROR_TYPE_MISMATCH

static void method(Object o):

  expected: JVMTI_ERROR_INVALID_SLOT
  actual: JVMTI_ERROR_NONE

static void method():

  expected: JVMTI_ERROR_INVALID_SLOT
  actual: JVMTI_ERROR_INVALID_SLOT

REPRODUCIBILITY :
This bug can be reproduced always.

CUSTOMER SUBMITTED WORKAROUND :
Detect whether the method is static (using GetFrameLocation + GetMethodModifiers) before calling GetLocalInstance.

Just treating VMTI_ERROR_TYPE_MISMATCH as VMTI_ERROR_INVALID_SLOT unfortunately doesn't work, as the static method's first argument may be an Object, in which case *no* error would be return, even though it should.