When using -Xcheck:jni an internally allocated buffer can leak

From discussion in:

http://mail.openjdk.java.net/pipermail/hotspot-dev/2017-December/029523.html

The checked version of getPrimitiveArrayCritical is implemented as follows:

JNI_ENTRY_CHECKED(void *,
  checked_jni_GetPrimitiveArrayCritical(JNIEnv *env,
                                        jarray array,
                                        jboolean *isCopy))
    functionEnterCritical(thr);
    IN_VM(
      check_is_primitive_array(thr, array);
    )
    void *result = UNCHECKED()->GetPrimitiveArrayCritical(env, array, isCopy);
    if (result != NULL) {
      result = check_jni_wrap_copy_array(thr, array, result);
    }
    functionExit(thr);
    return result;
JNI_END

This function always creates a copy of the array - check_jni_wrap_copy - keeps that fact private: *isCopy is not set to true (the real function always sets it to false).

The paired release function is implemented as follows:

JNI_ENTRY_CHECKED(void,
  checked_jni_ReleasePrimitiveArrayCritical(JNIEnv *env,
                                            jarray array,
                                            void *carray,
                                            jint mode))
    functionEnterCriticalExceptionAllowed(thr);
    IN_VM(
      check_is_primitive_array(thr, array);
    )
    // Check the element array...
    void* orig_result = check_wrapped_array_release(thr, "ReleasePrimitiveArrayCritical", array, carray, mode);
    UNCHECKED()->ReleasePrimitiveArrayCritical(env, array, orig_result, mode);
    functionExit(thr);
JNI_END

Note that "mode" is passed through to check_wrapped_array_release, which in turn does:

static void* check_wrapped_array_release(JavaThread* thr, const char* fn_name,
    void* obj, void* carray, jint mode) {
  size_t sz;
  void* orig_result = check_wrapped_array(thr, fn_name, obj, carray, &sz);
  switch (mode) {
  case 0:
    memcpy(orig_result, carray, sz);
    GuardedMemory::free_copy(carray);
    break;
  case JNI_COMMIT:
    memcpy(orig_result, carray, sz);
    break;
  case JNI_ABORT:
    GuardedMemory::free_copy(carray);
    break;
  default:

this mirrors how the real release operation handles the "mode" flag, but in this case we are not dealing with a "real copy" but the internal copy made just for checking purposes. If JNI_COMMIT is passed then we never free this internal copy and it leaks.

The same is true for release<type>ArrayElements functions.

Either check_wrapped_array_release should always be passed a mode of 0; or it should not do any mode processing at all and always free the known copy.