ArrayTypeImp::newInstance() is called by the following code, which calls all the debug agent command handlers:

            } else {
                /* Call the command handler */
                replyToSender = func(&in, &out);
            }

And then the reply to the command is done by the following:

            /* Reply to the sender */
            if (replyToSender) {
                if (inStream_error(&in)) {
                    outStream_setError(&out, inStream_error(&in));
                }
                outStream_sendReply(&out);
            }

So if the command handler returns FALSE, no reply is sent. The JDWP ArrayType.newInstance command is suppose to always send a reply, even if there is an error, but there is one error condition where it doesn't:

    error = classSignature(arrayClass, &signature, NULL);
    if ( error == JVMTI_ERROR_NONE ) {
        outStream_setError(out, map2jdwpError(error));
        return JNI_FALSE;
    }

If for any reason classSignature() should fail, FALSE is returned from ArrayTypeImp::newInstance(), thus no reply is sent. TRUE should be returned, allowing the reply to be sent with the error.

I only noticed this problem by observation. I went looking for when command handlers that returned FALSE (and thus no reply) and stumbled on this one. It turns out there aren't any that are allowed to return FALSE, and this is the only one I found, which the exception of VirtualMachineImp::doExit(), which actually exits the process before returning.

We have no test for this failure and it's hard to write one. I did however modify the above code to always produce an error and then ran vmTestbase/nsk/jdi/ArrayType/newInstance/newinstance001. It timed out waiting for a reply packet. I then fixed the above code to return TRUE when there is a failure, and the test then failed quickly with the expected exception in the output.