The code in attachListener.cpp does this:

  DCmd::parse_and_execute(DCmd_Source_AttachAPI, out, op->arg(0), ' ', THREAD);
  if (HAS_PENDING_EXCEPTION) {
    java_lang_Throwable::print(PENDING_EXCEPTION, out);
    out->cr();
    CLEAR_PENDING_EXCEPTION;
    // The exception has been printed on the output stream
    // If the JVM returns JNI_ERR, the attachAPI throws a generic I/O
    // exception and the content of the output stream is not processed.
    // By returning JNI_OK, the exception will be displayed on the client side
  }
  return JNI_OK;

That was correct before the fix of JDK-8039173. After that fix, the attach framework is able to propagate error messages when an attach command fails.

The code in attachListener.cpp should be updated to:

  DCmd::parse_and_execute(DCmd_Source_AttachAPI, out, op->arg(0), ' ', THREAD);
  if (HAS_PENDING_EXCEPTION) {
    java_lang_Throwable::print(PENDING_EXCEPTION, out);
    out->cr();
    CLEAR_PENDING_EXCEPTION;
    return JNI_ERR;
  }
  return JNI_OK;