The following code in MethodHandleNatives.varHandleOperationLinkerMethod implements a check which is not specified by JVMS:

   if (ak.at.isMonomorphicInReturnType) {
       if (ak.at.returnType != mtype.returnType()) {
           // The caller contains a different return type than that
           // defined by the method
           throw newNoSuchMethodErrorOnVarHandle(name, mtype);
       }
       // Adjust the return type of the signature method type
       guardReturnType = ak.at.returnType;
   }

(I am assuming this method is called by Hotspot during method resolution.)

Expected behavior is for linkage to succeed and for run-time invocation to fail with a WrongMethodTypeException.

Specification:

- JVMS 5.4.3.3: "If C declares exactly one method with the name specified by the method reference, and the declaration is a signature polymorphic method (§2.9.3), then method lookup succeeds. All the class names mentioned in the descriptor are resolved (§5.4.3.1)."

- VarHandle javadoc: "The first time an invokevirtual instruction is executed it is linked by symbolically resolving the names in the instruction and verifying that the method call is statically legal. This also holds for calls to access mode methods. In this case, the symbolic type descriptor emitted by the compiler is checked for correct syntax, and names it contains are resolved. Thus, an invokevirtual instruction which invokes an access mode method will always link, as long as the symbolic type descriptor is syntactically well-formed and the types exist."

  "When the invokevirtual is executed after linking, the receiving VarHandle's access mode type is first checked by the JVM to ensure that it matches the symbolic type descriptor. If the type match fails, it means that the access mode method which the caller is invoking is not present on the individual VarHandle being invoked."

- From VarHandle.compareAndSet, etc., javadoc: "throws WrongMethodTypeException - if the access mode type does not match the caller's symbolic type descriptor."

This issue was raised by Jorn Vernee, following a StackOverflow post:
https://stackoverflow.com/questions/52962939/java-lang-nosuchmethoderror-varhandle-compareandsetvariablehandlesexample-stat