FULL PRODUCT VERSION :
openjdk version "9.0.1"
OpenJDK Runtime Environment (build 9.0.1+11-Ubuntu-1)
OpenJDK 64-Bit Server VM (build 9.0.1+11-Ubuntu-1, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Linux yushijinhun-laptop 4.15.0-10-generic #11-Ubuntu SMP Tue Feb 13 18:23:35 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

A DESCRIPTION OF THE PROBLEM :
In the JavaDoc of Instrumentation.retransformClasses() it says:
"@param classes array of classes to retransform; a zero-length array is allowed, in this case, this method does nothing"

However, when I pass a zero-length array into it, I got a NullPointerException and following log:
*** java.lang.instrument ASSERTION FAILED ***: "numClasses != 0" at JPLISAgent.c line: 1138

This is where the problem happens: http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/3a4370604bab/src/share/instrument/JPLISAgent.c#l1101

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Pass a zero-length array into Instrumentation.retransformClasses().

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The method does nothing.
ACTUAL -
The method throws a NullPointerException.
And following log is printed:
*** java.lang.instrument ASSERTION FAILED ***: "numClasses != 0" at JPLISAgent.c line: 1138

REPRODUCIBILITY :
This bug can be reproduced always.

CUSTOMER SUBMITTED WORKAROUND :
Check the length of the array before passing it into Instrumentation.retransformClasses().