In ClassLoader::get_canonical_path() calls into the Canonicalize() function in java.dll on windows. The Canonicalize() calls the single-byte version of canonicalize() in java.base/windows/native/libjava/canonicalize_md.c.

With path longer than MAX_PATH, canonicalize() fails in the _fullpath():
    if(!_fullpath(path, orig_path, sizeof(path))) {
        return -1;
    }

resulting in the following vm error:
Error occurred during initialization of VM
Exception: Bad pathname

This can be reproduced by having a very long path to a jar file specified to the -Xbootclasspath/a. There's a related bug JDK-8188122 addresses a different scenario when the -Xbootclasspath/a contains a long path without a jar file.

A possible fix is to use the Unicode version of canonicalize() function, wcanonicalize() in canonicalize_md.c for long path.