The background for this issue, and the test case, is in JDK-8224791.

jdk.boot.class.path.append is only readable via JVM TI GetSystemProperty function (not System::getProperty). It is not set by default consistent with other VM properties e.g. jdk.module.path. It is only available if -Xbootclasspath/a option is set at startup or JVM TI AppendBootstrapClassLoaderSearch function is called during onLoad phase.

Defaulting to the empty string is a bug and GetSystemProperty("jdk.boot.class.path.append", &value) should return JVMTI_ERROR_NOT_AVAILABLE if not set.

A CSR request will need to be filed for this behaviour change to re-align with the intended specification.