If a malformed policy file is encountered when running with a custom system class loader then it leads to a recursive initialization issue.

To demonstrate then see the attached test case and run it with:

java -Djava.system.class.loader=MyClassLoader -Djava.security.manager -Djava.security.policy=java.policy Main

The stack trace with jdk-9+140 is below. It's possible to change SL to not use getSystemClassLoader but that just moves the problem to ResourceBundle. The stack trace with JDK 8 is different but amounts to the same thing.

Error occurred during initialization of VM
java.lang.Error: java.lang.reflect.InvocationTargetException
at java.lang.ClassLoader.initSystemClassLoader(java.base@9-internal/ClassLoader.java:1748)
at java.lang.System.initPhase3(java.base@9-internal/System.java:1979)
Caused by: java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(java.base@9-internal/Native Method)
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(java.base@9-internal/NativeConstructorAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(java.base@9-internal/DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(java.base@9-internal/Constructor.java:455)
at java.lang.ClassLoader.initSystemClassLoader(java.base@9-internal/ClassLoader.java:1746)
at java.lang.System.initPhase3(java.base@9-internal/System.java:1979)
Caused by: java.lang.InternalError: getSystemClassLoader should only be called after VM booted
at java.lang.ClassLoader.getSystemClassLoader(java.base@9-internal/ClassLoader.java:1698)
at java.util.ServiceLoader.loadInstalled(java.base@9-internal/ServiceLoader.java:1053)
at java.util.ResourceBundle.<clinit>(java.base@9-internal/ResourceBundle.java:462)
at sun.security.util.ResourcesMgr$1.run(java.base@9-internal/ResourcesMgr.java:47)
at sun.security.util.ResourcesMgr$1.run(java.base@9-internal/ResourcesMgr.java:44)
at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method)
at sun.security.util.ResourcesMgr.getString(java.base@9-internal/ResourcesMgr.java:43)
at sun.security.provider.PolicyParser.match(java.base@9-internal/PolicyParser.java:746)