ADDITIONAL SYSTEM INFORMATION :
The problem only occurs on JDK 11.0.11, not on 11.0.10. Also no problem on the other ones I tested: 9.0.4, 13.0.2, 14.0.2, 15.0.2, 16. I tested on Windows 10 Professional.

A DESCRIPTION OF THE PROBLEM :
A detailed description can be found in my StackOverflow answer here:
https://stackoverflow.com/a/68295398/1082681

The problem occurs if and only if
a) -Djava.system.class.loader is used and
b) the system class loader in question is found in a signed JAR and
c) the application runs on JDK 11.0.11 (not 11.0.10 or any other JDK 9-16 I tested).

As soon as I remove the JAR signature from the META-INF directory, it also works on JDK 11.0.11. IMO, this is a regression bug in the JDK and should be fixed.

REGRESSION : Last worked in version 11

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Source code for minimal test case see below.

Please make sure that both classes are in different packages, otherwise later the JVM will complain about not both of them being signed.

We are is a base directory, the source code is in folder 'src'.

------------------------------------------------------------------------

# Compile source files (JDK used for compilation is unimportant)
javac --release 8 src\org\acme\app\Main.java src\org\acme\loader\CustomClassLoader.java

# Create JAR containing custom class loader
jar cf CustomClassLoader.jar -C src org\acme\loader\CustomClassLoader.class

# Create signing key (default keystore has password 'changeit')
keytool -genkeypair -keyalg RSA -alias test-user

# Sign JAR (default keystore has password 'changeit')
jarsigner CustomClassLoader.jar test-user

# Run dummy application, setting custom class loader from JAR as system class loader
"c:\Program Files\Java\jdk-11.0.11\bin\java.exe" -Djava.security.debug="jca" -Djava.system.class.loader=org.acme.loader.CustomClassLoader -cp "CustomClassLoader.jar;src" org.acme.app.Main

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Output should be:

ProviderList: provider configuration: [SUN, SunRsaSign, SunEC, SunJSSE, SunJCE, SunJGSS, SunSASL, XMLDSig, SunPCSC, JdkLDAP, JdkSASL, SunMSCAPI, SunPKCS11]
ProviderList: config configuration: null
ProviderList: ThreadLocal providers: [SUN, SunRsaSign, SunEC]
ProviderList: Disabling ThreadLocal providers
ProviderList: ThreadLocal providers: [SUN, SunRsaSign, SunEC]
ProviderList: Disabling ThreadLocal providers
ACTUAL -
Actual output is:

ProviderList: provider configuration: [SUN, SunRsaSign, SunEC, SunJSSE, SunJCE, SunJGSS, SunSASL, XMLDSig, SunPCSC, JdkLDAP, JdkSASL, SunMSCAPI, SunPKCS11]
ProviderList: config configuration: null
ProviderList: ThreadLocal providers: [SUN, SunRsaSign, SunEC, SunJCE]
ProviderList: Disabling ThreadLocal providers
ProviderList: ThreadLocal providers: [SUN, SunRsaSign, SunEC, SunJCE]
ProviderList: Loading all providers
java.lang.Exception: Debug Info. Call trace:
        at java.base/sun.security.jca.ProviderList.loadAll(ProviderList.java:311)
        at java.base/sun.security.jca.ProviderList.removeInvalid(ProviderList.java:332)
        at java.base/sun.security.jca.Providers.getFullProviderList(Providers.java:165)
        at java.base/java.security.Security.getProviders(Security.java:457)
        at java.base/sun.security.x509.AlgorithmId.computeOidTable(AlgorithmId.java:632)
        at java.base/sun.security.x509.AlgorithmId.oidTable(AlgorithmId.java:622)
        at java.base/sun.security.x509.AlgorithmId.algOID(AlgorithmId.java:604)
        at java.base/sun.security.x509.AlgorithmId.get(AlgorithmId.java:436)
        at java.base/sun.security.pkcs.SignerInfo.verify(SignerInfo.java:379)
        at java.base/sun.security.pkcs.PKCS7.verify(PKCS7.java:578)
        at java.base/sun.security.pkcs.PKCS7.verify(PKCS7.java:595)
        at java.base/sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:283)
        at java.base/sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:259)
        at java.base/java.util.jar.JarVerifier.processEntry(JarVerifier.java:316)
        at java.base/java.util.jar.JarVerifier.update(JarVerifier.java:230)
        at java.base/java.util.jar.JarFile.initializeVerifier(JarFile.java:759)
        at java.base/java.util.jar.JarFile.ensureInitialization(JarFile.java:1038)
        at java.base/java.util.jar.JavaUtilJarAccessImpl.ensureInitialization(JavaUtilJarAccessImpl.java:69)
        at java.base/jdk.internal.loader.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:872)
        at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:786)
        at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:398)
        at java.base/java.lang.ClassLoader.initSystemClassLoader(ClassLoader.java:1975)
        at java.base/java.lang.System.initPhase3(System.java:2070)
ProviderConfig: Loading provider SunEC
ProviderConfig: Error loading provider SunEC
java.lang.ExceptionInInitializerError
        at java.base/sun.security.jca.ProviderConfig$3.run(ProviderConfig.java:244)
        at java.base/sun.security.jca.ProviderConfig$3.run(ProviderConfig.java:238)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:238)
        at java.base/sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:218)
        at java.base/sun.security.jca.ProviderList.loadAll(ProviderList.java:315)
        at java.base/sun.security.jca.ProviderList.removeInvalid(ProviderList.java:332)
        at java.base/sun.security.jca.Providers.getFullProviderList(Providers.java:165)
        at java.base/java.security.Security.getProviders(Security.java:457)
        at java.base/sun.security.x509.AlgorithmId.computeOidTable(AlgorithmId.java:632)
        at java.base/sun.security.x509.AlgorithmId.oidTable(AlgorithmId.java:622)
        at java.base/sun.security.x509.AlgorithmId.algOID(AlgorithmId.java:604)
        at java.base/sun.security.x509.AlgorithmId.get(AlgorithmId.java:436)
        at java.base/sun.security.pkcs.SignerInfo.verify(SignerInfo.java:379)
        at java.base/sun.security.pkcs.PKCS7.verify(PKCS7.java:578)
        at java.base/sun.security.pkcs.PKCS7.verify(PKCS7.java:595)
        at java.base/sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:283)
        at java.base/sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:259)
        at java.base/java.util.jar.JarVerifier.processEntry(JarVerifier.java:316)
        at java.base/java.util.jar.JarVerifier.update(JarVerifier.java:230)
        at java.base/java.util.jar.JarFile.initializeVerifier(JarFile.java:759)
        at java.base/java.util.jar.JarFile.ensureInitialization(JarFile.java:1038)
        at java.base/java.util.jar.JavaUtilJarAccessImpl.ensureInitialization(JavaUtilJarAccessImpl.java:69)
        at java.base/jdk.internal.loader.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:872)
        at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:786)
        at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:398)
        at java.base/java.lang.ClassLoader.initSystemClassLoader(ClassLoader.java:1975)
        at java.base/java.lang.System.initPhase3(System.java:2070)
Caused by: java.lang.IllegalStateException: getSystemClassLoader cannot be called during the system class loader instantiation
        at java.base/java.lang.ClassLoader.getSystemClassLoader(ClassLoader.java:1931)
        at java.base/sun.security.jca.ProviderConfig$ProviderLoader.<init>(ProviderConfig.java:319)
        at java.base/sun.security.jca.ProviderConfig$ProviderLoader.<clinit>(ProviderConfig.java:309)
        ... 34 more
ProviderList: Disabling ThreadLocal providers

---------- BEGIN SOURCE ----------
package org.acme.app;

public class Main {
  public static void main(String[] args) {}
}

------------------------------------------------------------------------

package org.acme.loader;

import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;

public class CustomClassLoader extends ClassLoader {

  public CustomClassLoader(ClassLoader parent) {
    super(parent);
  }

  @Override
  public Class<?> findClass(String name) throws ClassNotFoundException {
    byte[] b = loadClassFromFile(name);
    return defineClass(name, b, 0, b.length);
  }

  private byte[] loadClassFromFile(String fileName) {
    InputStream inputStream = getClass().getClassLoader()
      .getResourceAsStream(fileName.replace('.', File.separatorChar) + ".class");
    byte[] buffer;
    ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
    int nextValue = 0;
    try {
      while ((nextValue = inputStream.read()) != -1) {
        byteStream.write(nextValue);
      }
    } catch (IOException e) {
      e.printStackTrace();
    }
    buffer = byteStream.toByteArray();
    return buffer;
  }
}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Do not sign the JAR or remove an existing JAR signature from META-INF.

FREQUENCY : always