ADDITIONAL SYSTEM INFORMATION :
openjdk version "21.0.5" 2024-10-15.
OpenJDK Runtime Environment (build 21.0.5+11-Ubuntu-1ubuntu124.04)
OpenJDK 640Bit Server VM (build 21.0.5+11Ubuntu=1ubuntu124.04, mixed mode, sharing)

A DESCRIPTION OF THE PROBLEM :
Setting jdk.security.provider.preferred:SHA256withRSA:<provider> to anything other than SunRsaSign prevents all providers apart from SUN, SunRsaSign, SunJSSE and SunJCE registering.
This appears to be a bootstrapping problem where the SHA256withRSA algorithm is required in order to verify a provider as part of the registration process and this can't be performed as the new preferred provider isn't yet registered.
This was found with an HSM (hardware) provider that fully supports SHA256withRSA.
The HSM provider was found by using the --module-path option. It was set as a provider in java.security at the lowest priority. Setting jdk.security.provider.preferred:SHA256withRSA:SunRsaSign works as expected


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
A JCE provider other than SunRsaSign that supports SHA56withRSA is required. Set that provider in the java.security providers list and set jdk.security.provider.preferred:SHA256withRSA:<provider>
Run a program that enumerates the JCE providers

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
All 12 built-in JCE providers and the additional hardware provider are enumerated.
ACTUAL -
Only SUN, SunRsaSign, SunJSSE and SunJCE are registered. All other providers fail to register with Exceptions as below.
If java.security.preferred:SHA256withRSA is not set, all providers are registered as expected.

First exception when experimenting with -Djava.security.debug=all
ProviderConfig: Loading provider SunEC
ProviderConfig: Attempt to load SunEC using SL
ProviderConfig: Loading legacy provider: SunEC
ProviderConfig: Error loading legacy provider SunEC
java.lang.ClassNotFoundException: SunEC
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
        at java.base/sun.security.jca.ProviderConfig$ProviderLoader.legacyLoad(ProviderConfig.java:391)
        at java.base/sun.security.jca.ProviderConfig$ProviderLoader.load(ProviderConfig.java:368)
        at java.base/sun.security.jca.ProviderConfig$3.run(ProviderConfig.java:253)
        at java.base/sun.security.jca.ProviderConfig$3.run(ProviderConfig.java:247)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)
        at java.base/sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:247)
        at java.base/sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:225)
        at java.base/sun.security.jca.ProviderList.getProvider(ProviderList.java:271)
        at java.base/sun.security.jca.ProviderList.getIndex(ProviderList.java:301)
        at java.base/sun.security.jca.ProviderList.getProviderConfig(ProviderList.java:285)
        at java.base/sun.security.jca.ProviderList.getProvider(ProviderList.java:291)
        at java.base/sun.security.jca.ProviderList$ServiceList.tryGet(ProviderList.java:506)
        at java.base/sun.security.jca.ProviderList$ServiceList$1.hasNext(ProviderList.java:570)
        at java.base/java.security.Signature.getInstance(Signature.java:267)
        at java.base/sun.security.pkcs.SignerInfo.makeSigAlg(SignerInfo.java:542)
        at java.base/sun.security.pkcs.SignerInfo.verify(SignerInfo.java:399)
        at java.base/sun.security.pkcs.PKCS7.verify(PKCS7.java:534)
        at java.base/sun.security.pkcs.PKCS7.verify(PKCS7.java:551)
        at java.base/sun.security.pkcs.SignerInfo.getTimestamp(SignerInfo.java:645)
        at java.base/sun.security.pkcs.SignerInfo.verify(SignerInfo.java:314)
        at java.base/sun.security.pkcs.PKCS7.verify(PKCS7.java:534)
        at java.base/sun.security.pkcs.PKCS7.verify(PKCS7.java:551)
        at java.base/sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:306)
        at java.base/sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:282)
        at java.base/java.util.jar.JarVerifier.processEntry(JarVerifier.java:320)
        at java.base/java.util.jar.JarVerifier.update(JarVerifier.java:232)
        at java.base/java.util.jar.JarFile.initializeVerifier(JarFile.java:760)
        at java.base/java.util.jar.JarFile.getInputStream(JarFile.java:858)
        at java.base/jdk.internal.module.ModulePath.readJar(ModulePath.java:652)
        at java.base/jdk.internal.module.ModulePath.readModule(ModulePath.java:332)
        at java.base/jdk.internal.module.ModulePath.scanDirectory(ModulePath.java:285)
        at java.base/jdk.internal.module.ModulePath.scan(ModulePath.java:233)
        at java.base/jdk.internal.module.ModulePath.scanNextEntry(ModulePath.java:191)
        at java.base/jdk.internal.module.ModulePath.findAll(ModulePath.java:167)
        at java.base/java.lang.module.ModuleFinder$2.lambda$findAll$2(ModuleFinder.java:365)
        at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
        at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
        at java.base/java.lang.module.ModuleFinder$2.findAll(ModuleFinder.java:366)
        at java.base/java.lang.module.Resolver.findAll(Resolver.java:856)
        at java.base/java.lang.module.Resolver.bind(Resolver.java:234)
        at java.base/java.lang.module.Configuration.resolveAndBind(Configuration.java:313)
        at java.base/java.lang.module.ModuleDescriptor$1.resolveAndBind(ModuleDescriptor.java:2764)
        at java.base/jdk.internal.module.Modules.newBootLayerConfiguration(Modules.java:184)
        at java.base/jdk.internal.module.ModuleBootstrap.boot2(ModuleBootstrap.java:383)
        at java.base/jdk.internal.module.ModuleBootstrap.boot(ModuleBootstrap.java:174)
        at java.base/java.lang.System.initPhase2(System.java:2231)
ProviderConfig: Error loading provider SunEC


---------- BEGIN SOURCE ----------
import java.security.Provider;
import java.security.Security;

public class ProviderTest {
    public static void main (String[] args) throws Exception {
        Provider [] providers = Security.getProviders();
        for(int i=0;i<providers.length;i++){
            System.out.println(String.valueOf(i+1) + ": " + providers[i].getName()) ;
        }
    }
}
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
If you have access to the code that consumes the cryptographic algorithms, then calling e.g. Signature sig = Signature.getInstance("SHA256withRSA","PreferredProvider"); instead of accepting the default provider achieves the same as setting jdk.security.provider.preferred.
Changing the default provider ordering so the preferred provider is higher in the provider list than SunRsaSign and then explicitly setting SunRsaSign as the preferred provider for any other algorithms that are now going to the preferred SHA256withRSA provider would achieve the same result

FREQUENCY : always