FULL PRODUCT VERSION :
Reproduced on J2SE 1.5 and 1.6

java version "1.5.0_19"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_19-b02)
Java HotSpot(TM) Client VM (build 1.5.0_19-b02, mixed mode, sharing)

java version "1.6.0_13"
Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
Java HotSpot(TM) Client VM (build 11.3-b02, mixed mode, sharing)

ADDITIONAL OS VERSION INFORMATION :
Microsoft Windows XP [Version 5.1.2600]

A DESCRIPTION OF THE PROBLEM :
For a JCA Provider that happens to call Security.getProviders() (or several other APIs that access the a provider) from it's constructor and is installed as an installed extension, the installation silently fails and the provider is not available.

I have read the "How to Implement a Provider for the Java Cryptography Architecture" guide and it does not mention anything about not making calls to Security.getProviders() in the constructor of a Provider.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. Create a provider that calls Security.getProviders() from its constructor.
2. Install that provider as an installed extension (place the JAR file containing the provider in the <java-home>\lib\ext directory, then add an entry in the <java-home>\lib\security\java.security policy file to register the provider, i.e. security.provider.n=masterClassName)
3. Attempt to access this provider by calling Security.getProvider(String)

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The provider is successfully installed and can be used.
ACTUAL -
The provider is not installed and cannot be used.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
Calls to Security.getProvider(String name) return null. Attempts to access an algorithm from this provider (i.e. MessageDigest.getInstance(String, String)) return the following exception.

java.security.NoSuchProviderException: no such provider: Entrust Test Provider (Bad)

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
Here is a sample test provider that demonstrates the issue (failure)

public class TestProviderBad extends java.security.Provider
{
    public TestProviderBad()
    {
        super("Entrust Test Provider (Bad)", 1.0, "N/A");
        java.security.Security.getProviders();
    }
}

Here is a sample test provider that demonstrates the desired result (success)

public class TestProviderGood extends java.security.Provider
{
    public TestProviderGood()
    {
        super("Entrust Test Provider (Good)", 1.0, "N/A");
    }
}

Compile these two classes, place them into a JAR file, then install this JAR file as an installed extension by placing the JAR file in <java-home>\lib\ext directory. Then modify the <java-home>\lib\security\java.security file to add registrations for these two providers; install them in last position

security.provider.7=TestProviderBad
security.provider.8=TestProviderGood

Finally, run the following sample application to reproduce the issue

import java.security.GeneralSecurityException;
import java.security.MessageDigest;
import java.security.Provider;
import java.security.Security;

public class ProviderTest
{
    public static void main(String[] args)
    {
        Provider[] providers = Security.getProviders();
        System.out.println("\nInstalled JCA/JCE CSPs: ");
        for (int i = 0; i < providers.length; i++) {
            System.out.println(" " + providers[i].getName());
        }
        System.out.println();
        for (int i = 0; i < providers.length; i++) {
            System.out.println("Security.getProvider(\"" + providers[i].getName() + "\"): " + Security.getProvider(providers[i].getName()));
        }
        
        try {
            MessageDigest.getInstance("SHA1", "Entrust Test Provider (Bad)");
        } catch(GeneralSecurityException e) {
            e.printStackTrace();
        }
        
        try {
            MessageDigest.getInstance("SHA1", "Entrust Test Provider (Good)");
        } catch(GeneralSecurityException e) {
            e.printStackTrace();
        }
    }
}

You'll see the following. Once the issue is resolved, the output for the "Bad" provider should be identical to the output for the "Good" provider

Installed JCA/JCE CSPs:
  SUN
  SunRsaSign
  SunJSSE
  SunJCE
  SunJGSS
  SunSASL
  Entrust Test Provider (Bad)
  Entrust Test Provider (Good)

Security.getProvider("SUN"): SUN version 1.5
Security.getProvider("SunRsaSign"): SunRsaSign version 1.5
Security.getProvider("SunJSSE"): SunJSSE version 1.5
Security.getProvider("SunJCE"): SunJCE version 1.5
Security.getProvider("SunJGSS"): SunJGSS version 1.0
Security.getProvider("SunSASL"): SunSASL version 1.5
Security.getProvider("Entrust Test Provider (Bad)"): null
Security.getProvider("Entrust Test Provider (Good)"): Entrust Test Provider (Good) version 1.0
java.security.NoSuchProviderException: no such provider: Entrust Test Provider (Bad)
at sun.security.jca.GetInstance.getService(GetInstance.java:66)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:190)
at java.security.Security.getImpl(Security.java:661)
at java.security.MessageDigest.getInstance(MessageDigest.java:172)
at ProviderTest.main(ProviderTest.java:22)
java.security.NoSuchAlgorithmException: no such algorithm: SHA1 for provider Entrust Test Provider (Good)
at sun.security.jca.GetInstance.getService(GetInstance.java:70)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:190)
at java.security.Security.getImpl(Security.java:661)
at java.security.MessageDigest.getInstance(MessageDigest.java:172)
at ProviderTest.main(ProviderTest.java:28)



---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
The only workaround is not to call Security.getProviders(). Unfortunately, there are certain circumstances under which this is very useful to do.

  To fix this problem, make the following modification to the sun.security.jca.Providers class (single line change). Currently, the code properly detects recursion, but does not handle the case correctly with respect to the resulting provider list.

OLD:

    public static synchronized ProviderList getFullProviderList() {
ProviderList list = getThreadProviderList();
if (list != null) {
ProviderList newList = list.removeInvalid();
if (newList != list) {
changeThreadProviderList(newList);
list = newList;
}
return list;
}
list = getSystemProviderList();
ProviderList newList = list.removeInvalid();
if (newList != list) {
setSystemProviderList(newList);
list = newList;
}
return list;
    }

NEW:

    public static synchronized ProviderList getFullProviderList() {
ProviderList list = getThreadProviderList();
if (list != null) {
ProviderList newList = list.removeInvalid();
if (newList != list) {
changeThreadProviderList(newList);
list = newList;
}
return list;
}
list = getSystemProviderList();
ProviderList newList = list.removeInvalid();
if (newList != getSystemProviderList()) {
setSystemProviderList(newList);
list = newList;
}
return list;
    }