Customer runs numerous applications on the WLS platform, which they access from approx. 150K client systems. Said client systems are based on Windows XP (SP3), and are all homogeneously configured via a centralized software distribution mechanism. The client systems use Hotspot JRE 1.6_u21 as their Java runtime (locally installed on every system).

Two of the mentionned applications are called ATV and Delta, repectively. The ATV / Delta application-clients are loaded and executed via the JRE 1.6 Java Webstart mechanism.

Since the customer promoted their client systems from JRE 1.5 to 1.6 some months ago, they are frequently encountering problems with the Java Webstart load/execution mechanism. These problems manifest themselves as an error while loading the standard installed JCA security providers. Here is a sample stack trace:

...
java.lang.NoClassDefFoundError: Could not initialize class sun.security.mscapi.SunMSCAPI.
...
Caused by: java.lang.NoClassDefFoundError: sun/security/mscapi/SunMSCAPI$1
at sun.security.mscapi.SunMSCAPI.<clinit>(SunMSCAPI.java:32)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at java.lang.Class.newInstance0(Unknown Source)
at java.lang.Class.newInstance(Unknown Source)
at sun.security.jca.ProviderConfig$3.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.jca.ProviderConfig.doLoadProvider(Unknown Source)
at sun.security.jca.ProviderConfig.getProvider(Unknown Source)
at sun.security.jca.ProviderList.loadAll(Unknown Source)
at sun.security.jca.ProviderList.removeInvalid(Unknown Source)
at sun.security.jca.Providers.getFullProviderList(Unknown Source)
at java.security.Security.getProviders(Unknown Source)
at sun.security.jgss.ProviderList.<init>(Unknown Source)
at sun.security.jgss.GSSManagerImpl.<init>(Unknown Source)
at org.ietf.jgss.GSSManager.getInstance(Unknown Source)
at xx.yyy.sso.common.authentication.InitSecContextAction.run(InitSecContextAction.java:55)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at xx.yyy.sso.common.authentication.KerberosAuthentication.requestServiceTicketInternal(KerberosAuthentication.java:218)
at xx.yyy.sso.common.authentication.KerberosAuthentication$2.execute(KerberosAuthentication.java:232)
at xx.yyy.sso.common.authentication.InterruptedExecution.run(InterruptedExecution.java:53)
Caused by: java.lang.ClassNotFoundException: sun.security.mscapi.SunMSCAPI$1
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at sun.misc.Launcher$ExtClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 24 more
Caused by: java.util.zip.ZipException: error reading zip file
at java.util.zip.ZipFile.read(Native Method)
at java.util.zip.ZipFile.access$1200(Unknown Source)
at java.util.zip.ZipFile$ZipFileInputStream.read(Unknown Source)
at java.util.zip.ZipFile$1.fill(Unknown Source)
at java.util.zip.InflaterInputStream.read(Unknown Source)
at java.util.jar.JarVerifier$VerifierStream.read(Unknown Source)
at sun.misc.Resource.getBytes(Unknown Source)
at java.net.URLClassLoader.defineClass(Unknown Source)
at java.net.URLClassLoader.access$000(Unknown Source)
... 30 more

When this type of error, which occurs randomly amongst the client base, is thrown, the application-client won't start. After an indistinct number of retries, the error usually goes away, and the application-client starts up nicely again.

Whenever said error occurs, it seems to be related to the SunMSCAPI security provider (see stack trace above), which was introduced with JRE 1.6. It must be noted, though, that none of the customer applications uses any of the functionality provided by this security provider. The provider is configured with the lowest preference (#9) in the JCA configuration file, $JAVA_HOME/jre/lib/security/java.security :

...
security.provider.1=sun.security.provider.Sun
security.provider.2=sun.security.rsa.SunRsaSign
security.provider.3=com.sun.net.ssl.internal.ssl.Provider
security.provider.4=com.sun.crypto.provider.SunJCE
security.provider.5=sun.security.jgss.SunProvider
security.provider.6=com.sun.security.sasl.Provider
security.provider.7=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.8=sun.security.smartcardio.SunPCSC
security.provider.9=sun.security.mscapi.SunMSCAPI
...

The above error cannot reliably be reproduced -- as mentionned, it comes and goes on different systems.


TESTS TO NAIL DOWN THE PROBLEM

We did some tests to isolate the problem:

1) Changed the preference order in "java.security" (esp. moving SunMSCAPI to a higher preference) and started/stopped the applications in question on given clients 2000 (or more) times (script-based) -- to no avail; the error still occurs

2) Commented out the SunMSCAPI security provider in "java.security" -- this seems to remedy the situation; more than 2000 application start/stop iterations did not produce a single error

3) Changed the load/execution mechanism for the applications, so that they are started standalone, i.e. WITHOUT Java Webstart, but with SunMSCAPI commented in in "java.security" and JVM flag '-verbose:class' -- more than 2000 application start/stop iterations did not produce a single error

Using a Microsoft diagnosis tool, Process Monitor, we were able to see that whenever the above error is thrown, $JRE_HOME/lib/ext/sunmscapi.jar is opened and its content is extensively read (readFile offset ...). When said error is not thrown (success case), the extensive reading of the provider jar does not occur. What we also see is that all the security providers seem to be initially loaded, however not sequentially according to their preference.

We suspect that the cause of the above error could be related to synchronization problems in class loading, i.e. the way Java webstart does its stuff and than hands off execution to the JVM. It is not fully transparent to us, though, what is exactly happening in the collaboration of Java Webstart and the JVM.

Attached in BA_JNLP_Logs.zip are the following files:

* ATV_jnlp.zip -- the Java Webstart JNLP files for application 'ATV'

* Delta.jnlp -- the Java Webstart JNLP file for application 'Delta'

* Delta_Standalone_verboseClass.txt -- verbose output of class loading activity when (successfully) starting application 'Delta' WITHOUT Java Webstart (see test #3 above)


#

java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b07)
Java HotSpot(TM) Client VM (build 17.0-b17, mixed mode, sharing)

---------------------------
Java(TM) Web Start 1.6.0_21
---------------------------
Launching: C:\Programme\Java\jre1.6.0\bin\javaw.exe
C:\Programme\Java\jre1.6.0\bin\javaw.exe
 -Xbootclasspath/a:C:\Programme\Java\jre1.6.0\lib\javaws.jar;C:\Programme\Java\jre1.6.0\lib\deploy.jar;C:\Programme\Java\jre1.6.0\lib\plugin.jar
 -classpath
 C:\Programme\Java\jre1.6.0\lib\deploy.jar
 -Djava.security.policy=file:C:\Programme\Java\jre1.6.0\lib\security\javaws.policy
 -DtrustProxy=true
 -Xverify:remote
 -Djnlpx.home=C:\Programme\Java\jre1.6.0\bin
 -Djavaplugin.user.profile=C:\Dokumente und Einstellungen\BullmannV\Anwendungsdaten
 -Dsun.awt.warmup=true
 -Djnlpx.origFilenameArg=C:\Programme\BA\ATV\launch_atv.jlnp
 -Djnlpx.remove=false
 -Djnlpx.splashport=4944
 -Djnlpx.jvm=C:\Programme\Java\jre1.6.0\bin\javaw.exe
 com.sun.javaws.Main
 C:\Programme\BA\ATV\launch_atv.jlnp