FULL PRODUCT VERSION :
java version "1.6.0-rc"
Java(TM) SE Runtime Environment (build 1.6.0-rc-b97)
Java HotSpot(TM) Client VM (build 1.6.0-rc-b97, mixed mode, sharing)


ADDITIONAL OS VERSION INFORMATION :
Linux apetersen 2.6.15-26-686 #1 SMP PREEMPT Thu Aug 3 03:13:28 UTC 2006 i686 GNU/Linux


A DESCRIPTION OF THE PROBLEM :
The basic problem is that you can't provide a custom default SSLSocketFactory when using Java Web Start. Assumingly because of the way that the classloader is set up in Web Start, combined with the implementation of javax.net.ssl.DefaultSSLSocketFactory, you end up getting a ClassNotFoundException looking for your custom SSLSocketFactory.

To give some context, I'm using the JavaMail 1.4 API to make a TLS connection to an IMAP server. JavaMail 1.4 uses the default SSLSocketFactory to create these connections, so the only way to use a custom SSLSocketFactory is to set it as the default one. And for a user-facing mail app, it's necessary to override the default SSLSocketFactory because the default TrustManager is far too draconian for user apps, as it lacks any way to accept a self-signed certificate on the fly.

I'm not sure if this is actually a bug in the SSLSocketFactory implementation or in the Web Start classloader system. I'm pretty sure that it's not a bug in JavaMail, since it seems to be trying to do the right thing.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1) Make a subclass of javax.net.ssl.SSLSocketFactory

2) Make this the default SSLSocketFactory:
      java.security.Security.setProperty("ssl.SocketFactory.provider","TestSocketFactory");

3) Try to make an SSL Connection
      javax.net.SocketFactory sf = javax.net.ssl.SSLSocketFactory.getDefault();
      java.net.Socket socket = sf.createSocket("localhost", 8080);

4) Deploy this code in a Web Start application.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
I would expect that this would work exactly the same way under Web Start as it works when you invoke it directly using 'java -jar TestSocketFactory.jar': you'd get a new SSL socket that was created using your SSLSocketFactory.
ACTUAL -
Instead, you get a ClassNotFoundException saying that the TestSocketFactory class can't be found.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
java.net.SocketException: java.lang.ClassNotFoundException: TestSocketFactory
at javax.net.ssl.DefaultSSLSocketFactory.throwException(SSLSocketFactory.java:179)
at javax.net.ssl.DefaultSSLSocketFactory.createSocket(SSLSocketFactory.java:192)
at SecurityTest.main(SecurityTest.java:13)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.javaws.Launcher.executeApplication(Launcher.java:1196)
at com.sun.javaws.Launcher.executeMainClass(Launcher.java:1142)
at com.sun.javaws.Launcher.doLaunchApp(Launcher.java:989)
at com.sun.javaws.Launcher.run(Launcher.java:105)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.ClassNotFoundException: TestSocketFactory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at javax.net.ssl.SSLSocketFactory.getDefault(SSLSocketFactory.java:85)
at SecurityTest.main(SecurityTest.java:11)
... 9 more


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
---- SecurityTest.java
import javax.swing.*;
import java.io.*;

public class SecurityTest {
  public static void main(String[] argv) {
    String defaultFactory = null;
    String connectionlabel = null;
    String stackTrace = null;
    try {
      java.security.Security.setProperty("ssl.SocketFactory.provider","TestSocketFactory");
      javax.net.SocketFactory sf = javax.net.ssl.SSLSocketFactory.getDefault();
      defaultFactory = "property = '" + java.security.Security.getProperty("ssl.SocketFactory.provider") + "'; got factory " + sf.toString();
      java.net.Socket socket = sf.createSocket("localhost", 8080);
      connectionlabel = "got socket.";
    } catch (Exception e) {
      connectionlabel = "error: " + e.getMessage();
      StringWriter sw = new StringWriter();
      e.printStackTrace(new PrintWriter(sw));
      stackTrace = sw.toString();
    }
    JFrame jf = new JFrame();
    JTextArea jta = new JTextArea("Result of getDefault(): " + defaultFactory + "\r\n getConnection(): " + connectionlabel + "\r\n" + "Error:\r\n" + stackTrace);
    jf.add(jta);
    jf.pack();
    jf.addWindowListener(new java.awt.event.WindowAdapter() {
        public void windowClosed(java.awt.event.WindowEvent e) {
          System.exit(0);
        }
        public void windowClosing(java.awt.event.WindowEvent e) {
          System.exit(0);
        }
      });
    jf.setVisible(true);
  }
}


---- TestSocketFactory.java
import java.net.*;
import java.io.IOException;

public class TestSocketFactory extends javax.net.ssl.SSLSocketFactory {

  /**
   * Creates an SSL Socket.
   */
  public Socket createSocket(Socket s, String host, int port, boolean autoClose) throws IOException {
    return null;
  }

  public Socket createSocket(String host, int port, java.net.InetAddress addr )
throws IOException {
    return null;
  }

  public Socket createSocket(String host, int port, java.net.InetAddress addr, int port2 ) throws IOException {
    return null;
  }

  public Socket createSocket(String host, int port ) throws IOException {
    return null;
  }

  public Socket createSocket(java.net.InetAddress addr,int port ,java.net.InetAddress addr2,int port2) {
    return null;
  }

  public Socket createSocket(java.net.InetAddress addr,int port) {
    return null;
  }

  public String[] getDefaultCipherSuites() {
    return null;
  }

  public String[] getSupportedCipherSuites() {
    return null;
  }
}

----- testsocketfactory.jnlp
<jnlp spec="1.0+" codebase="http://localhost/" href="http://localhost/testsocketfactory.jnlp">
  <information>
    <title>Socket Factory Test</title>
    <homepage href="null"/>
    <description>Socket Factory Test</description>
    <offline-allowed/>
  </information>
  <security>
    <all-permissions/>
  </security>
  <update check="timeout" policy="always"/>
  <resources>
    <java initial-heap-size="25165824" max-heap-size="134217728" version="1.4+"/>
    <jar href="http://localhost/TestSocketFactory.jar" download="eager" main="false"/>
  </resources>
  <application-desc main-class="SecurityTest"/>
</jnlp>

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
The only workarounds that I've found are pretty bad: don't use Web Start, don't support TLS (JavaMail 1.4 has an alternate SSL Factory mechanism for imaps), or don't override the default SSLSocketFactory (which effectively means not supporting self-signed certificates, since adding them by hand using keytool is ridiculously complicated for endusers).