A DESCRIPTION OF THE REQUEST :
Requesting support for doing raw RSA signing with SunMSCAPI - at present only SHA-1, MD5 and MD2 signing is possible.

This would include the SunMSCAPI provider having the following property or similar:

Signature.NONEwithRSA=sun.security.mscapi.RSASignature$NONE

Alternatively, the SunMSCAPI cipher class could be changed to support encrypting with private keys, although i suspect that the underlying MSCAPI doesn't allow that.

JUSTIFICATION :
NONEwithRSA Signing is required by JSSE for client authentication enabled SSL. At the moment because SunMSCAPI doesn't include a NONEwithRSA property, the default NONEwithRSA signature class is being used. This class merely wraps the cipher class - and the MSCAPI cipher class doesn't support encrypting with private keys. A "Bad Key" exception results.

See: http://forum.java.sun.com/thread.jspa?threadID=5188515&tstart=0

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
After setting a client application to use the Windows key and trust stores in the normal manner, and given that the server and client trust each other's certificates, using JSSE to connect to a server using SSL with client authentication should work.
ACTUAL -
The following error is thrown during the client verification stage, on the client end:

Exception in thread "main" javax.net.ssl.SSLException: java.security.ProviderException: java.security.KeyException: Bad Key.
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:190)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1520)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1487)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1470)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1396)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:64)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
at SunMSCAPITest.main(SunMSCAPITest.java:37)
Caused by: java.security.ProviderException: java.security.KeyException: Bad Key.

at sun.security.mscapi.RSACipher.doFinal(RSACipher.java:242)
at sun.security.mscapi.RSACipher.engineDoFinal(RSACipher.java:266)
at javax.crypto.Cipher.doFinal(DashoA13*..)
at java.security.Signature$CipherAdapter.engineSign(Signature.java:1225)
at java.security.Signature$Delegate.engineSign(Signature.java:1128)
at java.security.Signature.sign(Signature.java:522)
at com.sun.net.ssl.internal.ssl.RSASignature.engineSign(RSASignature.java:149)
at java.security.Signature$Delegate.engineSign(Signature.java:1128)
at java.security.Signature.sign(Signature.java:522)
at com.sun.net.ssl.internal.ssl.HandshakeMessage$CertificateVerify.<init>(HandshakeMessage.java:1216)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:715)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:197)
at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:511)
at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:449)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:817)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1029)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:621)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
... 6 more
Caused by: java.security.KeyException: Bad Key.

at sun.security.mscapi.RSACipher.encryptDecrypt(Native Method)
at sun.security.mscapi.RSACipher.doFinal(RSACipher.java:222)
... 23 more

---------- BEGIN SOURCE ----------
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.io.Writer;
import java.net.Socket;

import javax.net.ssl.SSLSocketFactory;

public class SunMSCAPITest {

// Website to access
public static final String TARGET_HTTPS_SERVER = "localhost";
public static final int TARGET_HTTPS_PORT = 8443;

public static void main(String[] args) throws Exception {

System.setProperty("javax.net.ssl.keyStoreProvider","SunMSCAPI");
System.setProperty("javax.net.ssl.keyStoreType","Windows-MY");
System.setProperty("javax.net.ssl.trustStoreProvider","SunMSCAPI");
System.setProperty("javax.net.ssl.trustStoreType","Windows-ROOT");

Socket socket = SSLSocketFactory.getDefault().createSocket(TARGET_HTTPS_SERVER,
TARGET_HTTPS_PORT);

try {
Writer out = new OutputStreamWriter(socket.getOutputStream(),
"ISO-8859-1");
out.write("GET / HTTP/1.1\r\n");
out.write("Host: " + TARGET_HTTPS_SERVER + ":" + TARGET_HTTPS_PORT
+ "\r\n");
out.write("Agent: SSL-TEST\r\n");
out.write("\r\n");
out.flush();
BufferedReader in = new BufferedReader(new InputStreamReader(socket
.getInputStream(), "ISO-8859-1"));
String line = null;
while ((line = in.readLine()) != null) {
System.out.println(line);
}
} finally {
socket.close();
}
}
}
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
At the moment this can be worked around by setting both the client and server applications to use one of the other signature types instead of NONEwithRSA:

i.e.
Provider p = Security.getProvider("SunMSCAPI");
p.setProperty("Signature.NONEwithRSA","sun.security.mscapi.RSASignature$SHA1");

This isn't always possible to do on the server end though, and breaks SSL compatibility.