ADDITIONAL SYSTEM INFORMATION :
HTTP server: (embedded Jetty) on Windows Server 2016 Datacenter or Windows Server 2019
HTTP client - BigIP F5 (unknown version - customer env.); IIS reverse proxy on Windows server (unknown version - customer env.), Qlik Sense Enterprise REST connector on Windows Server 2016 Datacenter or Windows Server 2019
Java runtime: one of the available builds of OpenJDK 17, also tried with Oracle Java 22.0.2 with the same result.

A DESCRIPTION OF THE PROBLEM :
When a client makes a HTTPS request to our service (which uses embedded Jetty and uses a private key in system keystore for TLS through MS CryptoAPI), under specific conditions the connection fails. With javax.net.debug logging enabled we can see that an exception was thrown in the MSCAPI code (see "Actual result" below).

Other clients (browsers) have no problem establishing a connection.



STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Once the conditions are right, the problem can be reproduced by attempting to make an HTTPS request. However, we haven't been able to establish what exactly is the factor that triggers the problem. We have seen this on three different environments, mostly on customer, one our own.

In all cases web browsers had no problem making HTTPS requests, while an BigIP F5 or IIS reverse proxy or Qlik Sense Enterprise REST connector trigger the problem.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
We expect that all clients are able to connect.
ACTUAL -
javax.net.ssl|ERROR|91|qtp953528583-25|2024-07-22 14:13:37.552 UTC|TransportContext.java:370|Fatal (INTERNAL_ERROR): Failed to sign ecdhe parameters: RSA (
"throwable" : {
  java.security.SignatureException: error 2148073494, Keyset does not exist
  
   at jdk.crypto.mscapi/sun.security.mscapi.CSignature.signHash(Native Method)
   at jdk.crypto.mscapi/sun.security.mscapi.CSignature$RSA.engineSign(CSignature.java:220)
   at java.base/java.security.Signature$Delegate.engineSign(Signature.java:1410)
   at java.base/java.security.Signature.sign(Signature.java:713)
   at java.base/sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeMessage.<init>(ECDHServerKeyExchange.java:175)
   at java.base/sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeProducer.produce(ECDHServerKeyExchange.java:490)
   at java.base/sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1108)
   at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:841)
   at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:800)
   at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393)
   at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476)
   at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1274)
   at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260)
   at java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
   at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1205)
   at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:657)
   at org.eclipse.jetty.server.HttpConnection.fillRequestBuffer(HttpConnection.java:373)
   at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
   at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
   at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
   at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:558)
   at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:379)
   at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:146)
   at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
   at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
   at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
   at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
   at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
   at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193)
   at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
   at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
   at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
   at java.base/java.lang.Thread.run(Thread.java:1570)}

CUSTOMER SUBMITTED WORKAROUND :
- Attach a debugger to the JVM and set a breakpoing in sun.security.mscapi.CSignature$RSA.engineSign (line 211 in CSignature.java -- just on the if-statement)
- force a return from the method by evaluating signCngHash(1, hash, hash.length, 0, this instanceof NONEwithRSA ? null : messageDigestAlgorithm, privateKey.getHCryptProvider(), privateKey.getHCryptKey()) -- i.e. force use of signCngHash instead of signHash

This, if done fast enough to avoid the client from timing out, results in a successful TLS handshake and connection.

FREQUENCY : always