ADDITIONAL SYSTEM INFORMATION :
Windows Server 2016
JRE 17 (and later)
Wildfly 26.1.3
C# app connecting to Wildfly 26.1.3 running with OpenJDK 17.

A DESCRIPTION OF THE PROBLEM :
This bug report emphasizes the existing bug report JDK-8337122.
We stumbled on this on our windows servers also with OpenJDK 17 (and I suppose it never worked with any JDK).
By re-compiling the current OpenJDK 17, changing CSignature.java, we could confirm that using the signCngHash instead of signHash resolves the issue. Nevertheless, this needs a close look, since the if-clause and the signHash call are for sure relevant for other use cases.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Connect to server and perform the TLS handshake (as written above and in JDK-8337122, we can reproduce it on our systems, but the specific conditions are not 100% clear)

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
This class is: SHA256withRSA

 javax.net.ssl|DEBUG|17|default task-2|2025-05-20 13:54:20.114 CEST|ECDHServerKeyExchange.java:492|Produced ECDH ServerKeyExchange handshake message (
 "ECDH ServerKeyExchange": {
   "parameters": {
     "named group": "x25519"
     "ecdh public": {
(...)
     },
   },
   "digital signature": {
     "signature algorithm": "rsa_pkcs1_sha256"
ACTUAL -
This class is: SHA256withRSA

 javax.net.ssl|ERROR|27|default task-1|2025-05-19 15:26:36.727 CEST|TransportContext.java:375|Fatal (INTERNAL_ERROR): Failed to sign ecdhe parameters: RSA (
 "throwable" : {
   java.security.SignatureException: error 2148073494, Keyset does not exist
   
    at jdk.crypto.mscapi/sun.security.mscapi.CSignature.signHash(Native Method)
    at jdk.crypto.mscapi/sun.security.mscapi.CSignature$RSA.engineSign(CSignature.java:222)
    at java.base/java.security.Signature$Delegate.engineSign(Signature.java:1423)
    at java.base/java.security.Signature.sign(Signature.java:712)
    at java.base/sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeMessage.<init>(ECDHServerKeyExchange.java:175)
    at java.base/sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeProducer.produce(ECDHServerKeyExchange.java:489)
    at java.base/sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1107)
    at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:840)
    at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:799)
    at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
    at io.undertow.core@2.2.19.Final//io.undertow.protocols.ssl.SslConduit$5.run(SslConduit.java:1136)
    at org.jboss.threads@2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
    at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
    at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
    at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
    at org.jboss.xnio@3.8.16.Final//org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)

---------- BEGIN SOURCE ----------
Workaround:

In line 211 of CSignature.java, add (included a small debugging statement and then the workaround as described in JDK-8337122):

System.err.println("This class is: " + this.getClass().getSimpleName());
return signCngHash(1, hash, hash.length,
    0,
this instanceof NONEwithRSA ? null : messageDigestAlgorithm,
    privateKey.getHCryptProvider(), privateKey.getHCryptKey());

and remove the whole if and else clause (original lines 211-227)
---------- END SOURCE ----------