-
Bug
-
Resolution: Fixed
-
P4
-
22
-
None
-
b13
Found when compiling with Clang for Windows; the following tests crash:
sun/security/mscapi/InteropWithSunRsaSign.java
sun/security/mscapi/ShortRSAKeyWithinTLS.java
The produced stack trace points inside NCryptSignHash, which is called from Java_sun_security_mscapi_CSignature_signCngHash.
The problem here is that the 2 local variables:
BCRYPT_PKCS1_PADDING_INFO pkcs1Info;
BCRYPT_PSS_PADDING_INFO pssInfo;
are scoped to the "switch" statement, but used later. Clang optimizes away writes to these variables, which in turn leads to crashes. It works just fine with the MSVC compiler, presumably its optimizer is less aggressive.
The fix involves extending the scope of these variables to the entire method.
sun/security/mscapi/InteropWithSunRsaSign.java
sun/security/mscapi/ShortRSAKeyWithinTLS.java
The produced stack trace points inside NCryptSignHash, which is called from Java_sun_security_mscapi_CSignature_signCngHash.
The problem here is that the 2 local variables:
BCRYPT_PKCS1_PADDING_INFO pkcs1Info;
BCRYPT_PSS_PADDING_INFO pssInfo;
are scoped to the "switch" statement, but used later. Clang optimizes away writes to these variables, which in turn leads to crashes. It works just fine with the MSVC compiler, presumably its optimizer is less aggressive.
The fix involves extending the scope of these variables to the entire method.