We run into assertion errors like this one in the jtreg tests sun/security/pkcs11/Signature/SignatureTestPSS2.java and sun/security/pkcs11/Signature/SigInteropPSS2.java .

java.lang.AssertionError
        at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.CK_MECHANISM.setParameter(CK_MECHANISM.java:167)
        at jdk.crypto.cryptoki/sun.security.pkcs11.P11PSSSignature.ensureInitialized(P11PSSSignature.java:234)
        at jdk.crypto.cryptoki/sun.security.pkcs11.P11PSSSignature.engineUpdate(P11PSSSignature.java:559)
        at java.base/java.security.Signature$Delegate.engineUpdate(Signature.java:1394)
        at java.base/java.security.Signature.update(Signature.java:902)
        at java.base/java.security.Signature.update(Signature.java:871)
        at SignatureTestPSS2.checkSignature(SignatureTestPSS2.java:115)
        at SignatureTestPSS2.test(SignatureTestPSS2.java:92)
        at SignatureTestPSS2.runTest(SignatureTestPSS2.java:85)
        at SignatureTestPSS2.main(SignatureTestPSS2.java:71)
        at PKCS11Test.premain(PKCS11Test.java:174)
        at PKCS11Test.testNSS(PKCS11Test.java:550)
        at PKCS11Test.main(PKCS11Test.java:210)
        at SignatureTestPSS2.main(SignatureTestPSS2.java:56)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        at java.base/java.lang.reflect.Method.invoke(Method.java:577)
        at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
        at java.base/java.lang.Thread.run(Thread.java:833)
 
when removing the assertion assert(this.mechanism == CKM_RSA_PKCS_PSS); at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.CK_MECHANISM.setParameter , the test passes .
On a RedHat 8 machine where it fails, "this.mechanism" in the assertion is 71 (47hex) which seems to be CKM_SHA224_RSA_PKCS_PSS .

The RedHat 8 machine where the test fails seems to have (from the yum package output) NSS 3.67.0-6.el8_4 .
The test fails too on SUSE linux SLES 12.X machines, NSS version there is mozilla-nss-3.68-58.54.1.x86_64 .