JCK test is testing the below assertion from the Signature#initSign
"InvalidKeyException - if the key is invalid"

This test is failing on solaris sparcv9 platform.

On solaris , the SunPKCS11 provider is out of the box configured as below:

Shared library is : /usr/lib/libpkcs11.so
Configuration: {jdk_home}/conf/security/sunpkcs11-solaris.cfg

Test Code:
I have an invalid key:

PrivateKey invalidKey = new PrivateKey() {
                                    public String getAlgorithm() {
                                        return "Fake";
                                    }
                                    public byte[] getEncoded() {
                                        return "Fake".getBytes();
                                    }
                                    public String getFormat() {
                                        return "Fake";
                                    }
                            };

Signature sig = Signature.getInstance("SHA1withRSA", "SunPKCS11-Solaris");
sig.initSign(invalidKey );

As per the specification i am expecting:
"InvalidKeyException "

However i receive the following error: (tested with JDK9b181 and JDK10b42)
===============================================
Exception in thread "main" java.lang.ClassCastException: SigPKCS11Test$1 cannot be cast to java.base/java.security.interfaces.RSAKey
        at jdk.crypto.cryptoki/sun.security.pkcs11.P11Signature.checkKeySize(P11Signature.java:377)
        at jdk.crypto.cryptoki/sun.security.pkcs11.P11Signature.engineInitSign(P11Signature.java:455)
        at java.base/java.security.Signature$Delegate.engineInitSign(Signature.java:1200)
        at java.base/java.security.Signature.initSign(Signature.java:545)
================================================

Similarly if i construct an Invalid PublicKey as above and invoke Signture#initVerify passing that invalid public key, the CCE is thrown instead of InvalidKeyException

PS:
I could not test this on Linux/Windows , because by default the SunPKCS11 provider is unconfigured and unusable in those platforms.

I am not really sure which dll/ so files needs to be properly installed or configured in those machines. Hence i cannot say if this error on those platforms is reproducible .