OpenJDK's Secmod class makes the assumptions that under a FIPS configuration there will be only one slot available in the NSS Internal Module [1], and that under a non-FIPS configuration there will be 2 slots [2]. This is not necessarily true (verified on NSS 3.53.1). Many Linux distributions use NSS's libnsssysinit.so initialization library. When run with a non-root user, this library ends up loading the 'NSS User database' if available (located at /home/<non-root-user>/.pki/nssdb) and the 'NSS system database' (located at /etc/pki/nssdb). In FIPS, the former DB will open the FIPS_SLOT_ID (0x3) slot, while the latter a new slot with a higher slot ID. In non-FIPS, the slots are NETSCAPE_SLOT_ID (0x1), PRIVATE_KEY_SLOT_ID (0x2) and a higher slot ID.

An example of the previous can be seen with a 'modutil' modules listing ('modutil -list -dbdir /etc/pki/nssdb' command).

FIPS case:

Listing of PKCS #11 Modules
-----------------------------------------------------------
  1. NSS Internal Crypto Services
uri: pkcs11:library-manufacturer=Mozilla%20Foundation;library-description=NSS%20Internal%20Crypto%20Services;library-version=3.53
slots: 2 slots attached
status: loaded

slot: NSS FIPS 140-2 User Private Key Services
token: NSS FIPS 140-2 Certificate DB
uri: pkcs11:token=NSS%20FIPS%20140-2%20Certificate%20DB;manufacturer=Mozilla%20Foundation;serial=0000000000000000;model=NSS%203

slot: NSS Application Slot 00000065
token: NSS system database
uri: pkcs11:token=NSS%20system%20database;manufacturer=Mozilla%20Foundation;serial=0000000000000000;model=NSS%203

-----------------------------------------------------------

Non-FIPS case:

Listing of PKCS #11 Modules
-----------------------------------------------------------
  1. NSS Internal Crypto Services
uri: pkcs11:library-manufacturer=Mozilla%20Foundation;library-description=NSS%20Internal%20Crypto%20Services;library-version=3.53
slots: 3 slots attached
status: loaded

slot: NSS Internal Cryptographic Services
token: NSS Generic Crypto Services
uri: pkcs11:token=NSS%20Generic%20Crypto%20Services;manufacturer=Mozilla%20Foundation;serial=0000000000000000;model=NSS%203

slot: NSS User Private Key and Certificate Services
token: NSS Certificate DB
uri: pkcs11:token=NSS%20Certificate%20DB;manufacturer=Mozilla%20Foundation;serial=0000000000000000;model=NSS%203

slot: NSS Application Slot 00000004
token: NSS system database
uri: pkcs11:token=NSS%20system%20database;manufacturer=Mozilla%20Foundation;serial=0000000000000000;model=NSS%203

Note: to reproduce this behavior, check that:

 1) You run 'modutil' with a non-root user

 2) You have a local NSSDB in /home/<non-root-user>/.pki/nssdb

 3) You have 'library=libnsssysinit.so' in /etc/pki/nssdb/pkcs11.txt (use 'sudo setup-nsssysinit on' command in RHEL)

OpenJDK is currently unable to handle this configuration, and either: an exception is thrown in FIPS mode [3]; or the slot is considered of ModuleType.KEYSTORE type when it may be not [4]. In the latter case, looks to me that only one ModuleType.KEYSTORE slot was expected [5][6] and the real ModuleType.KEYSTORE slot may not be used depending on the initialization order (there will be 2 Secmod::Module instances of ModuleType.KEYSTORE type in the modules list [7]). Even if it's used by chance, the assumption was broken and may cause a problem in the future.

Just for the record, the NSS modules information is obtained by OpenJDK in [8], and Secmod::Module instances created in [9].

One possible approach to fix this issue would be to pick the slots with expected slot IDs only: FIPS_SLOT_ID, NETSCAPE_SLOT_ID and PRIVATE_KEY_SLOT_ID. The extra slots would be ignored while creating Secmod::Module instances. This would continue OpenJDK's previous behavior. The new extra slots would not be available for use from OpenJDK, though. It would be up to the user to configure NSS so OpenJDK uses the intended NSSDBs. Some possible NSS configurations include:

1) Do not use libnsssysinit.so, or use a different initialization library to change the order of the NSSDBs (the first one to be initialized gets the FIPS_SLOT_ID, NETSCAPE_SLOT_ID and PRIVATE_KEY_SLOT_ID slot IDs)

2) Delete the local NSSDB if it's not needed

3) Store keys needed from OpenJDK in the local NSSDB if it's needed.

In other words, the user must configure NSS as needed with the understanding that OpenJDK will use FIPS_SLOT_ID, NETSCAPE_SLOT_ID and PRIVATE_KEY_SLOT_ID slot IDs.

In the future we can consider an enhancement on the OpenJDK side so it lets the user to pick the slot among several slots of the same type in the 'NSS Internal Module' module. For example, by extending SunPKCS11 configuration syntax. Having said that, I'd first focus on fixing this as a bug and keeping the current behavior working -which is particularly needed for update releases-.

--
[1] - https://github.com/openjdk/jdk/blob/ebbce91e79ff5e145166b01310d4a1ebbbbd5178/src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/Secmod.java#L405
[2] - https://github.com/openjdk/jdk/blob/ebbce91e79ff5e145166b01310d4a1ebbbbd5178/src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/Secmod.java#L402
[3] - https://github.com/openjdk/jdk/blob/ebbce91e79ff5e145166b01310d4a1ebbbbd5178/src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/Secmod.java#L406
[4] - https://github.com/openjdk/jdk/blob/ebbce91e79ff5e145166b01310d4a1ebbbbd5178/src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/Secmod.java#L402
[5] - https://github.com/openjdk/jdk/blob/ebbce91e79ff5e145166b01310d4a1ebbbbd5178/src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/SunPKCS11.java#L240
[6] - https://github.com/openjdk/jdk/blob/ebbce91e79ff5e145166b01310d4a1ebbbbd5178/src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/Secmod.java#L338
[7] - https://github.com/openjdk/jdk/blob/ebbce91e79ff5e145166b01310d4a1ebbbbd5178/src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/SunPKCS11.java#L220
[8] - https://github.com/openjdk/jdk/blob/b44c24d290362e4edf5b0bf18b1ecce1583daeff/src/jdk.crypto.cryptoki/share/native/libj2pkcs11/j2secmod.c#L180
[9] - https://github.com/openjdk/jdk/blob/b44c24d290362e4edf5b0bf18b1ecce1583daeff/src/jdk.crypto.cryptoki/share/native/libj2pkcs11/j2secmod.c#L235