ADDITIONAL SYSTEM INFORMATION :
Fedora 34
$ java -version
java version "1.8.0_301"
Java(TM) SE Runtime Environment (build 1.8.0_301-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.301-b09, mixed mode)

It can be reproduce on jdk-11.0.2 as well.

It should be reproducible on any other system with following environmental variable set:
NSS_DEFAULT_DB_TYPE="sql"

A DESCRIPTION OF THE PROBLEM :
NSS on fedora 34 does not support dmb format. Modutil, when creating a new NSS DB will create a SQL-only one by default:

$ mkdir nssdb
$ modutil -create -dbdir nssdb
$ ls nssdb
cert9.db key4.db pkcs11.txt

If you set a modutil-created NSS DB in jre/lib/security/java.security, the JDK will choke with FileNotFoundExceprion.

If you create empty secmod.db it works as expected

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
$ mkdir /tmp/nssdb
$ modutil -create -dbdir /tmp/nssdb
$ cat > /tmp/nss.cfg <<EOT
nssModule = fips
nssSecmodDirectory = /tmp/nssdb
nssLibraryDirectory = /usr/lib64
name = testPkcs
nssDbMode = readWrite
EOT

In java.security file:
add following line
security.provider.1=sun.security.pkcs11.SunPKCS11 /tmp/nss.cfg
then adjust all other providers priority ( security.provider.X -> security.provider.X+1)
then change security.provider.5=com.sun.net.ssl.internal.ssl.Provider
to security.provider.5=com.sun.net.ssl.internal.ssl.Provider SunPKCS11-nss-fips
$ modutil -fips true -dbdir /tmp/nssdb
$ keytool -list -storetype pkcs11




EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
prompt to enter password, printed keystore information
ACTUAL -
keytool error: java.security.ProviderException: Could not initialize NSS

CUSTOMER SUBMITTED WORKAROUND :
$ touch /tmp/nssdb/secmod.db

FREQUENCY : always