In the context of backporting JDK-8195607 to 8u, I found that jtreg does not run tests in separated 'test.classes' directories because the TEST.ROOT file does not specify a minimum jtreg version required. Thus, Secmod tests share the same temporary NSS DB directory, even when run in different JVM instances. This has never been a problem until we have a test such as the one in JDK-8195607. See the comments related to this particular case below:

===
Comment #1 (https://github.com/openjdk/jdk8u-dev/pull/106#issue-1343866824)

I also run all the tests under the jdk/sun/security/pkcs11 category and noticed no-regressions strictly related to this fix. The test 'jdk/sun/security/pkcs11/Secmod/TrustAnchors.java' requires a special note, though. This test passes if run standalone but fails when run as part of a group. The reason is because when run as part of a group, other tests copy the file 'pkcs11.txt' to the temporary nssdb directory which is shared between all of them. This DB does not register the trust anchors module called 'Builtin Roots Module', which was available in the old secmod.db and is required by the test (which expects to see a Secmod$Module of TRUSTANCHOR type). While the test leaves the sun.security.pkcs11.SecmodTest::useSqlite field set to false and tries to access the legacy DB (also contained in the shared temporary nssdb directory), NSS realizes that there is a new SQLite DB in the same directory and reads the configuration from pkcs11.txt. Otherwise, if pkcs11.txt does not exist (as when the test is run standalone), it creates one with the information in secmod.db and it registers the 'Builtin Roots Module' module. All these findings have been verified with the 'modutil -list' command. While the issue is unrelated to what JDK-8195607 fixes, it has been uncovered by the existence of the pkcs11.txt file introduced with this change.

Why newer releases are not affected? Because jtreg runs tests (including TrustAnchors.java) in separated directories, so they don't share the nssdb temporary directory. In newer releases, TEST.ROOT has a requiredVersion attribute [1] indicating to jtreg that this can be done [2].

[1] - https://github.com/openjdk/jdk11u/blob/master/test/jdk/TEST.ROOT#L61
[2] - https://github.com/openjdk/jtreg/blob/jtreg-6.2%2B1/src/share/classes/com/sun/javatest/regtest/config/Locations.java#L174

===
Comment # 2 (https://github.com/openjdk/jdk8u-dev/pull/106#issuecomment-1227922783)

Let me clarify the jtreg issue. It's not that the test fails because it's run with an old jtreg. In fact, I'm running with jtreg 5 and it's failing. The problem is that the TEST.ROOT in JDK-8 does not set a "minimum jtreg required" (probably because it's a newer feature than JDK-8). In these cases, jtreg takes a conservative approach and does not split the text execution into different directories. Setting a "minimum jtreg required" in JDK-8 would be a high risk as we don't know the impact of other tests in newer or older jtreg releases. Having this test broken would be a regression. I hope we can have this fix, or you might come up with a different one. Let me know if that helps for a better understanding.

===

More generally, it's possible to have both a secmod.db (legacy NSS DB format) and a pkcs11.txt (new NSS DB format) files in the same directory if an SQLite Secmod test and a legacy DB Secmod test are run one after the other. The configuration in these databases could be different. The NSS library, even when pointed to the legacy DB, realizes that new DB information exists in the same directory and prefers it. Thus, the legacy DB test will experience a different configuration than expected and fail.

One way to fix this would be to set a minimum jtreg version required for JDK-8 tests (greater than 4.2 b08). A more conservative approach -and preferred in my view- would be to cleanup the NSS DB temporary directory before running a Secmod test.