ADDITIONAL SYSTEM INFORMATION :
macOS Catalina 10.15.7
JDK 16

A DESCRIPTION OF THE PROBLEM :
A custom DefaultFileSystemProvider with -Djava.security.manager and -Djava.security.policy=<policy> loops in FileSystemProvider newByteChannel.

REGRESSION : Last worked in version 12

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
It can be reproduced with the JDK TestProvider class
/usr/libexec/java_home -v 12 --exec javac -d . ~/Documents/GitHub/jdk16-master/test/jdk/java/nio/file/spi/TestProvider.java

JDK 12 is the last version where it works for me. (I don't have a working 13. It doesn't work at 14).
So I compile against that.
See test class source below.

It uses the all.poicy file with...
//

grant {
    permission java.security.AllPermission;
};

Reproduce as shown in Results

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
/usr/libexec/java_home -v 12 --exec java -cp . -Djava.security.manager -Djava.security.policy=all.policy -Djava.nio.file.spi.DefaultFileSystemProvider=TestProvider org.test.Test
TestProvider$TestFileSystem@23fc625e
TestProvider

ACTUAL -
/usr/libexec/java_home -v 14 --exec java -cp . -Djava.security.manager -Djava.security.policy=Classes/all.policy -Djava.nio.file.spi.DefaultFileSystemProvider=TestProvider org.test.Test
TestProvider$TestFileSystem@23fc625e
Exception in thread "main" java.lang.StackOverflowError
at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:593)
at java.base/java.lang.StringBuilder.append(StringBuilder.java:173)
at java.base/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:269)
at TestProvider$TestFileSystem.getPath(TestProvider.java:261)
at java.base/java.nio.file.Path.of(Path.java:147)
at java.base/sun.security.provider.PolicyFile.initDefaultPolicy(PolicyFile.java:478)
at java.base/sun.security.provider.PolicyFile$2.run(PolicyFile.java:340)
at java.base/sun.security.provider.PolicyFile$2.run(PolicyFile.java:337)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:312)
at java.base/sun.security.provider.PolicyFile.initPolicyFile(PolicyFile.java:337)
at java.base/sun.security.provider.PolicyFile.init(PolicyFile.java:330)
at java.base/sun.security.provider.PolicyFile.<init>(PolicyFile.java:283)
at java.base/java.security.Policy.loadPolicyProvider(Policy.java:207)
at java.base/java.security.Policy.getPolicyNoCheck(Policy.java:178)
at java.base/java.security.ProtectionDomain.implies(ProtectionDomain.java:321)
at java.base/java.security.ProtectionDomain.impliesWithAltFilePerm(ProtectionDomain.java:353)
at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:450)
at java.base/java.security.AccessController.checkPermission(AccessController.java:1036)
at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
at java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:747)
at java.base/sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:255)
at java.base/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:143)
at java.base/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:156)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:217)
at TestProvider.newByteChannel(TestProvider.java:173)
...

---------- BEGIN SOURCE ----------

package org.test;

import java.nio.file.FileSystems;
import java.nio.file.spi.FileSystemProvider;
import java.lang.reflect.Constructor;

public class Test {

public static void main(String [] args) {
System.out.println(FileSystems.getDefault());
System.out.println(System.getProperty("java.nio.file.spi.DefaultFileSystemProvider"));
}

}

/usr/libexec/java_home -v 12 --exec javac -d . ~/testfsp/Test.java
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Run JDK12 for custom DefaultFileSystemProvider with -Djava.security.manager and -Djava.security.policy

FREQUENCY : always