This issue is similar to JDK-8155090, but with lambdas:

public class WithLambdas {
    public static void main(String[] args) throws Throwable {
        SecurityManager sm = new SecurityManager() {
            @Override
            public void checkPermission(Permission perm) {
                  Runnable r = () -> System.out.println("Boo");
            }
        };
        System.setSecurityManager(sm);
        ClassLoader cl = new ClassLoader() {};
    }
}

...fails with bootstrapping errors, because lambda linkage requires java.lang.invoke, which does privileged calls, which doubles back on custom SecurityManager. This eventually fails with the stack overflow:

java.lang.BootstrapMethodError: call site initialization exception
at java.lang.invoke.CallSite.makeSite(java.base@9-internal/CallSite.java:347)
at java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(java.base@9-internal/MethodHandleNatives.java:250)
at java.lang.invoke.MethodHandleNatives.linkCallSite(java.base@9-internal/MethodHandleNatives.java:240)
at WithLambdas$1.checkPermission(WithLambdas.java:40)
at java.lang.Class.checkMemberAccess(java.base@9-internal/Class.java:2585)
at java.lang.Class.getDeclaredConstructors(java.base@9-internal/Class.java:2197)
at java.lang.invoke.InnerClassLambdaMetafactory$1.run(java.base@9-internal/InnerClassLambdaMetafactory.java:198)
at java.lang.invoke.InnerClassLambdaMetafactory$1.run(java.base@9-internal/InnerClassLambdaMetafactory.java:195)
at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method)
at java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(java.base@9-internal/InnerClassLambdaMetafactory.java:194)
at java.lang.invoke.LambdaMetafactory.metafactory(java.base@9-internal/LambdaMetafactory.java:304)
at java.lang.invoke.CallSite.makeSite(java.base@9-internal/CallSite.java:308)
at java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(java.base@9-internal/MethodHandleNatives.java:250)
at java.lang.invoke.MethodHandleNatives.linkCallSite(java.base@9-internal/MethodHandleNatives.java:240)
at WithLambdas$1.checkPermission(WithLambdas.java:40)
at java.lang.Class.checkMemberAccess(java.base@9-internal/Class.java:2585)
at java.lang.Class.getDeclaredConstructors(java.base@9-internal/Class.java:2197)
at java.lang.invoke.InnerClassLambdaMetafactory$1.run(java.base@9-internal/InnerClassLambdaMetafactory.java:198)
at java.lang.invoke.InnerClassLambdaMetafactory$1.run(java.base@9-internal/InnerClassLambdaMetafactory.java:195)
at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method)
at java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(java.base@9-internal/InnerClassLambdaMetafactory.java:194)
at java.lang.invoke.LambdaMetafactory.metafactory(java.base@9-internal/LambdaMetafactory.java:304)
at java.lang.invoke.CallSite.makeSite(java.base@9-internal/CallSite.java:308)
at java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(java.base@9-internal/MethodHandleNatives.java:250)
at java.lang.invoke.MethodHandleNatives.linkCallSite(java.base@9-internal/MethodHandleNatives.java:240)
at WithLambdas$1.checkPermission(WithLambdas.java:40)