During class resolution, the JVM calls ClassLoader.checkPackageAccess(), and then updates DictionaryEntry::_pd_set accordingly.

See
http://hg.openjdk.java.net/jdk/hs/file/5264a11d3753/src/hotspot/share/classfile/systemDictionary.cpp#l438

    JavaCalls::call_special(&result,
                         class_loader,
                         system_loader,
                         vmSymbols::checkPackageAccess_name(),
                         ...);
    ....
    if (HAS_PENDING_EXCEPTION) return;
    ....
    dictionary->add_protection_domain(d_index, d_hash, klass,
                                      protection_domain, THREAD);

However, when the security manager is not installed, ClassLoader.checkPackageAccess() does nothing:

http://hg.openjdk.java.net/jdk/hs/file/5264a11d3753/src/java.base/share/classes/java/lang/ClassLoader.java#l669

    private void checkPackageAccess(Class<?> cls, ProtectionDomain pd) {
        final SecurityManager sm = System.getSecurityManager();
        if (sm != null) {
            ... do some checks ....
        }
    }

So the JVM is making all these Java upcalls and maintaining a complicated cache for nothing. This causes slow down both in class loading time, as well as in GC pauses.

Preliminary benchmarking shows about 1.5% speed up when running hello world in clojure.