In CDS terms, "Unregistered classes" are the archived classes that should be loaded by custom class loaders during runtime. Archiving of class loader constraints (JDK-8178349) should be disabled for such classes.

During CDS dump time (-Xshare:dump), unregistered classes are loaded by the boot loader (see also JDK-8261941). SystemDictionaryShared::record_linking_constraint doesn't properly recognize such classes, leading to the following assert.

#
# Internal Error (/jdk2/quo/open/src/hotspot/share/classfile/systemDictionaryShared.cpp:1898), pid=14653, tid=14654
# assert(klass_loader != __null) failed: should not be called for boot loader
#
# JRE version: Java(TM) SE Runtime Environment (17.0) (fastdebug build 17-internal+0-adhoc.iklam.open)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 17-internal+0-adhoc.iklam.open, interpreted mode, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V [libjvm.so+0x1841be4] SystemDictionaryShared::record_linking_constraint(Symbol*, InstanceKlass*, Handle, Handle)+0x384
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P" (or dumping to /jdk2/tmp/jtreg/work/scratch/core.14653)
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#

--------------- S U M M A R Y ------------

Command Line: -Xshare:dump -XX:SharedClassListFile=eclipse.list -XX:SharedArchiveFile=eclipse.jsa -Xlog:cds /home/iklam/eclipse/java-photon-R/eclipse/plugins/org.eclipse.equinox.launcher_1.5.0.v20180512-1130.jar

Host: ioilinux, Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz, 32 cores, 62G, Ubuntu 16.04.6 LTS
Time: Tue May 18 14:28:06 2021 PDT elapsed time: 2.643222 seconds (0d 0h 0m 2s)

--------------- T H R E A D ---------------

Current thread (0x00007f0dc0028a90): JavaThread "main" [_thread_in_vm, id=14654, stack(0x00007f0dca591000,0x00007f0dca692000)]

Stack: [0x00007f0dca591000,0x00007f0dca692000], sp=0x00007f0dca68e1d0, free space=1012k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x1841be4] SystemDictionaryShared::record_linking_constraint(Symbol*, InstanceKlass*, Handle, Handle)+0x384
V [libjvm.so+0x182e942] SystemDictionary::add_loader_constraint(Symbol*, Klass*, Handle, Handle)+0x2e2
V [libjvm.so+0x182eb16] SystemDictionary::check_signature_loaders(Symbol*, Klass*, Handle, Handle, bool)+0xb6
V [libjvm.so+0x122599f] klassItable::check_constraints(GrowableArray<Method*>*, Thread*)+0x56f
V [libjvm.so+0x1226711] klassItable::initialize_itable_and_check_constraints(Thread*)+0x1b1
V [libjvm.so+0xe281d3] InstanceKlass::link_class_impl(Thread*)+0x5f3
V [libjvm.so+0x1448732] MetaspaceShared::try_link_class(Thread*, InstanceKlass*)+0xf2
V [libjvm.so+0x976b44] ClassListParser::parse(Thread*)+0x6e4
V [libjvm.so+0x14496d1] MetaspaceShared::preload_classes(Thread*)+0x71
V [libjvm.so+0x144bcd3] MetaspaceShared::preload_and_dump_impl(Thread*)+0x13
V [libjvm.so+0x144bf7b] MetaspaceShared::preload_and_dump()+0x10b
V [libjvm.so+0x18a009d] Threads::create_vm(JavaVMInitArgs*, bool*)+0xa0d
V [libjvm.so+0xfa5a78] JNI_CreateJavaVM+0x98
C [libjli.so+0x3a7f] JavaMain+0x8f
C [libjli.so+0x7ea9] ThreadJavaMain+0x9