JDK-8307106 introduced the ability to walk the created CLDs list in the CLDG without a lock. This was primarily introduced to allow lockless concurrent CLD roots scanning for young collections in generational ZGC. However because the CLD _next node pointer is shared between the two list this can lead to a concurrent iteration of the created CLDs missing list entries. Specifically a young root scanning which occurs concurrently with class unloading.

This change introduces a second _unloading_next node pointer which is used for the unloading CLDs list. The set_next is now maintains the invariant that it only ever unlinks is_unloading() CLDs and maintains a consistent view of the tail list for anyone reading the list concurrently

Below is the crash which is a symptom of this. What effectively happens is that some CLD roots are not traced due to the race described above, which means that the young collection ends is mark phase with a broken object graph (some roots are not the correctly coloured/proccessed). This then confuses the load barriers in the verification step which is run before the young collection enters its relocation phase to try an relocated the object, which then catches the bug with assert(_generation->is_phase_relocate()).

The following test failed in the JDK21 CI:

gc/stringdedup/TestStringDeduplicationPrintOptions.java#Z

Here's a snippet from the log file:

#section:driver
----------messages:(8/405)----------
command: driver gc.stringdedup.TestStringDeduplicationPrintOptions Z
reason: User specified action: run driver gc.stringdedup.TestStringDeduplicationPrintOptions Z
started: Thu May 18 19:48:00 UTC 2023
Mode: agentvm
Agent id: 9
Additional opens to unnamed modules from @modules: java.base/jdk.internal.misc java.base/java.lang
finished: Thu May 18 19:48:05 UTC 2023
elapsed time (seconds): 5.182
----------configuration:(18/1749)*----------

<snip>

----------System.out:(29/3505)*----------
Command line: [c:\\ade\\mesos\\work_dir\\jib-master\\install\\jdk-21+24-2008\\windows-x64-debug.jdk\\jdk-21\\fastdebug\\bin\\java.exe -cp C:\\sb\\prod\\1684439107\\testoutput\\test-support\\jtreg_open_test_hotspot_jtreg_hotspot_gc\\classes\\5\\gc\\stringdedup\\TestStringDeduplicationPrintOptions_Z.d;C:\\ade\\mesos\\work_dir\\jib-master\\install\\jdk-21+24-2008\\src.full\\open\\test\\hotspot\\jtreg\\gc\\stringdedup;C:\\sb\\prod\\1684439107\\testoutput\\test-support\\jtreg_open_test_hotspot_jtreg_hotspot_gc\\classes\\5;C:\\sb\\prod\\1684439107\\testoutput\\test-support\\jtreg_open_test_hotspot_jtreg_hotspot_gc\\classes\\5\\test\\lib;C:\\ade\\mesos\\work_dir\\jib-master\\install\\jtreg\\7.2\\1\\bundles\\jtreg-7.2+1.zip\\jtreg\\lib\\jtreg.jar;C:\\ade\\mesos\\work_dir\\jib-master\\install\\jtreg\\7.2\\1\\bundles\\jtreg-7.2+1.zip\\jtreg\\lib\\junit-platform-console-standalone-1.9.2.jar;C:\\ade\\mesos\\work_dir\\jib-master\\install\\jtreg\\7.2\\1\\bundles\\jtreg-7.2+1.zip\\jtreg\\lib\\testng-7.3.0.jar;C:\\ade\\mesos\\work_dir\\jib-master\\install\\jtreg\\7.2\\1\\bundles\\jtreg-7.2+1.zip\\jtreg\\lib\\jcommander-1.78.jar;C:\\ade\\mesos\\work_dir\\jib-master\\install\\jtreg\\7.2\\1\\bundles\\jtreg-7.2+1.zip\\jtreg\\lib\\guice-4.2.3.jar -XX:MaxRAMPercentage=4.16667 -Dtest.boot.jdk=c:\\ade\\mesos\\work_dir\\jib-master\\install\\jdk\\20\\36\\bundles\\windows-x64\\jdk-20_windows-x64_bin.zip\\jdk-20 -Djava.io.tmpdir=c:\\sb\\prod\\1684439107\\testoutput\\test-support\\jtreg_open_test_hotspot_jtreg_hotspot_gc\\tmp -XX:+CreateCoredumpOnCrash -XX:+UseZGC -XX:+ZGenerational -XX:+UseZGC -Xmn50m -Xms100m -Xmx100m -XX:+UnlockDiagnosticVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED -XX:+VerifyAfterGC -XX:+UseStringDeduplication -XX:StringDeduplicationAgeThreshold=3 gc.stringdedup.TestStringDeduplicationTools$DeduplicationTest 10 5 3 YoungGC ]
[2023-05-18T19:48:00.275380500Z] Gathering output for process 43868
Begin: DeduplicationTest
Creating strings: total=10, unique=5
Creating strings: total=10, unique=10
Begin: Young GC 1/6
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (c:\\sb\\prod\\1684436204\\workspace\\open\\src\\hotspot\\share\\gc\\z\\zRelocate.cpp:378), pid=43868, tid=38256
# assert(_generation->is_phase_relocate()) failed: Must be
#
# JRE version: Java(TM) SE Runtime Environment (21.0+24) (fastdebug build 21-ea+24-LTS-2008)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 21-ea+24-LTS-2008, mixed mode, sharing, tiered, compressed class ptrs, z gc, windows-amd64)
# Core dump will be written. Default location: C:\\sb\\prod\\1684439107\\testoutput\\test-support\\jtreg_open_test_hotspot_jtreg_hotspot_gc\\scratch\\0\\hs_err_pid43868.mdmp
#
# An error report file with more information is saved as:
# C:\\sb\\prod\\1684439107\\testoutput\\test-support\\jtreg_open_test_hotspot_jtreg_hotspot_gc\\scratch\\0\\hs_err_pid43868.log
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#

[2023-05-18T19:48:05.089860300Z] Waiting for completion for process 43868
[2023-05-18T19:48:05.150305Z] Waiting for completion finished for process 43868
[2023-05-18T19:48:05.150305Z] Waiting for completion for process 43868
[2023-05-18T19:48:05.150871100Z] Waiting for completion finished for process 43868
[2023-05-18T19:48:05.176122800Z] Waiting for completion for process 43868
[2023-05-18T19:48:05.176688700Z] Waiting for completion finished for process 43868
----------System.err:(37/1963)*----------

 stdout: [Begin: DeduplicationTest
Creating strings: total=10, unique=5
Creating strings: total=10, unique=10
Begin: Young GC 1/6
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (c:\\sb\\prod\\1684436204\\workspace\\open\\src\\hotspot\\share\\gc\\z\\zRelocate.cpp:378), pid=43868, tid=38256
# assert(_generation->is_phase_relocate()) failed: Must be
#
# JRE version: Java(TM) SE Runtime Environment (21.0+24) (fastdebug build 21-ea+24-LTS-2008)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 21-ea+24-LTS-2008, mixed mode, sharing, tiered, compressed class ptrs, z gc, windows-amd64)
# Core dump will be written. Default location: C:\\sb\\prod\\1684439107\\testoutput\\test-support\\jtreg_open_test_hotspot_jtreg_hotspot_gc\\scratch\\0\\hs_err_pid43868.mdmp
#
# An error report file with more information is saved as:
# C:\\sb\\prod\\1684439107\\testoutput\\test-support\\jtreg_open_test_hotspot_jtreg_hotspot_gc\\scratch\\0\\hs_err_pid43868.log
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#
];
 stderr: []
 exitValue = 1

java.lang.RuntimeException: Expected to get exit value of [0], exit value is: [1]
at jdk.test.lib.process.OutputAnalyzer.shouldHaveExitValue(OutputAnalyzer.java:490)
at gc.stringdedup.TestStringDeduplicationTools.testPrintOptions(TestStringDeduplicationTools.java:517)
at gc.stringdedup.TestStringDeduplicationPrintOptions.main(TestStringDeduplicationPrintOptions.java:94)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:578)
at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:333)
at java.base/java.lang.Thread.run(Thread.java:1583)

JavaTest Message: Test threw exception: java.lang.RuntimeException
JavaTest Message: shutting down test

result: Failed. Execution failed: `main' threw exception: java.lang.RuntimeException: Expected to get exit value of [0], exit value is: [1]