Kitchensink fails with: assert(!method->is_old()) failed: Should not be installing old methods

Kitchensink test with the Instrumentation module enabled can fail with the following assert in the compiled code:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (/opt/mach5/mesos/work_dir/slaves/805146e6-8fdb-4552-bf9e-385b73cf7129-S235/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/c5812956-db92-4493-b2ef-110650bcdca6/runs/9f403178-bf07-4927-851f-76543ee49c1d/workspace/open/src/hotspot/share/code/nmethod.cpp:2328), pid=17594, tid=17608
# assert(!method->is_old()) failed: Should not be installing old methods
#
# JRE version: Java(TM) SE Runtime Environment (15.0) (fastdebug build 15-internal+0-2020-05-15-1614430.serguei.spitsyn.jdk15.2)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 15-internal+0-2020-05-15-1614430.serguei.spitsyn.jdk15.2, compiled mode, sharing, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# V [libjvm.so+0x1325fc4] nmethod::metadata_do(MetadataClosure*)+0x7b4
#
# Core dump will be written. Default location: Core dumps may be processed with "/opt/core.sh %p" (or dumping to /opt/mach5/mesos/work_dir/slaves/805146e6-8fdb-4552-bf9e-385b73cf7129-S289/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/45a8c92b-127f-4604-a23c-44c322d60174/runs/6244c07e-518c-4ba5-b079-722ac9cbf54e/testoutput/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_Kitchensink_java/scratch/0/core.17594)
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#

--------------- S U M M A R Y ------------

Command Line: -Xbootclasspath/a:/opt/mach5/mesos/work_dir/slaves/805146e6-8fdb-4552-bf9e-385b73cf7129-S289/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/45a8c92b-127f-4604-a23c-44c322d60174/runs/6244c07e-518c-4ba5-b079-722ac9cbf54e/testoutput/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_Kitchensink_java/scratch/0/wb.jar -XX:+UnlockDiagnosticVMOptions -XX:+WhiteBoxAPI -XX:MaxRAMPercentage=6 -Xlog:class+redefine*=info -Xcomp -ea -esa -XX:CompileThreshold=100 -XX:+UnlockExperimentalVMOptions -XX:-TieredCompilation -XX:+CreateCoredumpOnCrash -XX:MaxRAMPercentage=50 -XX:MaxMetaspaceSize=512m -XX:+HeapDumpOnOutOfMemoryError -XX:+CrashOnOutOfMemoryError -Djava.net.preferIPv6Addresses=false -XX:+DisplayVMOutputToStderr -Xlog:gc*,gc+heap=debug:gc.log:uptime,timemillis,level,tags -XX:+DisableExplicitGC -XX:+StartAttachListener --illegal-access=deny --add-modules=jdk.incubator.foreign --add-exports=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-exports=java.base/jdk.internal.org.objectweb.asm=ALL-UNNAMED --add-exports=java.xml/com.sun.org.apache.xerces.internal.parsers=ALL-UNNAMED --add-exports=java.xml/com.sun.org.apache.xerces.internal.util=ALL-UNNAMED -Djava.io.tmpdir=/opt/mach5/mesos/work_dir/slaves/805146e6-8fdb-4552-bf9e-385b73cf7129-S289/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/45a8c92b-127f-4604-a23c-44c322d60174/runs/6244c07e-518c-4ba5-b079-722ac9cbf54e/testoutput/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_Kitchensink_java/scratch/0/java.io.tmpdir -Duser.home=/opt/mach5/mesos/work_dir/slaves/805146e6-8fdb-4552-bf9e-385b73cf7129-S289/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/45a8c92b-127f-4604-a23c-44c322d60174/runs/6244c07e-518c-4ba5-b079-722ac9cbf54e/testoutput/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_Kitchensink_java/scratch/0/user.home -agentpath:/opt/mach5/mesos/work_dir/jib-master/install/2020-05-15-1614430.serguei.spitsyn.jdk15.2/linux-x64-debug.test/hotspot/jtreg/native/libJvmtiStressModule.so -Xverify:all -javaagent:redefineagent.jar -XX:NativeMemoryTracking=detail -Djdk.test.lib.random.seed=1910729248517374629 applications.kitchensink.process.stress.Main /opt/mach5/mesos/work_dir/slaves/805146e6-8fdb-4552-bf9e-385b73cf7129-S289/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/45a8c92b-127f-4604-a23c-44c322d60174/runs/6244c07e-518c-4ba5-b079-722ac9cbf54e/testoutput/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_Kitchensink_java/scratch/0/kitchensink.final.properties

Host: ol7-build-test-205274, Intel(R) Xeon(R) Platinum 8167M CPU @ 2.00GHz, 8 cores, 58G, Oracle Linux Server release 7.8
Time: Fri May 15 17:19:41 2020 UTC elapsed time: 1224.225740 seconds (0d 0h 20m 24s)

--------------- T H R E A D ---------------

Current thread (0x00007f447831b800): JavaThread "C2 CompilerThread0" daemon [_thread_in_vm, id=17608, stack(0x00007f442477a000,0x00007f442487b000)]


Current CompileTask:
C2:1224225 71066 b java.lang.ref.Reference::get (5 bytes)

Stack: [0x00007f442477a000,0x00007f442487b000], sp=0x00007f4424875fb0, free space=1007k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x1325fc4] nmethod::metadata_do(MetadataClosure*)+0x7b4
V [libjvm.so+0x132ef14] nmethod::verify()+0x3f4
V [libjvm.so+0x1328afd] nmethod::new_nmethod(methodHandle const&, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int, char*, int, int, char const*, FailedSpeculation**)+0x3dd
V [libjvm.so+0x7d146b] ciEnv::register_method(ciMethod*, int, CodeOffsets*, int, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, bool, bool, RTMState)+0x56b
V [libjvm.so+0x13d37ad] PhaseOutput::install_code(ciMethod*, int, AbstractCompiler*, bool, bool, RTMState)+0x17d
V [libjvm.so+0x13d39a4] PhaseOutput::install()+0x164
V [libjvm.so+0x8fa111] Compile::Code_Gen()+0x461
V [libjvm.so+0x901a43] Compile::Compile(ciEnv*, ciMethod*, int, bool, bool, bool, DirectiveSet*)+0x1763
V [libjvm.so+0x74aa25] C2Compiler::compile_method(ciEnv*, ciMethod*, int, DirectiveSet*)+0x175
V [libjvm.so+0x91094e] CompileBroker::invoke_compiler_on_method(CompileTask*)+0x45e
V [libjvm.so+0x912488] CompileBroker::compiler_thread_loop()+0xab8
V [libjvm.so+0x16a7ac6] JavaThread::thread_main_inner()+0x226
V [libjvm.so+0x16ad960] Thread::call_run()+0x100
V [libjvm.so+0x13b2396] thread_native_entry(Thread*)+0x116

It seems, a method can be redefined/retransformed before the compiler gets new nmethod installed. So, the compiler may need to check if the method has not become old before nmethod installation.

This problem is very intermittent and rarely reproducible.
The Kitchensink Instrumentation module needs to be enabled.
Also, the options to use are:
  -Xlog:class+redefine*=trace -Xcomp -ea -esa -XX:CompileThreshold=100 -XX:+UnlockExperimentalVMOptions -XX:-TieredCompilation -XX:+CreateCoredumpOnCrash