The timeout, which was mostly observed on the macosx-x64 platform, was first discovered in tier8 testing (see JDK-8273109).

When the test runtime/cds/appcds/loaderConstraints/DynamicLoaderConstraintsTest.java is run with the following vmoptions:
"-Xcomp -XX:+CreateCoredumpOnCrash -ea -esa -XX:CompileThreshold=100 -XX:+UnlockExperimentalVMOptions -server -XX:+TieredCompilation -XX:+DeoptimizeALot",
without the fix for JDK-8272873, the custom-cl case took around totalTime=744005. With the fix, the custom-cl timeout totalTime=1361245.

The slow down seems to be related to excessive inlining. Here's one example from the two attached log files:

Before:

  12578 2425 b 4 java.lang.invoke.InvokerBytecodeGenerator::emitPushArgument (150 bytes)
  12588 2424 3 java.lang.invoke.InvokerBytecodeGenerator::emitPushArgument (150 bytes) made not entrant
  12588 2425 size: 3264(1088) time: 9 inlined: 60 bytes

After:
  54396 5810 b 4 java.lang.invoke.InvokerBytecodeGenerator::emitPushArgument (150 bytes)
  54796 5809 3 java.lang.invoke.InvokerBytecodeGenerator::emitPushArgument (150 bytes) made not entrant
  54799 5810 size: 54872(23776) time: 402 inlined: 4651 bytes