This has been seen in tier 4 testing - details in comments.

Running the tests locally on Solaris x86 there are two interesting observations. Here's a normal test run:

TEST: com/sun/jdi/RedefineNestmateAttr/TestNestmateAttr.java
  build: 0.001 seconds
  compile: 2.409 seconds
  compile: 2.317 seconds
  build: 0.0 seconds
  main: 7.226 seconds
  compile: 2.257 seconds
  build: 0.0 seconds
  main: 102.009 seconds
  compile: 2.232 seconds
  build: 0.0 seconds
  main: 7.322 seconds
  compile: 2.473 seconds
  build: 0.0 seconds
  main: 7.603 seconds
TEST RESULT: Passed. Execution successful

This test have 4 @run commands, hence the 4 "main" entries. But note that the 2nd one takes 102 seconds compared to the 7-8 seconds of all the others. This huge difference is currently inexplicable.

Here's a Xcomp -XX:CompileThreshold=100 -XX:-TieredCompilation run:

TEST: com/sun/jdi/RedefineNestmateAttr/TestNestmateAttr.java
  build: 0.001 seconds
  compile: 2.437 seconds
  compile: 2.336 seconds
  build: 0.001 seconds
  main: 133.23 seconds
  compile: 2.28 seconds
  build: 0.0 seconds
  main: 288.456 seconds
  compile: 2.285 seconds
  build: 0.0 seconds
  main: 133.609 seconds
  compile: 2.242 seconds
  build: 0.0 seconds
  main: 135.068 seconds
TEST RESULT: Passed. Execution successful

Interesting observation here is that in general it takes 18x long to run this way: 135s versus 8. But the really slow test only takes a bit over twice as long.

While I need to investigate the execution anomaly, it seems evident that we need to ensure the timeout allows for a 18x slow down in general.

section:main
----------messages:(7/289)----------
command: main TestNestmateAttr HostA
reason: User specified action: run main/othervm TestNestmateAttr HostA
Mode: othervm [/othervm specified]
Additional options from @modules: --add-modules java.compiler
Timeout information:
--- Timeout information end.
elapsed time (seconds): 1586.496
----------configuration:(3/41)----------
Boot Layer
  add modules: java.compiler

----------System.out:(1155/69563)----------
vmOpts: '-XX:MaxRAMPercentage=3 -ea -esa -Xmx512m'
javaOpts: '-Xcomp -XX:+CreateCoredumpOnCrash -ea -esa -XX:CompileThreshold=100 -XX:+UnlockExperimentalVMOptions -server -XX:-TieredCompilation'
JVM version:11-internal
JDI version: 11.0
JVM description: Java Debug Interface (Reference Implementation) version 11.0
Java Debug Wire Protocol (Reference Implementation) version 11.0
JVM Debug Interface version 11.0
JVM version 11-internal (Java HotSpot(TM) 64-Bit Server VM, compiled mode)
Target: Testing original Host class from HostA
TestNestmateAttr: Testing original Host class from HostA
Trying bad retransform from Host/redef
Timeout refired 1200 times
...