The suggestion from hotspot developers is to add a control variable TEST_DBG=[y/true/whatever], which would be equivalent to setting e.g. JTREG="VM_OPTIONS=-XX:+ShowMessageBoxOnError", for all test frameworks that support VM_OPTIONS.

Future modifications might be needed if this is not enough to provide the functionality needed.