Details
-
Sub-task
-
Resolution: Unresolved
-
P4
-
17, 18
Description
There are different ways how an IR framework test can fail: There was a VM crash (1), an exception while running a test (2), or an IR verification failed (3). Depending on whether scenarios were used (A) or not (B), the exception message reported can be different and might not contain enough information or too much for a quick analysis (for example on Mach5).
This RFE should check all 6 cases (1A, 1B, 2A, 2B, 3A, 3B) and filter/redirect too much details to the stderr and/or add more concise information about the nature of the failure and where to start looking.
We should also think about providing a concise summary of what went wrong at the very end to quickly get an idea of what went wrong. The exception message, stderr and stdout could be structured like this:
<stdout> (Compilation output, VM output etc.)
<stderr> (Verbose summary of failures with matched regexes, rules etc.)
<exception message> (Concise summary of failed scenario and tests)
This RFE should check all 6 cases (1A, 1B, 2A, 2B, 3A, 3B) and filter/redirect too much details to the stderr and/or add more concise information about the nature of the failure and where to start looking.
We should also think about providing a concise summary of what went wrong at the very end to quickly get an idea of what went wrong. The exception message, stderr and stdout could be structured like this:
<stdout> (Compilation output, VM output etc.)
<stderr> (Verbose summary of failures with matched regexes, rules etc.)
<exception message> (Concise summary of failed scenario and tests)