-
Enhancement
-
Resolution: Unresolved
-
P3
-
None
-
None
-
None
There are may occasions - eg fastdebug builds testing - where only crashes is what the tester is interested in.
In the current implementation of jtreg/jtharness it is left on the downstream user to post process results, and to consider as failed tests only those who failed, and crashed.
jtreg/jtharness should contain and tool chain, where the jvm crashes will be considered, before reporting.
The basic use case should some --crashesOnly switch, which will switch jtreg/jtharness to mode, where only the test which failed, and crashed together will be reported as pass or skipped (configurable?). Another configurable part may be to consider passed test with crash as failure, as although there are many tests intentionally crashing jvm and passing, some other tests with crash may be hidden false positives.
It is not a goal of this bug to track implementation of this on junit/testng level, but an attempt to implement similar feature is not an bad idea.
In the current implementation of jtreg/jtharness it is left on the downstream user to post process results, and to consider as failed tests only those who failed, and crashed.
jtreg/jtharness should contain and tool chain, where the jvm crashes will be considered, before reporting.
The basic use case should some --crashesOnly switch, which will switch jtreg/jtharness to mode, where only the test which failed, and crashed together will be reported as pass or skipped (configurable?). Another configurable part may be to consider passed test with crash as failure, as although there are many tests intentionally crashing jvm and passing, some other tests with crash may be hidden false positives.
It is not a goal of this bug to track implementation of this on junit/testng level, but an attempt to implement similar feature is not an bad idea.
- links to
-
Review openjdk/jtharness/57
-
Review openjdk/jtreg/164
-
Review(master) openjdk/jtreg/235