Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8145079

Introduce ProblemList.txt to quarantine tests in hotspot.

XMLWordPrintable

    • Icon: Enhancement Enhancement
    • Resolution: Duplicate
    • Icon: P3 P3
    • None
    • None
    • hotspot
    • None


      Add hotspot/test/ProblemList.txt, similar to how there is a jdk/test/ProblemList.txt and use it to quarantine hotspot jtreg tests.


      Background:

      It is useful to be able to both quarantine and exclude tests.
       - A quarantined test is one that for some reason is expected to fail, either deterministically or intermittently. In order to reduce the noise, it is put in a quarantine zone (added to the problem list). Quarantined tests is still beneficial to run, for example in order to gather statistics or possibly find other issues as well. Running them at a special occasion will however set the expectations and the quarantine mechanism enables this.
       - An excluded tests is just as a quarantined test expected to fail, but it should not be run because it is too costly, for example if it takes long to timeout, or if it produces crashes.

      Putting a test on ProblemList.txt is the mechanism to quarantine a test.
      Using @ignore in a test is the mechanism to exclude a test.

      The benefit of using the ProblemList over @ignore, is that the problem list gives additional possibilities to define platform for exclusion. The expectation is that there will be more quarantined tests than excluded tests.

      And to state the obvious, the goal in the end is of course to have neither quarantined nor excluded tests.

            Unassigned Unassigned
            ssarne Stefan Sarne (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: