https://openjdk.org/jtreg/faq.html has a section titled "What is a ProblemList.txt file?" which documents the expected format of ProblemList.txt files (a.k.a. exclude lists).
Currently, jtreg ignores nonconforming lines. This leads to confusion because developers can commit problemlist additions that they believe to exclude certain tests, but which don't because of typos or misunderstandings of the format. Some examples include:
* Misspelling a platform name
* Assuming that a "-debug" postfix on the platform component would exclude the test on debug builds
* Assuming that the test name is the path of the test file, when the file contains several test cases
* Assuming that folders can be problemlisted
* Adding the entry to the wrong problemlist
All of these have happened in real life. They all result in no test being problemlisted.
I propose to add a verification step to jtreg which goes over the contents of the files passed through "-exclude" and applies a set of checks. If a check fails, jtreg should print the offending line, explain the error, and refuse to start running. This is to prevent drift of the problemlist files when e.g. tests change or are renamed or moved. The verification step should be switched on by a new boolean flag -verifyexclude. This is to make adoption easier to control gradually.
I propose that the checks include, for starters:
1. Verify that each test exists within the test suite (this includes verifying #id-named tests)
2. Verify that each line is either a comment or else follows the format defined in "What is a ProblemList.txt file?"
3. Each test is listed at most once in the same Problemlist file
More checks could be added in the future. One additional check I want to add is
4. Verify that the platform actually exists
But this represents a challenge because jtreg cannot know the full set of OS/ARCH/REV strings that could potentially be valid. One solution could be to introduce another flag where the caller passes all the recognized OS/ARCH/REV options that it recognizes. However I suggest to leave this for another ticket.
Currently, jtreg ignores nonconforming lines. This leads to confusion because developers can commit problemlist additions that they believe to exclude certain tests, but which don't because of typos or misunderstandings of the format. Some examples include:
* Misspelling a platform name
* Assuming that a "-debug" postfix on the platform component would exclude the test on debug builds
* Assuming that the test name is the path of the test file, when the file contains several test cases
* Assuming that folders can be problemlisted
* Adding the entry to the wrong problemlist
All of these have happened in real life. They all result in no test being problemlisted.
I propose to add a verification step to jtreg which goes over the contents of the files passed through "-exclude" and applies a set of checks. If a check fails, jtreg should print the offending line, explain the error, and refuse to start running. This is to prevent drift of the problemlist files when e.g. tests change or are renamed or moved. The verification step should be switched on by a new boolean flag -verifyexclude. This is to make adoption easier to control gradually.
I propose that the checks include, for starters:
1. Verify that each test exists within the test suite (this includes verifying #id-named tests)
2. Verify that each line is either a comment or else follows the format defined in "What is a ProblemList.txt file?"
3. Each test is listed at most once in the same Problemlist file
More checks could be added in the future. One additional check I want to add is
4. Verify that the platform actually exists
But this represents a challenge because jtreg cannot know the full set of OS/ARCH/REV strings that could potentially be valid. One solution could be to introduce another flag where the caller passes all the recognized OS/ARCH/REV options that it recognizes. However I suggest to leave this for another ticket.
- relates to
-
CODETOOLS-7903648 Cannot exclude a test if it appears in 2 ProblemList.txt files
-
- New
-
