P3
I noticed some irregularities/difficulties with choosing tests for jtreg to run.
-match:file
was recently added, but needs documentation.
Exclude list formats are currently documented only in the jtreg faq. Maybe that information belongs in the command line docs?
Apparently everything in an exclude list is optional, except the test name, but this is undocumented. I've been cargo-cult appending "generic-all" to test names for years, apparently with no effect?
test names in exclude lists (and match files?) are relative to the TEST.ROOT directory, which is not completely obvious. Nonexistent files in exclude lists are ignored, unlike on the command line, where they are an error. (neither choice is perfect).
Groups can be defined in a TEST.groups file. Unlike exclude files, these files can only be found in the test root directory. Negative exclusions, including directory exclusions, can be used in test group definitions but they cannot be used on the command line (no syntax) or in exclude files (directories are ignored). For comparison, bazel's test selection does allow a leading '-' to mean exclusion.
Users are very likely to want to define personal test groups to match projects they are working on. Although they can in effect do this by maintaining a jtreg wrapper as I have done for years, TEST.groups seems like a feature in need of generalization.
-match:file
was recently added, but needs documentation.
Exclude list formats are currently documented only in the jtreg faq. Maybe that information belongs in the command line docs?
Apparently everything in an exclude list is optional, except the test name, but this is undocumented. I've been cargo-cult appending "generic-all" to test names for years, apparently with no effect?
test names in exclude lists (and match files?) are relative to the TEST.ROOT directory, which is not completely obvious. Nonexistent files in exclude lists are ignored, unlike on the command line, where they are an error. (neither choice is perfect).
Groups can be defined in a TEST.groups file. Unlike exclude files, these files can only be found in the test root directory. Negative exclusions, including directory exclusions, can be used in test group definitions but they cannot be used on the command line (no syntax) or in exclude files (directories are ignored). For comparison, bazel's test selection does allow a leading '-' to mean exclusion.
Users are very likely to want to define personal test groups to match projects they are working on. Although they can in effect do this by maintaining a jtreg wrapper as I have done for years, TEST.groups seems like a feature in need of generalization.