Especially in Valhalla, we have large IR test files with a lot of individual tests. We frequently have tests that are not working anymore after a merge. Instead of problemlisting the entire test file, we choose to uncomment the non-working tests.

This RFE proposes to introduce a @Skip annotation that simply skips execution of a single @Test instead of uncommenting it. We can even go a step further and add another annotation @SkipIR to execute the @Test but ignore the @IR rules. One could think about having an additional attribute in @SkipIR to choose which @IR rules to ignore in case there are multiple rules and not all are problematic.

I think that this can also be useful in mainline, for example, when some IR tests are failing due to some missing features that are expected to go in later. Instead of commenting these things out, we can use @Skip annotations.

An additional benefit of having @Skip annotations is that we can introduce a "-DIgnoreSkip=true" mode which ignores all the skipped tests. This is an easy and low overhead way to check if the skipped tests are still a problem or not. This could also be done as part of ATR which runs problemlisted tests.