Jtreg has a feature where a default behavior is assumed if no @run tag is present in the defining file of a test. This behavior is convenient and saves typing, but it can lead to problems where a test appears to be passing when in fact it isn't being run at all.

For example, suppose we have a test that has these tags in a file named MyTest.java:

    @test
    @bug 1234567
    @summary My simple test.
    @author Fred

Since there are no @run tags, jtreg will implicitly act as if "@run main MyTest" were present. OK so far.

Now suppose we modify the test to use a helper class of some sort. We'd need to add an @build tag to ensure this class is built:

    @test
    @bug 1234567
    @summary My simple test.
    @author Fred
    @build Helper

Since @build is a shorthand for "@run build", there is implicitly an @run tag in this file, and the default behavior does not apply. "Running" this test will then consist of building the MyTest and Helper classes but not actually executing the test.

This kind of problem has occurred in practice when tests are modified or maintained. It's fairly easy for a test to end up in this state, and it's fairly difficult to detect, as the test "passes" when run, but one has to inspect the log carefully in order to determine that the test hasn't actually been run. The JDK regression suite has encountered this problem.

It would be nice to be able to require that every test have an @run tag. Since @build counts as an @run tag, the above example will still "pass" even though nothing was executed. The rule would have to be finer-grained, perhaps requiring that a @run tag with a test execution action (e.g., "main", "shell", "applet", "testng") be present in every test.

Of course, this is an incompatible change. It may be necessary to introduce this requirement via a command-line option or a property in TEST.ROOT.

There may be tests that need to be executed solely for their @compile or @build side effects. (The compiler test suite may be an example of this.) It may be sufficient not to have the option enabled such suites. Or, if a single suite has a mixture of execution and non-execution tests, it may be necessary to introduce a new action like "@run noop" that satisfies the test execution requirement, but that explicitly indicates that the test performs no action. (It's easy to write a no-op Java or shell test, but it seems like this ought to be built into jtreg.)