LIBRARY.properties introduced the 'enablePreview' setting to give a test library control over the default compilation option.
However, the final compilation option was a combination of options from \@build, \@compile, and \@run command lines.
Any test that referred to a library with enablePreview = true was also compiled and run with enablePreview = true, supplementing or overriding enablePreview options in the test itself.

Since test libraries are compiled on demand and shared thereafter, the first test using the library determined the enable-preview option. Subsequent tests with conflicting compilation options would see unexpected behavior.

A specific example occurred frequently in Valhalla when the ProcessTools test library class would be compiled with enable-preview but individual tests run without preview would fail to compile due to the restricted class file version.

How it works today (as implemented by CODETOOLS-7903775)

Explicit --release and -source arguments in @build and @compile actions of tests take precedence over declarative tags. That means, tests are in control. And for subsequent tests, they may produce unexpected outcomes when those explicit action tags compile library source files with custom options, such as --enable-preview.
A test with a declarative @enablePreview false tag will not be supplied an --enable-preview arguments at test compile time nor at test run-time. For more details consult: https://openjdk.org/jtreg/faq.html#my-test-uses-preview-features-how-do-i-specify-the-necessary-options
A test using one more @library tags (see @library 's FAQ entry) may refer to shared test source libraries that grew over time. A prominent candidate is the test/lib/ directory in the open JDK repository. If such a test source library directory:
No LIBRARY.properties or a LIBRARY.properties reading:

enablePreview = false

introduces no difference in behaviour compared to former jtreg versions.
If a LIBRARY.properties file exists and contains:

enablePreview = true

then --enable-preview and -source N (version of the JDK under test) are added to list of arguments passed to javac when compiling classes defined within the current library being compiled.

Potential Problems

Problems arise for test libraries that are compiled into shared classpath directories. Files compiled with @build and @compile from library source paths are compiled into corresponding destination directories. Additional source files referenced by the named files may be compiled implicitly by javac and placed in the same destination.
Test source files are compiled into test specific destination directory along with any implicitly compiled source files found in the sourcepath corresponding to the @library paths.

Implicit --enable-preview in tests

As implemented, any @compile, @build, or @run step may have a implicit --enable-preview , --source <n> , and --target <n> arguments added. The --enable-preview is added if any library has the enablePreview = true property set.
This implicit addition of --enable-preview hides an important change to @build, @compile, or @run execution. While the change may seem convenient in some cases, it is hidden without any visible declaration in the @test configuration.
Each test should explicitly opt-in to the use of preview features in the APIs and the test library.
Any inconsistency between the test and library should be resolved during test development.

Non-deterministic test library compiles

The original problem is unpredictable build and compiles of library classes depending on the --enable-preview flag of the first test that used the library. The addition in LIBRARY.properties of the enablePreview = true only addressed half of the problem, for libraries that needed --enable-preview. For test libraries that must not have --enable-preview it still leaves it up to the first compile or build.
The LIBRARY.properties enablePreview either true or false should not be overridden by the options in the test itself.

Valhalla work around

Valhalla has many tests that need to be run both with and without --enable-preview , configured by the @run command lines. And needs the corresponding test libraries.
A (temporary) workaround for Valhalla is to ensure that the test/lib test library is not compiled with --enable-preview. This achieved by its enablePreview = false and that test library compiles are done with @build and @enablePreview false or being absent.
It is however, fragile, since a change to an library could add enablePreview and implicitly change the test execution without resulting in an error.