ModulesInCustomFileSystem.java creates temporary directories containing .jar file(s) for the test, but does not clean up later:

cat -n jdk/test/java/lang/module/customfs/ModulesInCustomFileSystem.java
   [...]
     62 Path jar = Files.createTempDirectory("mlib").resolve("modules.jar");
   [...]
    73 Path contents = Files.createTempDirectory("contents");
   [...]
    78 Path jar = Files.createTempDirectory("mlib").resolve("modules.jar");

A better approach would be to create temporary test directories and files under the jtreg property user.dir instead, if user.dir is defined.

During a jtreg run, user.dir has the value $PWD/JTwork/scratch