JSSE debug output (controlled via javax.net.debug) can be lengthy. There are two issues here.

1. Several tests have JSSE debug output on by default, but don't really need it. To save disk space, it seems like output could be off when not needed.

There are a few primary offenders/generators of unneeded large output files.

    javax/net/ssl/{DTLSv10,ServerName,TLS,TLSv1,TLSv11,TLSv12}

Debugging output isn't required for almost all of these tests.

Some tests examine the output of the debug log files, so debug output must remain enabled.

2. If JSSE debug output is active in a JTREG test, tests often run into the max file size limitation of JT Harness, which by default is 100000 (set via javatest.maxOutputSize). When that size is hit, the middle parts of the JSSE handshake output are replaced with:
...
Output overflow:
JT Harness has limited the test output to the text
at the beginning and the end, so that you can see how the
test began, and how it completed.

If you need to see more of the output from the test,
set the system property javatest.maxOutputSize to a higher
value. The current value is 100000
...

which make the logs useless for debugging actual SSL/TLS connections.

Devs have to remember the magic incantation, which unfortunately is not as simple as adding -Djavatest.maxOutputSize=xxxxxxx to the test invocation, as that's the wrong VM. See the workaround buried in the JTREG FAQ. https://openjdk.org/jtreg/faq.html#how-to-set-javatest.maxOutputSize

The easiest way is to add TEST.properties to the directories involved:

    test/jdk/javax/net/ssl
    test/jdk/sun/security/ssl

JDK-8339261 did something similar for a single test, but consider adding this value for all JSSE tests.

Here's the amount of space currently needed for the JSSE tests (80MB) and how much is needed if only the second change is done (136MB) (adding maxOutputSize/do not disable unneeded output by default):

test/jdk/javax/net/ssl
    open: 55288 KB -> 108816 KB (+96%)
test/jdk/sun/security/ssl
    open: 844 KB -> 912 KB (+8%)

My plan:

    . add TEST.properties to the top level directories,
    . turn off the existing debug output if it's not directly needed for a test, as it's just noise for most runs.