SSLSocketOutputRecord was found to not calculate the buffer sizes correctly when it was doing a Math.min check:

~ line 170:
                int remains = (limit - offset) + (count - position);
                int fragLen = Math.min(fragLimit, remains);
                write(source, offset, fragLen);

fragLen should be:
                int fragLen = Math.min(fragLimit - count + position, limit - offset);

After this change, putting a lot of data in the HandshakeOutStream buffer resulted in no errors. Since handshake operations are usually small, this seldom occurs. This was not observed with application data, which would tend to be larger data sizes, so there must be other checks that mitigate this in that case.

Server failed: javax.net.ssl.SSLException: (internal_error) Range [0, 0 + 16367) out of bounds for length 1268
javax.net.ssl.SSLException: (internal_error) Range [0, 0 + 16367) out of bounds for length 1268
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:132)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:380)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316)
        at java.base/sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1713)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:470)
        at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:924)
        at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1015)
        at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:350)
        at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:393)
        at java.base/sun.nio.cs.StreamDecoder.lockedRead(StreamDecoder.java:217)
        at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:171)
        at java.base/java.io.InputStreamReader.read(InputStreamReader.java:190)
        at java.base/java.io.BufferedReader.fill(BufferedReader.java:160)
        at java.base/java.io.BufferedReader.implReadLine(BufferedReader.java:370)
        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:347)
        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:436)
        at SSLSocketBruteForceClose.runServerApplication(SSLSocketBruteForceClose.java:64)
        at SSLSocketTemplate.doServerSide(SSLSocketTemplate.java:223)
        at SSLSocketTemplate.startServer(SSLSocketTemplate.java:427)
        at SSLSocketTemplate.bootup(SSLSocketTemplate.java:341)
        at SSLSocketTemplate.run(SSLSocketTemplate.java:73)
        at SSLSocketBruteForceClose.main(SSLSocketBruteForceClose.java:48)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
        at java.base/java.lang.Thread.run(Thread.java:1575)
Caused by: java.lang.IndexOutOfBoundsException: Range [0, 0 + 16367) out of bounds for length 1268
        at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:100)
        at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckFromIndexSize(Preconditions.java:118)
        at java.base/jdk.internal.util.Preconditions.checkFromIndexSize(Preconditions.java:397)
        at java.base/java.util.Objects.checkFromIndexSize(Objects.java:417)
        at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:131)
        at java.base/sun.security.ssl.SSLSocketOutputRecord.encodeHandshake(SSLSocketOutputRecord.java:175)
        at java.base/sun.security.ssl.HandshakeOutStream.complete(HandshakeOutStream.java:62)
        at java.base/sun.security.ssl.SSLHandshake$HandshakeMessage.write(SSLHandshake.java:570)