With the change for supporting large buffers to SSLEngine, two httpserver tests started failing.

    test/com/sun/net/httpserver/Test7a.java and Test7b.java

Since we are getting close to shipping 5.0u8, I took a poke around to see what I could find.

It appears that there is a buffering problem in the mustang httpserver code that needs to be fixed.

As part of your test, your application was writing out encrypted application data, and the 0x401'st byte I saw:

03F0: ...deleted...
0400: xx FA C1 58 79 5A 37 74

which was then unwrapped on the peer side as *the start* of that application packet. This means the packet was:

   version major: 0xfa (250)
   version minor: 0xc1 (193)
   length major: 0x58
   length minor: 0x79 = 22649 (+5 for the header)

This caused:

pool-1-thread-2, Bad InputRecord size, count = 22654

2050: 3F 67 9A 05 06 71 AF 09 9F 39 FF 7C 23 BB 21 ED ?g...q...9..#.!.
2060pool-1-thread-2, READ: Unknown-250.193 contentType = 25, length = 22649

which was then reported as an error:

38D0: B1 93 pool-1-thread-2, fatal error: 20: bad record MAC
javax.net.ssl.SSLException: bad record MAC

So, for some reason, you've got some buffering problems, and you are passing SSLEngine the 0x401'st byte as the first byte, which is 1024 bytes in from the true start of the packet header, which BTW, was also the initial increment size for your test. I'm pretty certain the problem is in the httpserver code.

The changes in the SSLEngine are the following:

1) We no longer require 16384 bytes to be free in the unwrap dst buffer before returning BUFFER_OVERFLOW. We only require the packet header - 5 bytes.

2) The initial sizes reported by SSLSession.getApplicationBufferSize() is 5 less than SSLSession.getPacketBufferSize(). I notice your code doesn't use these calls at all.

3) In default mode, our application data packets are up to 2^14 kb. If we see a large packet, we will switch and allow up to 32Kb of data. Your length just happened to be in the zone between normal packets and large packets. The MAC checks later on caused things to fail as expected.

So I think this is a httpserver problem and not a SSLEngine problem. Let me know if you think otherwise.