ADDITIONAL SYSTEM INFORMATION :
Generic, Bug relates to all recent Java releases

A DESCRIPTION OF THE PROBLEM :
There appears to be a bug in the isCookieValid method of HelloCookieManager that can lead to improper cookie validation results after a new cookieSecret is generated. The cookie version (cookieVersion) is initialized with a random number and increments with each cookie, regenerating the secret every 0xFFFFFF cookies.

The problematic code is:

try {
    if (((cookieVersion >> 24) & 0xFF) == cookie[0]) {
        secret = cookieSecret;
    } else {
        secret = legacySecret;  // includes out-of-window cookies
    }
Issue:
When cookieVersion is negative, the expression ((cookieVersion >> 24) & 0xFF) results in an integer that does not match the signed byte value in cookie[0], causing the else clause to be executed incorrectly. This leads to the use of legacySecret even for valid cookies.

Suggested Fix:
To ensure proper comparison with the signed byte value in cookie[0], the expression should cast the result to byte:

if ((byte)((cookieVersion >> 24) & 0xFF) == cookie[0]) {
    secret = cookieSecret;
} else {
        secret = legacySecret;  // includes out-of-window cookies
    }


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Start number of DTLS sessions with sun.security.ssl.HelloCookieManager.D10HelloCookieManager#cookieVersion property. initialized with first byte in range 0x80-0xfe). Once new secret is generated (after maximum of 0xffffff cookies) - the engine will stop accepting handshakes due to invalid cookie bug

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Generating new cookieSecret should not break DTLS handshaking.
ACTUAL -
Generating new cookieSecret may break DTLS handshaking if cookieSecret has negative value.