FULL PRODUCT VERSION :
$ java -version
java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Linux,
Windows

A DESCRIPTION OF THE PROBLEM :
RFC 4752 section 4.3.1 (https://tools.ietf.org/html/rfc4752#section-3.1) states that:

The client then constructs data, with the first octet containing the bit-mask specifying the selected security layer, the second through fourth octets containing in network byte order the maximum size output_message the client is able to receive (which MUST be 0 if the client does not support any security layer).

With default settings, a SaslClient created for GSSAPI will violate the MUST in the parenthesized sentence above).

This is because javax.security.sasl.Sasl#QOP defaults to "auth", while javax.security.sasl.Sasl#MAX_BUFFER has an unspecified default, but in the implementation it ends up defaulting to 65536 (in the initializer for com.sun.security.sasl.util.AbstractSaslImpl#recvMaxBufSize).

So what ends up in happening in com.sun.security.sasl.gsskerb.GssKrb5Client#doFinalHandshake is that the first four bytes of the response is set to [1, 1, 0, 0], instead of the correct value of [1, 0, 0, 0].

I consider this a bug because it sends a message that violates the above-quoted specification. The fix is simple. Set the default for Sasl.MAX_BUFFER to "0" for the GSSAPI mechanism.

That means there is a simple workaround for users (set the Sasl.MAX_BUFFER property to "0" in application code), but it's not at all obvious that one would actually need to do such a thing.

While this may not seem serious, there are actually GSSAPI service implementation out in the wild that will error if they detect that the client is violating the specification in thist way. The one I know about and led me to find this issue is the MongoDB executable for Windows.



STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
This is easily reproduced by creating a SaslClient for GSSAPI and using it to run through a SASL conversation. If you enable sun.security.krb5.debug, you can clearly the incorrect message being sent.



EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The debug out shows:

    Krb5Context.wrap: data=[01 01 00 00 64 72 69 76 65 72 73 40 4c 44 41 50 54 45 53 54 2e 31 30 47 45 4e 2e 43 43 ]
    Krb5Context.wrap: token=[05 04 00 ff 00 0c 00 00 00 00 00 00 2f fb c6 03 01 01 00 00 64 72 69 76 65 72 73 40 4c 44 41 50 54 45 53 54 2e 31 30 47 45 4e 2e 43 43 75 c0 6e 14 6e 16 be e5 67 e2 e3 c3 ]
ACTUAL -
It should be:

Krb5Context.wrap: data=[01 00 00 00 64 72 69 76 65 72 73 40 4c 44 41 50 54 45 53 54 2e 31 30 47 45 4e 2e 43 43 ]
    Krb5Context.wrap: token=[05 04 00 ff 00 0c 00 00 00 00 00 00 2f fb c6 03 01 01 00 00 64 72 69 76 65 72 73 40 4c 44 41 50 54 45 53 54 2e 31 30 47 45 4e 2e 43 43 75 c0 6e 14 6e 16 be e5 67 e2 e3 c3 ]

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import org.ietf.jgss.GSSCredential;
import org.ietf.jgss.GSSException;
import org.ietf.jgss.GSSManager;
import org.ietf.jgss.GSSName;
import org.ietf.jgss.Oid;

import javax.security.sasl.Sasl;
import javax.security.sasl.SaslClient;
import javax.security.sasl.SaslException;
import java.util.HashMap;
import java.util.Map;

import static com.mongodb.AuthenticationMechanism.GSSAPI;

public class GssapiSaslBug {

    private static final String GSSAPI_OID = "1.2.840.113554.1.2.2";

    public static void authenticate(final String userName, final String serviceName, final String hostName)
            throws GSSException, SaslException {

        GSSManager manager = GSSManager.getInstance();
        GSSCredential gssCredential = manager.createCredential(
                manager.createName(userName, GSSName.NT_USER_NAME),
                GSSCredential.INDEFINITE_LIFETIME, new Oid(GSSAPI_OID),
                GSSCredential.INITIATE_ONLY);

        Map<String, Object> saslClientProperties = new HashMap<>();
        saslClientProperties.put(Sasl.CREDENTIALS, gssCredential);

        SaslClient saslClient = Sasl.createSaslClient(new String[]{GSSAPI.getMechanismName()}, userName,
                serviceName, hostName, saslClientProperties, null);


        byte[] message = (saslClient.hasInitialResponse() ? saslClient.evaluateChallenge(new byte[0]) : null);

        byte[] reply = sendAndReceive(message);

        while (!(done(saslClient))) {
            message = saslClient.evaluateChallenge(reply);

            reply = sendAndReceive(message);
        }

    }

    private static byte[] sendAndReceive(final byte[] message) {
        // send message and receive response using whatever service you're talking to
        return new byte[0];
    }

    private static boolean done(final SaslClient saslClient) {
        // figure out if you're done
        return false;
    }
}
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Create a map which sets the relevant property:

Map<String, Object> saslClientProperties = new HashMap<String, Object>();
saslClientProperties.put(Sasl.MAX_BUFFER, "0");

And pass it as the second-to-last argument to javax.security.sasl.Sasl#createSaslClient