FULL PRODUCT VERSION :
Windows:
1.6.0 and 1.60_01

java version "1.5.0_09"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
Java HotSpot(TM) Client VM (build 1.5.0_09-b03, mixed mode)

Linux:

java version "1.5.0_08"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_08-b03)
Java HotSpot(TM) Client VM (build 1.5.0_08-b03, mixed mode)



ADDITIONAL OS VERSION INFORMATION :
- Microsoft Windows XP [Version 5.1.2600]
- Linux XXXXXXXXXXX 2.6.9-42.ELsmp #1 SMP Sat Aug 12 09:39:11 CDT 2006
i686 i686 i386 GNU/Linux

EXTRA RELEVANT SYSTEM CONFIGURATION :
jaasLogin.conf:

TestKrb5JAAS {
   com.sun.security.auth.module.Krb5LoginModule required debug=true storeKey=true;
};

A DESCRIPTION OF THE PROBLEM :
I am experiencing a memory leak with com.sun.crypto.provider.SunJCE instances growing in count when using it in conjunction with LoginContext.
Running the attached program will eventually run out of memory.

YourKit profiler shows that com.security.crypto.provider.SunJCE is being allocated and hung onto by field "e" of javax.crypto.SunJCE_b which is an IdentityHashMap.

For every call to LoginContext.login() in the attached code, two instances of com.sun.crypto.provider.SunJCE are created and held onto by SunJCE_b.e

Debugging shows that SunJCE instance is allocated at the following stack:

Thread [main] (Suspended (breakpoint at line 115 in Provider))
            SunJCE(Provider).<init>(String, double, String) line: 115
            SunJCE.<init>() line: not available
            SunJCE_am.<init>(PBEKeySpec, String) line: not available
            PBKDF2HmacSHA1Factory.engineGenerateSecret(KeySpec) line: not available
           SecretKeyFactory.generateSecret(KeySpec) line: not available
            AesDkCrypto.PBKDF2(char[], byte[], int, int) line: 462
            AesDkCrypto.stringToKey(char[], byte[], byte[]) line: 111
            AesDkCrypto.stringToKey(char[], String, byte[]) line: 90
            Aes128.stringToKey(char[], String, byte[]) line: 29
            EncryptionKey.stringToKey(char[], String, byte[], int) line: 253
            EncryptionKey.acquireSecretKeys(char[], String, boolean, int, byte[]) line: 190
            EncryptionKey.acquireSecretKeys(char[], String) line: 158
            Krb5LoginModule.attemptAuthentication(boolean) line: 626
            Krb5LoginModule.login() line: 512
            NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
            NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
            DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
            Method.invoke(Object, Object...) line: 585
            LoginContext.invoke(String) line: 769
            LoginContext.access$000(LoginContext, String) line: 186
            LoginContext$4.run() line: 683
            AccessController.doPrivileged(PrivilegedExceptionAction<T>) line: not available [native method]
            LoginContext.invokePriv(String) line: 680
            LoginContext.login() line: 579
            TestKrbLeak.login(String, String, String) line: 52
            TestKrbLeak.main(String[]) line: 28



STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run the attached code with a very low -Xmx setting on either Windows XP or Linux. Make sure you change the lines for jaasConfName and an applicable kerberos DC setup.



EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
It doesn't leak SunJCE instances or releases them during LoginContext.logout
ACTUAL -
Leaks com.sun.crypto.provider.SunJCE instances

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------

import java.io.IOException;
import java.util.Set;

import javax.security.auth.Subject;
import javax.security.auth.callback.Callback;
import javax.security.auth.callback.CallbackHandler;
import javax.security.auth.callback.NameCallback;
import javax.security.auth.callback.PasswordCallback;
import javax.security.auth.callback.UnsupportedCallbackException;
import javax.security.auth.kerberos.KerberosTicket;
import javax.security.auth.login.LoginContext;
import javax.security.auth.login.LoginException;

public class TestKrbLeak {

    public static void main(String[] args) throws IOException {

        System.setProperty("java.security.krb5.realm", "MYREALM.AD");
        System.setProperty("java.security.krb5.kdc", "mytestdc.company.ad");
        System.setProperty("java.security.auth.login.config", "C:/temp/jaasLogin.conf");

        final String jaasConfName = "TestKrb5JAAS";
        final String username = "username1";
        final String password = "password1";

        while (true) {
            login(jaasConfName, username, password);
            try {
                Thread.sleep(5000);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }

    }

    private static void login(final String jaasConfName, final String username, final String password) {
        LoginContext loginContext = null;
        try {
            loginContext = new LoginContext(jaasConfName, new CallbackHandler() {
                public void handle(Callback[] callbacks) throws IOException, UnsupportedCallbackException {
                    for (Callback cb : callbacks) {
                        if (cb instanceof NameCallback) {
                            ((NameCallback) cb).setName(username);
                        } else if (cb instanceof PasswordCallback) {
                            ((PasswordCallback) cb).setPassword(password.toCharArray());
                        }
                    }
                }
            });
            loginContext.login();
            
            Subject subject = loginContext.getSubject();
            
            Set<KerberosTicket> tickets = subject.getPrivateCredentials(KerberosTicket.class);
            
            assert (tickets != null && !tickets.isEmpty());
            
        } catch (LoginException e) {
            e.printStackTrace();
        } finally {
            if (loginContext != null) {
                try {
                    loginContext.logout();
                } catch (LoginException e) {
                    e.printStackTrace();
                }
            }
        }
    }

}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
We placed our own backed ticket cache in front of LoginContext.login call to only call login occassionally. Note that this still leaks eventually..