FULL PRODUCT VERSION :
>java -version
java version "1.6.0_10-rc2"
Java(TM) SE Runtime Environment (build 1.6.0_10-rc2-b32)
Java HotSpot(TM) 64-Bit Server VM (build 11.0-b15, mixed mode)


FULL OS VERSION :
Linux RHAS5 2.6.18-8.el5 x86_64 GNU/Linux


EXTRA RELEVANT SYSTEM CONFIGURATION :
gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)
Target: x86_64-redhat-linux




A DESCRIPTION OF THE PROBLEM :
The AttachThread/DetachThread pair seems to leak on 64-bit Linux if they are not run in serial from a single thread. In other words, it appears that calling Attach from one thread and Attach from another thread, and then Detach, Detach will leak memory. Calling Attach from one thread and then Detach from that same thread seems to function fine.

My test cases uses a global mutex to ensure the latter scenario when I see no leaks. Commenting out the mutex, or releasing the mutex in between the Attach/Detach calls will show a leak. The leak is viewed through "top -p <PID>" where PID is the test case executable.

The test case was ported to 32-bit Windows and did not appear to show the same behavior. 32-bit Linux was not tested.

THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Did not try

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Build the test case, linking in the JVM library and including the JVM includes. Run the test case and procure its PID. Use "top -p <PID>" and watch the memory usage of the process. It should remain constant. Go to the test case and remove the noted comments (~line 68 and 72) for the mutex release and re-lock. This should now unserialize the Attach/Detach so multiple threads can attach before the first thread detaches. Rebuild and re-run the test case. Viewing the process with "top -p <PID>" should now show an obvious memory leak.

EXPECTED VERSUS ACTUAL BEHAVIOR :
No memory growth in the resident (RES) memory size of the test executable with the commented blocks. Obvious memory growth in the resident (RES) memory size of the test executable with the noted comments removed.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
#include <jni.h>

#include <pthread.h>
#include <list>
#include <string>
#include <sstream>

JavaVM *theVM_ = 0;
JNIEnv *theGlobalEnv_ = 0;

void
initVM()
{
  jint res;

  JavaVMInitArgs vm_args;

  fprintf(stderr, "Initializing Java Virtual Machine...\n");

  // number of jvm arguments that are always included
  int javaVMArgsCount = 3;
  JavaVMOption *options = new JavaVMOption[javaVMArgsCount];

  char *cp = std::getenv("CLASSPATH");

  std::string initclasspath = "";

  options[0].optionString = "-Djava.compiler=NONE"; /* disable JIT */
  std::string path = "-Djava.class.path=";

  if(cp != 0) {
    path += cp;
  }
  else {
    path += ".";
  }

  options[1].optionString = (char *)(path.c_str()); /* user classes */

  // Use the server version of the JVM
  options[2].optionString = "-server";

  vm_args.version = JNI_VERSION_1_4;
  vm_args.options = options;
  vm_args.nOptions = javaVMArgsCount;
  vm_args.ignoreUnrecognized = true;

  // create the global JNIEnv.
  res = JNI_CreateJavaVM(&theVM_, (void **)&theGlobalEnv_, &vm_args);

  delete[] options;

}


int fib(int j) {
  return (j < 2) ? j : fib(j-1) + fib(j-2);
}

pthread_mutex_t globalMutex_;

void* runThread(void* arg) {
  JNIEnv* tmpEnv;
  while (1) {
    fib(10);
    pthread_mutex_lock(&globalMutex_);
    jint res = theVM_->AttachCurrentThread((void **)&tmpEnv, 0);
    // Uncomment this unlock and the lock line below to see leak
    // pthread_mutex_unlock(&globalMutex_);
    
    fib(10);
    // pthread_mutex_lock(&globalMutex_);
    jint res2 = theVM_->DetachCurrentThread();
    pthread_mutex_unlock(&globalMutex_);
  }
}

int main()
{
  initVM();

  printf("Starting test...\n");

  pthread_mutex_init(&globalMutex_, 0);

  std::list<pthread_t> l;
  for (int i = 0; i < 20; i++) {
    pthread_t thrId;
    pthread_create(&thrId, 0, runThread, 0);
    l.push_back(thrId);
  }


  std::list<pthread_t>::iterator it = l.begin();
  for (; it != l.end(); ++it) {
    pthread_join(*it, 0);
  }

  return 0;
}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
None, other than serializing every attach/detach call (and everything in between) in each thread.