<summary>
Inspection of the code in hotspot/src/os/solaris/vm/os_solaris.cpp
shows memory is free'd, but there are accesses through the memory
pointer in the next statement, leading to undefined behavior.
This is in isT2_libthread():

    FREE_C_HEAP_ARRAY(char, lwpArray);
    lwpSize = lwpArray->pr_nent * lwpArray->pr_entsize;

</summary>

FULL PRODUCT VERSION :
java version "1.6.0-beta2"
Java(TM) SE Runtime Environment (build 1.6.0-beta2-b86)
Java HotSpot(TM) Server VM (build 1.6.0-beta2-b86, mixed mode)

We're using ../jdk1.6.0/jre/lib/i386/server/libjvm.so as our shared lib we load
in the application.



FULL OS VERSION :
SunOS hotspot 5.10 Generic_118833-36 sun4v sparc SUNW,Sun-Fire-T1000

EXTRA RELEVANT SYSTEM CONFIGURATION :
CONTEXT :
---------
We use the JNI invocation API to load the VM into a native application. Running Solaris 10 with a 64 bit compiled native application using the VM (Hotspot / JRE 6u1 / JNI 1.6) we encounter a problem when calling the JNI function JNI_CreateJavaVM(). Important in this
context is that our native application uses the async. I/O API.



A DESCRIPTION OF THE PROBLEM :
PROBLEM :
---------
In the above context we see a single malloc() requesting +/- 94 Mb of memory. As our application runs under strict
memory requirements we did some further investigation and came to the below analysis :

In the debugger we see :

        [...]

=> [4] malloc(size = 954437177U)
    [5] os::malloc(0x38e38e39, 0xd4b08d60, 0x388, 0x5a0e34, 0xfd9e6000, 0x4000), at 0xfd4451ec
    [6] isT2_libthread(0x8f000, 0x14258, 0xfd9e6000, 0xfd956d4a, 0x1df3c8, 0x38e38e39), at 0xfd806d70
    [7] os::Solaris::libthread_init(0x0, 0xfee3f2e8, 0xfd9f2f88, 0xfd9f2f8c, 0x80000, 0xfd9e6000), at 0xfd44af9c
    [8] os::init_2(0x2000, 0x80000, 0x16c00, 0x18000, 0xfd9e6000, 0xfd9fa160), at 0xfd44ada4
    [9] Threads::create_vm(0xfd9f3066, 0xfdffbc9b, 0x0, 0x15cb64, 0x59f524, 0xfd9e6000), at 0xfd889598
    [10] JNI_CreateJavaVM(0xd4ba73c0, 0xd4ba73c8, 0xd569a7b4, 0xfd9fa284, 0xfd446ad4, 0xfd9e6000), at 0xfd44688c

        [...]

When browsing the hotspot source code, we did check the file pointed out in the
VM log :

../solaris/vm/os_solaris.cpp#lwpArray
   4209 bool isT2_libthread() {
   4210 int i, rslt;
   4211 static prheader_t * lwpArray = NULL;
   4212 static int lwpSize = 0;
   4213 static int lwpFile = -1;
   4214 lwpstatus_t * that;
   4215 int aslwpcount;
   4216 char lwpName [128];
   4217 bool isT2 = false;
   4218
   4219 #define ADR(x) ((uintptr_t)(x))
   4220 #define LWPINDEX(ary,ix) ((lwpstatus_t *)(((ary)->pr_entsize * (ix)) + (ADR((ary) + 1))))
   4221
   4222 aslwpcount = 0;
   4223 lwpSize = 16*1024;
   4224 lwpArray = ( prheader_t *)NEW_C_HEAP_ARRAY (char, lwpSize);
   4225 lwpFile = open ("/proc/self/lstatus", O_RDONLY, 0);
   4226 if (lwpArray == NULL) {
   4227 if ( ThreadPriorityVerbose ) warning ("Couldn't allocate T2 Check array\n");
   4228 return(isT2);
   4229 }
   4230 if (lwpFile < 0) {
   4231 if ( ThreadPriorityVerbose ) warning ("Couldn't open /proc/self/lstatus\n");
   4232 return(isT2);
   4233 }
   4234 for (;;) {
   4235 lseek (lwpFile, 0, SEEK_SET);
   4236 rslt = read (lwpFile, lwpArray, lwpSize);
   4237 if ((lwpArray->pr_nent * lwpArray->pr_entsize) <= lwpSize) {
   4238 break;
   4239 }
   4240 FREE_C_HEAP_ARRAY(char, lwpArray);
   4241 lwpSize = lwpArray->pr_nent * lwpArray->pr_entsize;
   4242 lwpArray = ( prheader_t *)NEW_C_HEAP_ARRAY (char, lwpSize);
   4243 if (lwpArray == NULL) {
   4244 if ( ThreadPriorityVerbose ) warning ("Couldn't allocate T2 Check array\n");
   4245 return(isT2);
   4246 }
   4247 }
   4248
   4249 // We got a good snapshot - now iterate over the list.
   4250 for (i = 0; i < lwpArray->pr_nent; i++ ) {
   4251 that = LWPINDEX(lwpArray,i);
   4252 if (that->pr_flags & PR_ASLWP) {
   4253 aslwpcount++;
   4254 }
   4255 }
   4256 if ( aslwpcount == 0 ) isT2 = true;
   4257
   4258 FREE_C_HEAP_ARRAY(char, lwpArray);
   4259 close (lwpFile);
   4260 if ( ThreadPriorityVerbose ) {
   4261 if ( isT2 ) tty->print_cr("We are running with a T2 libthread\n");
   4262 else tty->print_cr("We are not running with a T2 libthread\n");
   4263 }
   4264 return (isT2);
   4265 }

The problem is L4240-L4242. First the call FREE_C_HEAP_ARRAY(char, lwpArray); at L4240, this results in a free() at the end on lwpArray.

 What we see is that this freed memory already gets assigned to another thread which does write something else in it. Depending on the data we get insane memory requests. Of course this does not always works out the wrong way, sometimes the data at the address pointed by lwpArray is still valid. The free should be done after using the value in the
allocated memory.

REQUEST :
---------
As the usage of already freed memory leads to inconsistent memory allocation behavior we would like to see the above being fixed.


THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Did not try

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :

Quite difficult as it happens in our appilication which is heavily
using async I/o / Threading with the VM as an embedded extension Can not
just give a couple of lines of Java

Pls. have a look at the code. By doing a simple code inspection one
can see the memory is being used after doing a free() on the same memory,
just swap the 2 lines of code.

ERROR MESSAGES/STACK TRACES THAT OCCUR :

not a crash. We see >100 Mb of memory being allocated on VM boot. This is
somewhat difficult to explain to customers running this in a small setup.

REPRODUCIBILITY :
This bug can be reproduced always.

CUSTOMER SUBMITTED WORKAROUND :

disable async I/O. Then the L2 thread calls are not done. This is however
not acceptable as performance really goes down doing so.