A licensee reports the 1.3.1_01 JVM crashes at their customer's site
with the message.(More detail trace information, please see the
attached file "hs_err_pid794.log")


Local Time = Thu Jul 31 19:42:57 2003
Elapsed Time = 7871
#
# The exception above was detected in native code outside the VM
#
# Java VM: Java HotSpot(TM) Client VM (1.3.1_01 mixed mode)
#


There is no reproducible test program, but they suspect it is caused
from thread unsafe in deleteOnExitHook().
Please read their report and check if their opinion is correct or not.

====================================================
CONFIGURATION:
  OS : Solaris 8
  JRE: 1.3.1_01


BEHAVIOR and INVESTIGATION:

The application crashes with the hs_log and core file.
We looked into those files.

When java.io.File.deleteOnExit() is called, it registers a file
which should be deleted to the file list.
On exiting VM, the files in the above list are deleted.
This crash seems to occur because the file list is collapsed.

The below is a part of memory info.

......
00931300: 00 00 00 05 00 00 00 05 00 00 00 05 00 00 00 05 ;................;
00931310: 00 00 00 06 00 00 00 06 00 00 00 06 00 00 00 06 ;................;
00931320: 00 93 0e d0 00 00 00 09 00 00 00 83 00 00 00 09 ;...............;
00931330: 00 00 00 06 00 01 30 0a 00 00 00 00 00 2f 8a a8 ;......0....../.;
00931340: 05 00 00 52 00 00 00 26 00 00 00 00 bb 5d c3 46 ;...R...&....]F;
00931350: 00 00 01 3b 00 00 01 3e 00 00 01 87 00 00 01 89 ;...;...........;
00931360: 00 00 00 00 00 00 00 4b 00 00 00 01 00 00 00 01 ;.......K........;
00931370: 00 00 00 00 ff ff ff ff ff ff ff 3c a3 63 a3 63 ;...........<cc;
00931380: ff f9 f9 f9 f9 f9 f9 f9 ff ff ff ff ff ff ff 00 ;................;
00931390: 65 6a 62 2d 6a 61 72 20 73 72 63 3d 22 70 6a 64 ;ejb-jar src="pjd;
009313a0: 31 32 30 5f 6a 61 72 22 3e 0a 20 20 20 20 3c 65 ;120_jar". <e;
009313b0: 00 00 04 10 3c 2f 65 6a 00 8d 63 c0 ff 24 cc 30 ;....</ej..c.$0; <--- (*)
009313c0: ff ff ff ff 2f 74 6d 70 00 00 00 00 33 35 31 37 ;..../tmp....3517;
009313d0: 00 93 18 08 48 35 39 34 38 37 2f 32 30 30 33 30 ;....H59487/20030;
009313e0: 37 33 31 30 37 33 39 35 30 36 35 30 2e 78 6d 6c ;731073950650.xml;
009313f0: 00 20 20 20 20 3c 65 6a 62 2d 72 65 66 3e 0a 09 ;. <ejb-ref..;
00931400: 3c 65 6a 62 2d 72 65 66 2d 6e 61 6d 65 3e 65 6a ;<ejb-ref-nameej;
00931410: 62 2f 6a 70 2e 67 6f 2e 6b 6f 6b 75 6b 69 6e 2e ;b/jp.go.kokukin.;
00931420: 62 61 73 65 2e 70 6a 64 30 38 33 3c 2f 65 6a 62 ;base.pjd083</ejb;
00931430: 2d 72 65 66 2d 6e 61 6d 00 00 51 88 3c 65 6a 62 ;-ref-nam..Q.<ejb;
00931440: 00 00 00 00 2d 74 79 70 00 8d e8 f8 73 73 69 6f ;....-typ....ssio;
00931450: 00 8b 96 c8 6a 62 2d 72 00 00 00 00 79 70 65 3e ;...jb-r....ype;
00931460: 0a 09 3c 68 6f 6d 65 3e 6a 70 2e 67 6f 2e 6b 6f ;..<homejp.go.ko;
00931470: 6b 75 6b 69 6e 2e 62 61 73 65 2e 70 6a 64 30 38 ;kukin.base.pjd08;
....


The 0x009313b0 is reported as crash line.( Above (*) )

The line shows the list of file which will be deleted.
The struct of list is as follows.

static struct dlEntry {
    struct dlEntry *next; // Next dlEntry ptr
    DELETEPROC deleteProc; // ptr for function which delete the target file
    char name[JVM_MAXPATHLEN + 1]; // name of file
} *deletionList = NULL;

(Every time the deleteOnExit() is called, the above dlEntry is allocated by malloc()
and concatinated to the file list chain.)

The crash seems to occur because the deleteProc of dlEntry is invalid (the
value is 0x3c2f656a at 0x9313b0.)
When program tries to access to the address where deleteProc points, the value is invalid boundary. Then jvm crashes as SIGBUS.

Besides, the adjacent element, *next, also seems invalid value, 0x00000410

Lastly, when deleteOnExit is running, something overwrites the delete file
list data.
Then program tries to refer the broken file list data and use, the crash occurs.

Next we looked into the backtrace of core file(backtrace-en.txt) and found deleteOnExitHook is called by 2 different threads.(in the list, l@1 and l@10)


We saw deleteOnExitHook() in j2se/src/share/native/java/io/io_util.c .

static void
deleteOnExitHook(void) /* Called by the VM on exit */
{
    struct dlEntry *e, *next;
    for (e = deletionList; e; e = next) {
        next = e->next;
        e->deleteProc(e->name);
        free(e);
    }
}

This does not seem thread safe but the commment says,
 
/* File.deleteOnExit support
   No synchronization is done here; that is left to the Java level.
 */

Then we looked the java source code.

The deleteOnExitHook() is called as the below calling sequence.

  Runtime.exit -> Shutdown.exit -> Shutdown.halt
    -> JVM_Halt -> deleteOnExitHook()


As we say, because the deleteOnExitHook() is not threadsafe, caller(Runtime.exit
or Shutdown.exit) should be thradsafe on calling halt.

The following is extarcted Shutdown.exit() source code part from
 j2se/src/share/classes/java/lang/Shutdown.java

==== Shutdown.java ==============>
.....
    static void exit(int status) {
        boolean runMoreFinalizers = false;
        synchronized (lock) {
            if (status != 0) runFinalizersOnExit = false;
            switch (state) {
            case RUNNING: /* Initiate shutdown */
                state = HOOKS;
                break;
            case HOOKS: /* Stall and halt */
                break;
            case FINALIZERS:
                if (status != 0) {
                    /* Halt immediately on nonzero status */
                    halt(status); <------------------(1)
                } else {
                    /* Compatibility with old behavior:
                     * Run more finalizers and then halt
                     */
                    runMoreFinalizers = runFinalizersOnExit;
                }
                break;
            }
        }
        if (runMoreFinalizers) {
            runAllFinalizers();
            halt(status); <------------------(2)
        }
        synchronized (Shutdown.class) {
            /* Synchronize on the class object, causing any other thread
             * that attempts to initiate shutdown to stall indefinitely
             */
            sequence();
            halt(status); <------------------(3)
        }
    }
....
<==== Shutdown.java End ==============

We consider the scenario.
 
 When a thread enters into Shutdown sequence and is executing the halt() in (3),
 other(s) thread enters into Shutdown sequence and tries to execute the halt()
 either in (1) or (2).
 This means deleteOnExitHook is called by 2 different threads. Those 2 threads
 try to update the delete file list at the same time. One updates file list and
 another breaks the very updated file list data.

THEIR CONCLUSION:

 This crash seems caused from the imcomplete synchronization handling in
 exit method in Shutdown.java.

=====================================================