Problem has been seen on both 1.4.0_01 and 1.4.1 RC with application that uses a query interface to search for data. Customer indicates the problem is seen after entering queries which generate a parsing error (i.e., those that contain key words like OR, AND, NOT). About a minute after that error, the application crashes and a core file is generated.

The error on the server side is:

>> Server output:
>> Unexpected Signal : 11 occurred at PC=0xFFFFFFFF7D330C70
>> Function=JVM_SetClassSigners+0xE180
>> Library=/sbc/avse1/j2sdk1.4.0_01/jre/lib/sparcv9/server/libjvm.so
>>
>>
>> Dynamic libraries:
>> 0x100000000 /sbc/avse1/j2sdk1.4.0_01/bin/sparcv9/java
>> 0xffffffff7f200000 /usr/lib/sparcv9/libthread.so.1
>> 0xffffffff7f400000 /usr/lib/sparcv9/libdl.so.1
>> 0xffffffff7ef00000 /usr/lib/sparcv9/libc.so.1
>> 0xffffffff7ee00000
>> /usr/platform/SUNW,Ultra-80/lib/sparcv9/libc_psr.so.1
>> 0xffffffff7d000000
>> /sbc/avse1/j2sdk1.4.0_01/jre/lib/sparcv9/server/libjvm.s
>> o
>> 0xffffffff7ce00000 /usr/lib/sparcv9/libCrun.so.1
>> 0xffffffff7cc00000 /usr/lib/sparcv9/libsocket.so.1
>> 0xffffffff7ca00000 /usr/lib/sparcv9/libnsl.so.1
>> 0xffffffff7c800000 /usr/lib/sparcv9/libm.so.1
>> 0xffffffff7da00000 /usr/lib/sparcv9/libw.so.1
>> 0xffffffff7c500000 /usr/lib/sparcv9/libmp.so.2
>> 0xffffffff7c200000
>> /sbc/avse1/j2sdk1.4.0_01/jre/lib/sparcv9/native_threads/
>> libhpi.so
>> 0xffffffff7c000000
>> /sbc/avse1/j2sdk1.4.0_01/jre/lib/sparcv9/libverify.so
>> 0xffffffff7be00000
>> /sbc/avse1/j2sdk1.4.0_01/jre/lib/sparcv9/libjava.so
>> 0xffffffff7bb00000
>> /sbc/avse1/j2sdk1.4.0_01/jre/lib/sparcv9/libzip.so
>> 0xffffffff13e00000
>> /usr/lib/locale/en_US.ISO8859-1/sparcv9/en_US.ISO8859-1.
>> so.2
>> 0xffffffff11c00000 /sbc/avse1/AV21/bin64/libavsjava30.so
>> 0xffffffff11a00000 /usr/ucblib/sparcv9/libucb.so.1
>> 0xffffffff11800000 /usr/lib/sparcv9/libresolv.so.2
>> 0xffffffff11600000 /usr/lib/sparcv9/libelf.so.1
>> 0xffffffff10500000
>> /sbc/avse1/j2sdk1.4.0_01/jre/lib/sparcv9/libnet.so
>> 0xffffffff01200000
>> /sbc/avse1/j2sdk1.4.0_01/jre/lib/sparcv9/librmi.so
>> 0xfffffffefe800000 /sbc/avse1/AV21/bin64/libavsjavalinguistics30.so
>>
>> Local Time = Fri Aug 2 13:53:24 2002
>> Elapsed Time = 379
>> #
>> # HotSpot Virtual Machine Error : 11
>> # Error ID : 4F530E43505002D5 01
>> # Please report this error at
>> # http://java.sun.com/cgi-bin/bugreport.cgi
>> #
>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (1.4.0_01-b03 mixed mode)
>> #
>> # An error report file has been saved as hs_err_pid23755.log.
>> # Please refer to the file for further information.
>> #
>> 08:02:2002@01:53:28;Server Down! (rmi://otter:1099/IYPSearchServer)
>> is not resp
>> onding.
>> 08:02:2002@01:53:28;Server Down! (rmi://otter:1099/IYPSearchServer)
>> is not resp
>> onding.


Attached to the process with dbx and got this stack trace:

>> Reading java
>> Reading ld.so.1
>> Reading libthread.so.1
>> Reading libdl.so.1
>> Reading libc.so.1
>> Reading libc_psr.so.1
>> Reading libjvm.so
>> Reading libCrun.so.1
>> Reading libsocket.so.1
>> Reading libnsl.so.1
>> Reading libm.so.1
>> Reading libw.so.1
>> Reading libmp.so.2
>> Reading libhpi.so
>> Reading libverify.so
>> Reading libjava.so
>> Reading libzip.so
>> Reading en_US.ISO8859-1.so.2
>> Reading libavsjava30.so
>> Reading libucb.so.1
>> Reading libresolv.so.2
>> Reading libelf.so.1
>> Reading libnet.so
>> Reading librmi.so
>> detected a multithreaded program
>> Attached to process 23662 with 22 LWPs
>> t@1 (l@1) stopped in ___lwp_cond_wait at 0xffffffff7efa3ecc
>> 0xffffffff7efa3ecc: ___lwp_cond_wait+0x0004: ta %icc,%g0 + 64
>> (/opt/SUNWspro/bin//../WS5.0/bin/sparcv9/dbx) threads
>> > t@1 a l@1 ?() running in ___lwp_cond_wait()
>> t@2 b l@2 ?() running in _signotifywait()
>> t@3 a l@3 ?() sleep on 0xffffffff7f328150 in
>> __lwp_sema_wait()
>> t@4 b l@4 _start() running in
>> ___lwp_cond_wait()
>> t@5 b l@6 _start() running in
>> ___lwp_cond_wait()
>> t@6 b l@7 _start() running in
>> ___lwp_cond_wait()
>> t@7 b l@8 _start() running in _libc_poll()
>> t@8 b l@9 _start() running in
>> ___lwp_cond_wait()
>> t@9 b l@10 _start() sleep on 0xffffffff7d880a08 in
>> __lwp_sema_wa
>> it()
>> t@10 b l@11 _start() running in
>> ___lwp_cond_wait()
>> t@11 b l@12 _start() running in
>> ___lwp_cond_wait()
>> t@12 b l@13 _start() running in _libc_poll()
>> t@13 b l@14 _start() running in _libc_poll()
>> t@14 b l@15 _start() running in
>> ___lwp_cond_wait()
>> t@15 b l@16 _start() running in _so_accept()
>> t@16 b l@17 _start() running in _libc_poll()
>> t@17 b l@18 _start() running in _so_accept()
>> t@18 b l@19 _start() running in
>> ___lwp_cond_wait()
>> t@19 b l@20 _start() running in
>> ___lwp_cond_wait()
>> t@21 b l@22 _start() running in
>> ___lwp_cond_wait()
>> t@23 b l@24 _start() running in _libc_poll()
>> (/opt/SUNWspro/bin//../WS5.0/bin/sparcv9/dbx) cont
>> Reading libavsjavalinguistics30.so
>> t@4 (l@4) signal SEGV (no mapping at the fault address) in
>> ContiguousSpace::prep
>> are_for_compaction at 0xffffffff7d330c70
>> 0xffffffff7d330c70: prepare_for_compaction+0x0218: ld [%g5
>> + 0x28], %g
>> 4
>> (/opt/SUNWspro/bin//../WS5.0/bin/sparcv9/dbx) where
>> current thread: t@4
>> =>[1] ContiguousSpace::prepare_for_compaction(0x1001871d0,
>> 0xffffffff38701238, 0
>> x0, 0x0, 0x0, 0x0), at 0xffffffff7d330c70
>> [2] Generation::prepare_for_compaction(0x100187020,
>> 0xffffffff38701238, 0x0, 0
>> x0, 0x0, 0x0), at 0xffffffff7d33ccfc
>> [3] GenCollectedHeap::prepare_for_compaction(0x100186ed0,
>> 0xffffffff7d630f6c,
>> 0x0, 0x1, 0x0, 0x27150), at 0xffffffff7d371f90
>> [4] MarkSweep::mark_sweep_phase2(0x1, 0xffffffff38701420, 0x0,
>> 0xffffffff7d373
>> f3c, 0x100110db0, 0x0), at 0xffffffff7d3737b4
>> [5] MarkSweep::invoke_at_safepoint(0xb6c8, 0x0, 0x0, 0x0, 0x0, 0x0),
>> at 0xffff
>> ffff7d373fd4
>> [6] GenCollectedHeap::do_collection(0x0, 0x1, 0x0,
>> 0xffffffff7d79a000, 0xfffff
>> fff7d89a488, 0xffffffff7d7b9308), at 0xffffffff7d321c04
>> [7] GenCollectedHeap::do_full_collection(0x100186ed0, 0x0, 0x1,
>> 0xffffffff0170
>> 0d98, 0xffffffff7d2f6f80, 0x0), at 0xffffffff7d37c91c
>> [8] VM_Operation::evaluate(0xffffffff01700d58, 0xffffffff7efa1218,
>> 0x0, 0x1001
>> 111b0, 0xffffffff7d2f7614, 0x0), at 0xffffffff7d31ec70
>> [9] VMThread::evaluate_operation(0x1001ebc00, 0xffffffff01700d58,
>> 0xc800, 0xc8
>> 18, 0xffffffff7d43fbe0, 0x0), at 0xffffffff7d31ead0
>> [10] VMThread::loop(0xbc00, 0xa800, 0xaba0, 0xa800, 0xaaa0, 0x9c00),
>> at 0xffff
>> ffff7d43fc8c
>> [11] VMThread::run(0x1001ebc00, 0xffffffff7f322000,
>> 0xffffffff38701b58, 0x0, 0
>> x0, 0x0), at 0xffffffff7d43f040
>> [12] _start(0x1001ebc00, 0xffffffff7f324c00, 0x0, 0x1,
>> 0xffffffff7f322000, 0x0
>> ), at 0xffffffff7d344d6c
>> (/opt/SUNWspro/bin//../WS5.0/bin/sparcv9/dbx)threads
>> t@1 a l@1 ?() running in ___lwp_cond_wait()
>> t@2 b l@2 ?() running in _signotifywait()
>> t@3 a l@3 ?() sleep on 0xffffffff7f328150 in
>> __lwp_sema_wait()
>> *> t@4 b l@4 _start() signal SIGSEGV in
>> prepare_for_compaction()
>> t@5 b l@6 _start() running in
>> ___lwp_cond_wait()
>> t@6 b l@7 _start() running in
>> ___lwp_cond_wait()
>> t@7 b l@8 _start() running in _libc_poll()
>> t@8 b l@9 _start() running in
>> ___lwp_cond_wait()
>> t@9 b l@10 _start() sleep on 0xffffffff7d880a08 in
>> __lwp_sema_wa
>> it()
>> t@10 b l@11 _start() running in
>> ___lwp_cond_wait()
>> t@11 b l@12 _start() running in
>> ___lwp_cond_wait()
>> t@12 b l@13 _start() running in
>> ___lwp_mutex_lock()
>> t@13 b l@14 _start() running in _libc_poll()
>> t@14 b l@15 _start() running in
>> ___lwp_cond_wait()
>> t@15 b l@16 _start() running in _so_accept()
>> t@16 b l@17 _start() running in _libc_poll()
>> t@17 b l@18 _start() running in _so_accept()
>> t@18 b l@19 _start() running in
>> ___lwp_cond_wait()
>> t@19 b l@20 _start() running in
>> ___lwp_cond_wait()
>> t@21 b l@22 _start() running in
>> ___lwp_cond_wait()
>> t@23 b l@24 _start() running in _libc_poll()
>> t@26 b l@27 ?() running in __lwp_sema_wait()
>>



The error on the client side is:

>> otter% java com.smartpages.searchengine.testcases.ListingSearchTest
>> client_confi
>> g.xml "GE Capital IT Solutions"
>> Initializing..
>> Creating EventService.
>> LogEventHandler is subscribing to handle events of type
>> com.smartpages.searcheng
>> ine.common.LogEvent using filter null
>> AlertEventHandler is subscribing to handle events of type
>> com.smartpages.searche
>> ngine.common.AlertEvent using filter null
>> CommandEventHandler is subscribing to handle events of type
>> com.smartpages.searc
>> hengine.common.CommandEvent using filter null
>> Loading configuration file(client_config.xml)
>> Debug Log printing to: System.out
>> Error Log printing to: System.err
>> Event Log printing to: System.out
>> 08:02:2002@01:32:34;There was a problem with the remote procedure call.
>> Exception Message:null
>> java.lang.NullPointerException
>> at
>> com.smartpages.searchengine.testcases.ListingSearchTest.run(ListingSe
>> archTest.java:73)
>> at
>> com.smartpages.searchengine.testcases.ListingSearchTest.main(ListingS
>> earchTest.java:44)
>> Done with test.


Java command line args:

/usr/j2sdk1.4.0_01/bin/java -server -d64 -Xmx512M -DPORT=1100 -Djava.security.policy=custom.policy
com.smartpages.searchengine.IYPSearchEngineServer -config /dev_data/avse2/avserver/config_serve.xml &

The customer indicates they are using the 64-bit VM because they see improved performance during peak traffic.

Increasing initial and maximum size of overall heap, as well as initial and maximum size of permanent generation, has no effect. Running with alternate thread library has no effect. Access to the customer's testing environment is being provided separately.

The output from 1.4.1 with-XX:+PrintHeapAtGC and -verbose:class is attached.