OS/Platform: Windows Vista x32 ( also it can be reproduced on WinVista x64 with x32 JDK )
VM opts: -Xcomp
JDK:
  failed on: 6u3 b01, 6u2( all builds ), jdk6u2 ER Fiducia - windows-i586
  passed on: 6u1
  also passed: without -Xcomp opt (all updates/builds), on windows-amd64 (all upd/bld)
Failure type: VM crash
    
Test(s):
  JCK 6a b14
api/java_awt/SystemTray/index.html#SystemTrayMisc
api/java_awt/SystemTray/index.html#SystemTraySupported
api/java_awt/TrayIcon/index.html#ActionListeners
api/java_awt/TrayIcon/index.html#Ctors
api/java_awt/TrayIcon/index.html#DisplayMessage
api/java_awt/TrayIcon/index.html#FieldsAccessors
api/java_awt/TrayIcon/index.html#MouseListeners
api/java_awt/TrayIcon/index.html#MouseMotionListeners
      
How to reproduce:
See comments to get access on test Vista hosts:
On stt-27.russia or stt-40.russia under MKS bash:
------------
export JAVA_HOME=Z:/Links/stt/jdk_promotions/JDK6u3/b01/binaries/windows-i586/jdk1.6.0_03
export JAVA_COMMON_OPTS="-client -Xcomp -Xfuture"
export CP="Z:/Links/stt/jck_promotions/6a/fcs/b14/binaries/JCK-runtime-6a/classes"
export JAVA_TEST="javasoft.sqe.tests.api.java.awt.SystemTray.SystemTrayMiscTests"

$JAVA_HOME/bin/java.exe $JAVA_COMMON_OPTS -classpath $CP $JAVA_TEST
-------------
or use Z:\jck_workspace\manual_run\dg159268_runs\failed_tests\unsorted\jdk6u3_Vista\rerun.sh
Z is mapped as \\stt-13\export\stt

Additional info:
   hs_* files are attached ( hs_err_pid1872.log - common JDK 6u3 bundle,
                             hs_err_pid5636.log - info from fastdebug bundle )
   
Test output sample:
------------------------------
command: com.sun.jck.lib.ExecJCKTestOtherJVMCmd PATH=Z:\\Links\\stt\\jck_promotions\\6a\\fcs\\b14\\binaries\\JCK-runtime-6a\\lib\\win32;Z:\\Links\\stt\\jck_promotions\\6a\\fcs\\b14\\binaries\\JCK-runtime-6a\\lib\\win32\\jmx;Z:\\Links\\stt\\jck_promotions\\6a\\fcs\\b14\\binaries\\JCK-runtime-6a\\lib\\win32 SystemRoot=C:/Windows TMP=Z:\\Links\\stt\\jck_workspace\\jdk6u2_ER_Fiducia\\jck\\runtime\\WinVista-x86-JCK6.0-runtime\\workDir TEMP=Z:\\Links\\stt\\jck_workspace\\jdk6u2_ER_Fiducia\\jck\\runtime\\WinVista-x86-JCK6.0-runtime\\workDir C:\\jdk\\ER_6_02_Fiducia\\jre1.6.0_02\\bin\\java.exe -client -Xcomp -Djava.ext.dirs=Z:/Links/stt/jck_promotions/6a/fcs/b14/binaries/JCK-runtime-6a/lib/extensions;C:/jdk/ER_6_02_Fiducia/jre1.6.0_02//lib/ext;C:/jdk/ER_6_02_Fiducia/jre1.6.0_02//jre/lib/ext -Xfuture -classpath Z:\\Links\\stt\\jck_promotions\\6a\\fcs\\b14\\binaries\\JCK-runtime-6a\\classes;C:\\jdk\\ER_6_02_Fiducia\\jre1.6.0_02\\lib\\tools.jar -Djava.security.policy=Z:\\Links\\stt\\jck_promotions\\6a\\fcs\\b14\\binaries\\JCK-runtime-6a\\lib\\jck.policy javasoft.sqe.tests.api.java.awt.TrayIcon.MouseMotionListenersTests
----------out1:(0/0)----------
----------out2:(14/485)----------
#
# An unexpected error has been detected by Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6d92e426, pid=3620, tid=468
#
# Java VM: Java HotSpot(TM) Client VM (1.6.0_02-erdist-b20072507 compiled mode)
# Problematic frame:
# V [jvm.dll+0x16e426]
#
# An error report file with more information is saved as hs_err_pid3620.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
result: Failed. unexpected exit code: exit code 1
test result: Failed. unexpected exit code: exit code 1
----------------------------------------------------------



hs_err* :
--------------------------------------------------------------------------------------------------
#
# An unexpected error has been detected by Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6d9e4376, pid=1872, tid=832
#
# Java VM: Java HotSpot(TM) Client VM (1.6.0_03-ea-b01 compiled mode windows-x86)
# Problematic frame:
# V [jvm.dll+0x174376]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#

--------------- T H R E A D ---------------

Current thread (0x0ca5b000): JavaThread "AWT-EventQueue-0" [_thread_in_vm, id=832, stack(0x0d3e0000,0x0d430000)]

siginfo: ExceptionCode=0xc0000005, reading address 0x00000000

Registers:
EAX=0x00000000, EBX=0x0c9f5168, ECX=0x0d42f308, EDX=0x00000000
ESP=0x0d42f2a4, EBP=0x0caff038, ESI=0xffffffff, EDI=0x028000c8
EIP=0x6d9e4376, EFLAGS=0x00010246

Top of Stack: (sp=0x0d42f2a4)
0x0d42f2a4: 028000c8 0ca5b000 0d42f434 00000001
0x0d42f2b4: 0ca5b000 0ca5b000 00000000 00000000
0x0d42f2c4: 012142f8 0c9f5160 0c9f5168 0c9f5554
0x0d42f2d4: 00560000 00560148 0d42f328 0ca5b000
0x0d42f2e4: 6d8c69a5 027fff08 028000c8 0d42f308
0x0d42f2f4: 00000000 00000000 0ca5b000 0d42f558
0x0d42f304: 02800455 00000000 00000000 0ca5b000
0x0d42f314: c10028e9 0000000a 770c13dd 0058c8a0

Instructions: (pc=0x6d9e4376)
0x6d9e4366: 89 44 24 18 8d 9b 00 00 00 00 8b 4c 24 4c 8b 11
0x6d9e4376: 8b 02 8b 70 04 85 f6 c6 44 24 50 00 75 04 33 c9


Stack: [0x0d3e0000,0x0d430000], sp=0x0d42f2a4, free space=316k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [jvm.dll+0x174376]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J sun.awt.windows.WToolkit.isProtectedMode()Z
C 0x0d42f478

[error occurred during error reporting (printing Java stack), id 0xc0000005]


--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
=>0x0ca5b000 JavaThread "AWT-EventQueue-0" [_thread_in_vm, id=832, stack(0x0d3e0000,0x0d430000)]
  0x0ca59800 JavaThread "AWT-Windows" daemon [_thread_in_native, id=2220, stack(0x0d350000,0x0d3a0000)]
  0x0ca59000 JavaThread "AWT-Shutdown" [_thread_blocked, id=12, stack(0x0ce60000,0x0ceb0000)]
  0x0cafe000 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=2116, stack(0x0cdd0000,0x0ce20000)]
  0x0c910400 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=1460, stack(0x0cb40000,0x0cb90000)]
  0x00e07c00 JavaThread "CompilerThread0" daemon [_thread_blocked, id=1000, stack(0x0c8b0000,0x0c900000)]
  0x00e07400 JavaThread "Attach Listener" daemon [_thread_blocked, id=1052, stack(0x0c820000,0x0c870000)]
  0x00e03400 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=2028, stack(0x0c790000,0x0c7e0000)]
  0x00df5000 JavaThread "Finalizer" daemon [_thread_blocked, id=1304, stack(0x0c700000,0x0c750000)]
  0x00df1400 JavaThread "Reference Handler" daemon [_thread_blocked, id=2272, stack(0x0c670000,0x0c6c0000)]
  0x00d71c00 JavaThread "main" [_thread_blocked, id=2988, stack(0x00150000,0x001a0000)]

Other Threads:
  0x00ded000 VMThread [stack: 0x0c620000,0x0c670000] [id=1236]
  0x0c910c00 WatcherThread [stack: 0x0cbd0000,0x0cc20000] [id=1784]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
 def new generation total 960K, used 430K [0x04620000, 0x04720000, 0x04b00000)
  eden space 896K, 48% used [0x04620000, 0x0468bab0, 0x04700000)
  from space 64K, 0% used [0x04700000, 0x04700000, 0x04710000)
  to space 64K, 0% used [0x04710000, 0x04710000, 0x04720000)
 tenured generation total 4096K, used 0K [0x04b00000, 0x04f00000, 0x08620000)
   the space 4096K, 0% used [0x04b00000, 0x04b00000, 0x04b00200, 0x04f00000)
 compacting perm gen total 12288K, used 4381K [0x08620000, 0x09220000, 0x0c620000)
   the space 12288K, 35% used [0x08620000, 0x08a67758, 0x08a67800, 0x09220000)
No shared spaces configured.

Dynamic libraries:
0x00400000 - 0x00423000 Z:\Links\stt\jdk_promotions\JDK6u3\b01\binaries\windows-i586\jdk1.6.0_03\bin\java.exe
0x77690000 - 0x777e0000 C:\Windows\SysWOW64\ntdll.dll
0x76b40000 - 0x76c50000 C:\Windows\syswow64\kernel32.dll
0x76e80000 - 0x76f3f000 C:\Windows\syswow64\ADVAPI32.dll
0x771e0000 - 0x772d0000 C:\Windows\syswow64\RPCRT4.dll
0x75890000 - 0x758f0000 C:\Windows\syswow64\Secur32.dll
0x7c340000 - 0x7c396000 Z:\Links\stt\jdk_promotions\JDK6u3\b01\binaries\windows-i586\jdk1.6.0_03\jre\bin\msvcr71.dll
0x6d870000 - 0x6dac0000 Z:\Links\stt\jdk_promotions\JDK6u3\b01\binaries\windows-i586\jdk1.6.0_03\jre\bin\client\jvm.dll
0x76fd0000 - 0x770a0000 C:\Windows\syswow64\USER32.dll
0x75d90000 - 0x75e20000 C:\Windows\syswow64\GDI32.dll
0x751f0000 - 0x75223000 C:\Windows\system32\WINMM.dll
0x770a0000 - 0x7714a000 C:\Windows\syswow64\msvcrt.dll
0x75c40000 - 0x75d84000 C:\Windows\syswow64\ole32.dll
0x76f40000 - 0x76fcc000 C:\Windows\syswow64\OLEAUT32.dll
0x751b0000 - 0x751e8000 C:\Windows\system32\OLEACC.dll
0x75e20000 - 0x75e80000 C:\Windows\system32\IMM32.DLL
0x75f20000 - 0x75fe7000 C:\Windows\syswow64\MSCTF.dll
0x75490000 - 0x754bc000 C:\Windows\system32\apphelp.dll
0x77670000 - 0x77679000 C:\Windows\syswow64\LPK.DLL
0x758f0000 - 0x7596d000 C:\Windows\syswow64\USP10.dll
0x6d320000 - 0x6d328000 Z:\Links\stt\jdk_promotions\JDK6u3\b01\binaries\windows-i586\jdk1.6.0_03\jre\bin\hpi.dll
0x75b00000 - 0x75b07000 C:\Windows\syswow64\PSAPI.DLL
0x6d820000 - 0x6d82c000 Z:\Links\stt\jdk_promotions\JDK6u3\b01\binaries\windows-i586\jdk1.6.0_03\jre\bin\verify.dll
0x6d3c0000 - 0x6d3df000 Z:\Links\stt\jdk_promotions\JDK6u3\b01\binaries\windows-i586\jdk1.6.0_03\jre\bin\java.dll
0x6d860000 - 0x6d86f000 Z:\Links\stt\jdk_promotions\JDK6u3\b01\binaries\windows-i586\jdk1.6.0_03\jre\bin\zip.dll
0x6d0b0000 - 0x6d1de000 Z:\Links\stt\jdk_promotions\JDK6u3\b01\binaries\windows-i586\jdk1.6.0_03\jre\bin\awt.dll
0x75160000 - 0x751a1000 C:\Windows\system32\WINSPOOL.DRV
0x75690000 - 0x75710000 C:\Windows\system32\uxtheme.dll
0x75be0000 - 0x75c35000 C:\Windows\syswow64\SHLWAPI.dll
0x752f0000 - 0x75484000 C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.6000.16386_none_5d07289e07e1d100\comctl32.dll

VM Arguments:
jvm_args: -Xcomp -Xverify:all
java_command: javasoft.sqe.tests.api.java.awt.SystemTray.SystemTrayMiscTests
Launcher Type: SUN_STANDARD

Environment Variables:
PATH=C:\MKS\bin;C:\MKS\bin\X11;C:\MKS\mksnt;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem
USERNAME=Administrator
SHELL=C:/MKS/mksnt/bash.exe
DISPLAY=:0.0
OS=Windows_NT
PROCESSOR_IDENTIFIER=EM64T Family 15 Model 6 Stepping 4, GenuineIntel



--------------- S Y S T E M ---------------

OS: Windows Vista Build 6000

CPU:total 2 (2 cores per cpu, 1 threads per core) family 15 model 6 stepping 4, cmov, cx8, fxsr, mmx, sse, sse2, sse3

Memory: 4k page, physical 2078032k(1429580k free), swap 4194303k(3698464k free)

vm_info: Java HotSpot(TM) Client VM (1.6.0_03-ea-b01) for windows-x86, built on Jul 3 2007 01:45:06 by "java_re" with unknown MS VC++:1310

time: Mon Jul 30 22:58:14 2007
elapsed time: 4 seconds

--------------------------------------------------------------------------------------------------


Comments:
It looks like workaround for 6490142 related issue.
In WToolkit.java in IsTraySupported() we are calling native method IsProtectedMode() from awt_Toolkit.cpp
It causes VM crash when we are using -Xcomp VM option.