nsk/jvmti/GetImplementedInterfaces/getintrf007 timed out, hung during shutdown in NMT on ThreadLocal lock

nsk/jvmti/GetImplementedInterfaces/getintrf007

Process has only one thread left:

0:000> ~
. 0 Id: 21f4.3bd4 Suspend: 0 Teb: 000007ff`fff86000 Unfrozen

With stack:

0:000> kP
Child-SP RetAddr Call Site
00000000`40a9ee98 000007fe`fd4f10dc ntdll!NtWaitForSingleObject+0xa
00000000`40a9eea0 00000000`719f04ec KERNELBASE!WaitForSingleObjectEx+0x79
00000000`40a9ef40 00000000`719a9261 jvm!ThreadCritical::ThreadCritical(void)+0x5c [c:\jprt\t\p1\004345.ctornqvi\s\src\os\windows\vm\threadcritical_windows.cpp @ 71]
00000000`40a9ef70 00000000`719316ae jvm!MemTracker::Tracker::record(
unsigned char * addr = 0x00000000`7202c5e0 "???",
unsigned int64 size = 0x694a30,
unsigned short flags = 1,
unsigned char * pc = 0x00000000`719c903d "H???")+0x1a1 [c:\jprt\t\p1\004345.ctornqvi\s\src\share\vm\services\memtracker.cpp @ 865]
00000000`40a9efd0 00000000`719c8d6d jvm!os::free(
void * memblock = 0x00000000`006c0000,
unsigned short memflags = 0x7b01)+0x1ae [c:\jprt\t\p1\004345.ctornqvi\s\src\share\vm\runtime\os.cpp @ 806]
00000000`40a9f060 00000000`719c903d jvm!GrowableArray<PausePhase>::~GrowableArray<PausePhase>(void)+0x4d [c:\jprt\t\p1\004345.ctornqvi\s\src\share\vm\utilities\growablearray.hpp @ 193]
00000000`40a9f090 00000000`71f3d1e0 jvm!TimePartitions::~TimePartitions(void)+0x1d [c:\jprt\t\p1\004345.ctornqvi\s\src\share\vm\gc_implementation\shared\gctimer.cpp @ 110]
00000000`40a9f0c0 00000000`71f33e0f jvm!`dynamic atexit destructor for 'PSScavenge::_gc_timer''(void)+0x10
00000000`40a9f0f0 00000000`71f34059 jvm!_CRT_INIT(
void * hDllHandle = <Value unavailable error>,
unsigned long dwReason = <Value unavailable error>,
void * lpreserved = <Value unavailable error>)+0xf7 [f:\dd\vctools\crt_bld\self_64_amd64\crt\src\crtdll.c @ 415]
00000000`40a9f130 00000000`772e43a1 jvm!__DllMainCRTStartup(
void * hDllHandle = 0x00000000`71620000,
unsigned long dwReason = 0,
void * lpreserved = 0x00000000`00000001)+0xe5 [f:\dd\vctools\crt_bld\self_64_amd64\crt\src\crtdll.c @ 526]
00000000`40a9f170 00000000`772e41b0 ntdll!LdrShutdownProcess+0x1db
00000000`40a9f290 00000000`740b0ccd ntdll!RtlExitUserProcess+0x90
00000000`40a9f2c0 00000000`7191eb75 msvcr100!cinit+0x20d
00000000`40a9f330 00000000`71978a82 jvm!vm_direct_exit(
int code = 0n0)+0x15 [c:\jprt\t\p1\004345.ctornqvi\s\src\share\vm\runtime\java.cpp @ 616]
00000000`40a9f360 00000000`71977955 jvm!VM_Operation::evaluate(void)+0x102 [c:\jprt\t\p1\004345.ctornqvi\s\src\share\vm\runtime\vm_operations.cpp @ 61]
00000000`40a9f3d0 00000000`71977e36 jvm!VMThread::evaluate_operation(
class VM_Operation * op = 0x00000000`00000000)+0x155 [c:\jprt\t\p1\004345.ctornqvi\s\src\share\vm\runtime\vmthread.cpp @ 361]
00000000`40a9f4b0 00000000`7197813b jvm!VMThread::loop(void)+0x366 [c:\jprt\t\p1\004345.ctornqvi\s\src\share\vm\runtime\vmthread.cpp @ 479]
00000000`40a9f680 00000000`719ec57e jvm!VMThread::run(void)+0xcb [c:\jprt\t\p1\004345.ctornqvi\s\src\share\vm\runtime\vmthread.cpp @ 268]
00000000`40a9f6f0 00000000`740b1d9f jvm!java_start(
class Thread * thread = 0x00000000`0b0af800)+0xbe [c:\jprt\t\p1\004345.ctornqvi\s\src\os\windows\vm\os_windows.cpp @ 429]
00000000`40a9f930 00000000`740b1e3b msvcr100!endthreadex+0x43
00000000`40a9f960 00000000`771b652d msvcr100!endthreadex+0xdf
00000000`40a9f990 00000000`772ec541 kernel32!BaseThreadInitThunk+0xd
00000000`40a9f9c0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d


Looks like we're shutting down the process and the CRT is calling destructors for static objects. In the destructor for PSScavenge::_gc_timer we're trying to free some memory which calls in to NMT who attempts to take a ThreadLocal lock, and we end up hanging in WaitForSingleObject.

     if (initialized) {
        DWORD ret = WaitForSingleObject(lock_event, INFINITE);
 
0:000> dt lock_event
jvm!lock_event
0x00000000`00000080
Void

No handles are allocated according to windbg

0:000> !handle
0 Handles
Type Count


Options: -server -Xcomp -XX:MaxRAMFraction=8 -XX:+CreateMinidumpOnCrash -XX:NativeMemoryTracking=detail -XX:ReservedCodeCacheSize=256M