StubRoutines::_cont_thaw calls prepare_thaw() which returns the maximum stack space that would take to copy the frames currently stored in the top stackChunk. In prepare_thaw() we only check for stack overflow as related to the shadow_zone_safe_limit(). If that check succeeds, then back in the stub we bump the stack pointer by this maximum size and then call thaw_entry() where we copy frames from the heap to the stack within the allocated space.

But this increment of the stack pointer doesn't play nice with the way Windows sets up and manages stack pages. When a thread is created the stack is divided in 3 memory regions: regular committed pages, guard pages, reserved pages. The first pages are committed and the thread can read/write to them with no issues. The next pages(~2/3) are guard pages, which are committed but have the PAGE_GUARD attribute. When the thread tries to access a guard page the first time, the PAGE_GUARD attribute is removed and a STATUS_GUARD_PAGE_VIOLATION exception is generated. It is handled by the OS which adds a new guard page from the reserved region, i.e. this guard page mechanism allows automatic stack growth. The rest of the stack are reserved pages(rest of 1M if default stack size). If we try to access the reserved region directly we get an EXCEPTION_ACCESS_VIOLATION (0xc0000005) exception instead and our exception handler (if installed) is called, otherwise program terminates. Here is a blog with some visuals: https://devblogs.microsoft.com/oldnewthing/20220203-00/?p=106215.

The problem then is that we can bump the stack pointer too much and set it to point to somewhere in the reserved region. When we then execute the call instruction, we get an EXCEPTION_ACCESS_VIOLATION exception, but because we cannot access the memory at the current stack pointer, we cannot call any method anymore, including the exception handler. So the program terminates abruptly with exit code 0xc0000005.

I attached a simple Java test that reproduces the issue.