While working on JDK-8350441 I realized that the current Windows implementation to use placeholders for reservations are broken if we ever fallback to using the part that performs discontiguous heap reservation.

To understand this bug you first need to understand how and why we use the placeholder mechanism. From zMapper_windows.cpp:

// Memory reservation, commit, views, and placeholders.
//
// To be able to up-front reserve address space for the heap views, and later
// multi-map the heap views to the same physical memory, without ever losing the
// reservation of the reserved address space, we use "placeholders".
//
// These placeholders block out the address space from being used by other parts
// of the process. To commit memory in this address space, the placeholder must
// be replaced by anonymous memory, or replaced by mapping a view against a
// paging file mapping. We use the later to support multi-mapping.
//
// We want to be able to dynamically commit and uncommit the physical memory of
// the heap (and also unmap ZPages), in granules of ZGranuleSize bytes. There is
// no way to grow and shrink the committed memory of a paging file mapping.
// Therefore, we create multiple granule-sized page file mappings. The memory is
// committed by creating a page file mapping, map a view against it, commit the
// memory, unmap the view. The memory will stay committed until all views are
// unmapped, and the paging file mapping handle is closed.
//
// When replacing a placeholder address space reservation with a mapped view
// against a paging file mapping, the virtual address space must exactly match
// an existing placeholder's address and size. Therefore we only deal with
// granule-sized placeholders at this layer. Higher layers that keep track of
// reserved available address space can (and will) coalesce placeholders, but
// they will be split before being used.

And the way we implement this is through the callbacks in zVirtualMemory_windows.cpp:

      // Each reserved virtual memory address area registered in _manager is
      // exactly covered by a single placeholder. Callbacks are installed so
      // that whenever a memory area changes, the corresponding placeholder
      // is adjusted.
      //
      // The create and grow callbacks are called when virtual memory is
      // returned to the memory manager. The new memory area is then covered
      // by a new single placeholder.
      //
      // The destroy and shrink callbacks are called when virtual memory is
      // allocated from the memory manager. The memory area is then is split
      // into granule-sized placeholders.
      //
      // See comment in zMapper_windows.cpp explaining why placeholders are
      // split into ZGranuleSize sized placeholders.

So, we have the expectation that all memory areas in the memory manager should be covered by exactly one placeholder. We implement that by having the callbacks disabled while we initialize the reserved memory for the heap. This works as long as we get a contiguous memory reservation, and the code has various mechanisms to really try to get contiguous memory for the heap. However, if all those attempts fail, we have a fallback to reserve discontiguous memory. That mode uses interval halving to reserve exactly around the memory that is blocking use from getting a contiguous memory reservation. An example of this would be a request to reserve four "granules" (2MB), but the forth granule is already reserved:

+--A--+--B--+--C--+--D--+
                       D is pre-reserved

After failing to reserve the four granules (A, B, C, D), the code will split the range into two halves (A, B) and (C, D), and try to reserve them individually. It will succeed to reserve (A, B) but not (C, D). So, the code registers (A, B) and proceeds to split (C, D) into two parts (C) and (D), and try to reserve them individually. It will succeed with (C) but fail with (D). So, the code registers (C). When (C) is registered, the code sees that (A, B) and (C) are adjacent and fuse them into one region (A, B, C). The problem is that we don't have any callbacks to also fuse the placeholders, so we are left with reservation placeholders over (A, B) and (C). Later one, when we want to use use (A,B) for the heap, the code works under the impression that we have on single placeholder over (A, B, C), so it tries to split that memory are into two placeholders (A, B) and (C). This fails with a fatal error, because Windows will refuse make this split since we already have split the placeholder.