The following guarantee has been intermittently observed during the course of stress-testing GenShen.

I understand that forthcoming changes may make the guarantee moot, but if not the information recorded here may be useful for stress-tests in that section of the collector.

Note that this may be quite difficult to reproduce.

1. Command-line used:

```
$java -XX:+UseShenandoahGC -XX:ShenandoahGCMode=generational -XX:+UnlockDiagnosticVMOptions -XX:+ShenandoahVerify -XX:MaxTenuringThreshold=1 -XX:InitialTenuringThreshold=1 -Xmx4g -Xms4g -XX:+AlwaysPreTouch -Xlog:async -Xlog:gc*=info -jar ./target/HyperAlloc-1.0.jar -d 1200 -p 10 -r 50 -a 3072 -s 1536 -t 4 -b heapothesys.csv.allocation -l heapothesys.csv
```


2. GC log excerpt prior to verification failure:

```
[338.880s][info][gc ] Transition from: normal to: idle
[338.880s][info][gc ] Trigger: Handle Allocation Failure
[338.880s][info][gc ] Transition from: idle to: degenerated
[338.880s][info][gc,ergo ] Free: 0B, Max: 0B regular, 0B humongous, Frag: 0% external, 0% internal; Reserve: 138M, Max: 2048K
[338.880s][info][gc,start ] GC(143) Pause Degenerated YOUNG GC (Update References)
[338.880s][info][gc,task ] GC(143) Using 20 of 20 workers for stw degenerated gc
[338.880s][info][gc ] GC(143) GC cancellation took 0.001s
[338.890s][info][gc,start ] GC(143) Verify After Updating References, Level 4
[339.168s][info][gc ] GC(143) Verify After Updating References, Level 4 (2266527 reachable, 313897 marked)
[339.168s][info][gc ] GC(143) Thread Usr+Sys YOUNG = 22.762, OLD = 0.000, GLOBAL = 0.001
[339.168s][info][gc ] GC(143) OLD has no regions available for transfer to YOUNG
[339.168s][info][gc,start ] GC(143) Verify After Degenerated GC, Level 4
[339.444s][info][gc ] GC(143) Verify After Degenerated GC, Level 4 (2266527 reachable, 313897 marked)
[339.444s][info][gc,ergo ] GC(143) Bad progress for free space: 29773K, need 41943K
[339.444s][info][gc ] GC(143) Cancelling GC: Upgrade To Full GC
[339.444s][info][gc ] GC(143) Degenerate GC upgrading to Full GC
[339.444s][info][gc ] GC(143) Terminating old gc cycle.
[339.444s][info][gc ] GC(143) Cancel marking: OLD
[339.444s][info][gc ] GC(143) Old generation transition from Waiting to Idle
[339.444s][info][gc,start ] GC(143) Verify Before Full GC, Level 4
[339.696s][info][gc ] Average MMU = 9.640
[340.121s][info][gc ] GC(143) Verify Before Full GC, Level 4 (5547281 reachable, 0 marked)
[340.140s][info][gc,phases,start] GC(143) Phase 1: Mark live objects
[340.140s][info][gc,ref ] GC(143) Clearing All SoftReferences
[340.185s][info][gc,ref ] GC(143) Encountered references: Soft: 98, Weak: 339, Final: 0, Phantom: 9
[340.185s][info][gc,ref ] GC(143) Discovered references: Soft: 75, Weak: 127, Final: 0, Phantom: 5
[340.185s][info][gc,ref ] GC(143) Enqueued references: Soft: 1, Weak: 0, Final: 0, Phantom: 0
[340.186s][info][gc,phases ] GC(143) Phase 1: Mark live objects 46.703ms
[340.186s][info][gc,phases,start] GC(143) Phase 2: Compute new object addresses
[340.204s][info][gc,phases ] GC(143) Phase 2: Compute new object addresses 17.296ms
[340.204s][info][gc,phases,start] GC(143) Phase 3: Adjust pointers
[340.229s][info][gc,phases ] GC(143) Phase 3: Adjust pointers 25.238ms
[340.229s][info][gc,phases,start] GC(143) Phase 4: Move objects
[340.326s][info][gc ] GC(143) FullGC done: GLOBAL usage: 1675873696, young usage: 70674768, old usage: 1605198928
[340.326s][info][gc ] GC(143) GC cancellation took 0.882s
[340.326s][info][gc,phases ] GC(143) Phase 4: Move objects 97.044ms
[340.351s][info][gc ] GC(143) Thread Usr+Sys YOUNG = 0.000, OLD = 0.000, GLOBAL = 0.000
[340.351s][info][gc,ergo ] GC(143) Difference (0.000) for thread utilization for each generation is under threshold (3.000)
[340.351s][info][gc,start ] GC(143) Verify After Full Generational GC, Level 4
==============================================================================
Unexpected Error
------------------------------------------------------------------------------
Internal Error at shenandoahVerifier.cpp:414, pid=71781, tid=71807
guarantee(stats.span() <= capacity) failed: After Full Generational GC: generation (OLD) size spanned by regions (775) must not exceed current capacity (990M)
```


3. Some gdb-extracted information:

```(gdb) print *generation
$20 = {
  <CHeapObj<(MEMFLAGS)5>> = {<No data fields>},
  members of ShenandoahGeneration:
  _vptr.ShenandoahGeneration = 0x7fec6c6fcf00 <vtable for ShenandoahOldGeneration+16>,
  _generation_mode = OLD,
  _task_queues = 0x7fec640da060,
  _is_marking_complete = {
    _pad_buf0 = '\000' <repeats 63 times>,
    value = 1 '\001',
    _pad_buf1 = '\000' <repeats 63 times>
  },
  _ref_processor = 0x7fec640da1c0,
  _collection_thread_time_s = 0,
  _affiliated_region_count = 775,
  _used = 1605198928,
  _bytes_allocated_since_gc_start = 0,
  _max_capacity = 1038090240,
  _soft_max_capacity = 1038090240,
  _adjusted_capacity = 1038090240,
  _heuristics = 0x7fec641095e0
}
(gdb) print stats
$21 = (ShenandoahCalculateRegionStatsClosure &) @0x7febe38a8e28: {
  <ShenandoahHeapRegionClosure> = {
    <StackObj> = {<No data fields>},
    members of ShenandoahHeapRegionClosure:
    _vptr.ShenandoahHeapRegionClosure = 0x7fec6c64b178 <vtable for ShenandoahCalculateRegionStatsClosure+16>
  },
  members of ShenandoahCalculateRegionStatsClosure:
  _used = 1605198928,
  _committed = 1625292800,
  _garbage = 0,
  _regions = 775
}
(gdb) print 1038090240/2/1024/1024
$22 = 495
```

4. Stack retrace at error:

```
#7 0x00007fec6b71e949 in report_vm_error (
    file=file@entry=0x7fec6c1dd7b8 "src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp", line=line@entry=414,
    error_msg=error_msg@entry=0x7fec6c1de240 "guarantee(stats.span() <= capacity) failed",
    detail_fmt=detail_fmt@entry=0x7fec6c1de1e0 "%s: generation (%s) size spanned by regions (%lu) must not exceed current capacity (%lu%s)") at src/hotspot/share/utilities/debug.cpp:286
#8 0x00007fec6bf15a24 in ShenandoahGenerationStatsClosure::validate_usage (stats=..., generation=0x7fec640d9f50,
    label=0x7fec6c1ddd0d "After Full Generational GC") at src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp:411
#9 ShenandoahVerifier::verify_at_safepoint (this=0x7fec6411d2c0,
    label=label@entry=0x7fec6c1ddd0d "After Full Generational GC",
    remembered=remembered@entry=ShenandoahVerifier::_verify_remembered_after_full_gc,
    forwarded=forwarded@entry=ShenandoahVerifier::_verify_forwarded_none,
    marked=marked@entry=ShenandoahVerifier::_verify_marked_complete, cset=cset@entry=ShenandoahVerifier::_verify_cset_none,
    liveness=ShenandoahVerifier::_verify_liveness_disable, regions=ShenandoahVerifier::_verify_regions_notrash_nocset,
    gcstate=ShenandoahVerifier::_verify_gcstate_stable) at src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp:848
#10 0x00007fec6bf16a70 in ShenandoahVerifier::verify_after_generational_fullgc (this=<optimized out>)
    at src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp:1084
#11 0x00007fec6be28740 in ShenandoahFullGC::do_it (this=this@entry=0x7febe38a9160,
    gc_cause=gc_cause@entry=GCCause::_shenandoah_upgrade_to_full_gc) at src/hotspot/share/gc/shenandoah/shenandoahFullGC.cpp:355
#12 0x00007fec6be28901 in ShenandoahFullGC::op_full (this=this@entry=0x7febe38a9160,
    cause=cause@entry=GCCause::_shenandoah_upgrade_to_full_gc) at src/hotspot/share/gc/shenandoah/shenandoahFullGC.cpp:173
#13 0x00007fec6be209e9 in ShenandoahDegenGC::op_degenerated_futile (this=0x7fec501b8c30)
    at src/hotspot/share/gc/shenandoah/shenandoahDegeneratedGC.cpp:442
#14 ShenandoahDegenGC::op_degenerated (this=this@entry=0x7fec501b8c30)
    at src/hotspot/share/gc/shenandoah/shenandoahDegeneratedGC.cpp:314
#15 0x00007fec6be20ca8 in ShenandoahDegenGC::entry_degenerated (this=0x7fec501b8c30)
    at src/hotspot/share/gc/shenandoah/shenandoahDegeneratedGC.cpp:87
#16 0x00007fec6bf148bc in VM_ShenandoahDegeneratedGC::doit (this=0x7fec501b8b90)
    at src/hotspot/share/gc/shenandoah/shenandoahVMOperations.cpp:67
#17 0x00007fec6c0ae8ed in VM_Operation::evaluate (this=this@entry=0x7fec501b8b90)
    at src/hotspot/share/runtime/vmOperations.cpp:71
#18 0x00007fec6c0b0364 in VMThread::evaluate_operation (this=this@entry=0x7fec641f5e50, op=0x7fec501b8b90)
    at src/hotspot/share/runtime/vmThread.cpp:281
#19 0x00007fec6c0b0a45 in VMThread::inner_execute (this=this@entry=0x7fec641f5e50, op=<optimized out>)
    at src/hotspot/share/runtime/vmThread.cpp:428
#20 0x00007fec6c0b0c70 in VMThread::loop (this=0x7fec641f5e50) at src/hotspot/share/runtime/vmThread.cpp:495
#21 VMThread::run (this=0x7fec641f5e50) at src/hotspot/share/runtime/vmThread.cpp:175
#22 0x00007fec6c02368d in Thread::call_run (this=this@entry=0x7fec641f5e50) at src/hotspot/share/runtime/thread.cpp:224
#23 0x00007fec6bd10428 in thread_native_entry (thread=0x7fec641f5e50) at src/hotspot/os/linux/os_linux.cpp:739
#24 0x00007fec6cd5044b in start_thread () from /lib64/libpthread.so.0
#25 0x00007fec6c88752f in clone () from /lib64/libc.so.6
```

5. Proximate code:

```
353 if (ShenandoahVerify) {
354 if (heap->mode()->is_generational()) {
355 heap->verifier()->verify_after_generational_fullgc();
356 } else {
357 heap->verifier()->verify_after_fullgc();
358 }
359 }
360
361 // Having reclaimed all dead memory, it is now safe to restore capacities to original values.
362 heap->young_generation()->unadjust_available();
363 heap->old_generation()->unadjust_available();
364
```