I believe I have found a bug in Serial GC.

During phase2 of Serial full GC, we forward all live objects. When an object does *not* move, we initialize its mark-word. Notice that this makes the object no longer appear 'live':

  static void forward_obj(oop obj, HeapWord* new_addr) {
    prefetch_write_scan(obj);
    if (cast_from_oop<HeapWord*>(obj) != new_addr) {
      FullGCForwarding::forward_to(obj, cast_to_oop(new_addr));
    } else {
      assert(obj->is_gc_marked(), "inv");
      // This obj will stay in-place. Fix the markword.
      obj->init_mark(); <== here
    }
  }

Later, during phase 3, we adjust all references in all live-objects. However, since not-moving live objects appear no longer live, they would not get their references adjusted. When that happens, heap would be corrupted.

Notice that I could not reproduce with vanilla JDK. I believe this may be very hard to reproduce because almost all objects would be moved, and for the few objects towards the bottom of the heap that don't move, they probably tend to not need refs updated (primitive arrays?). Also, most full GCs typically skip dense unmovable prefix, further masking the problem.

The problem can be reproduced with Lilliput project, which has compact identity hash-code, which can produce more not-moving objects higher up in heap:

check out openjdk/lilliput
run:
make test TEST=gc/TestFullGCALot.java TEST_VM_OPTS="-XX:+UnlockExperimentalVMOptions -XX:+UseCompactObjectHeaders -XX:+UseSerialGC"

it might take a couple of attempts. For me it fails quite reliably.

I am not totally sure about this, I feel like I must be missing something here. If this bug is real, then it goes all the way back to jdk8 (perhaps earlier, I haven't checked < 8).