Design doc: https://docs.google.com/document/d/17RV8W6iqJj9HhjGSO9k1IXfs_9lvHDNgxxr8pFoeDys/edit?usp=sharing

As described in Caching Java Heap Objects [1], all archived Java objects are initially dormant objects when open archive heap regions are mapped to the runtime Java heap. A dormant object becomes a live object (materialized) when the associated archived class is loaded or initialized at runtime. For example, a mirror object is materialized when the VM restores the mirror during the loading process of an archived class at runtime. All objects directly or indirectly referenced from that mirror object also become live (with this proposal). After that point, G1 can find those objects and mark them live during GC operations.

A live object in the open archive heap region should not become dormant again, which was a restriction and a design decision for simplified GC requirements. The purpose of that was to ensure all outgoing pointers from the open archive region(s) were updated by GC correctly. With the only support for mirrors, resolved constant pool arrays, and a small set of selected JDK classes’ static field subgraphs in the open archive heap region, the limitation was acceptable.

With the latest design that enhances the subgraph archiving for more general class preinitialization, runtime may encounter cases where a live archived object in the open archive heap region may become dormant again. Those are not common but possible cases, which may be included unintentionally. Restricting the preinitialization support for final or @stable static field can help avoid the situation. Addressing this issue will enable a broader range of support for pre-initialization for application classes.

Following is a simple test to demonstrate the particular case.

public class A {
  static B b = new B();

  void foo () {
    b.addD();
    b.update();
  }

  public static void main(String args[]) {
    (new A()).foo();
  }
}

class B {
  C c;

  public B() {
    c = new C();
  }

  public void addD() {
    c.addD();
  }

  public void update() {
    c = new C();
  }
}

class C {
  D d;

  public void addD() {
    d = new D();
  }
}

class D {}

If class A is pre-initialized, the preserved subgraph from A’s mirror is mirror->B->C. All objects in the subgraph become live when materialized at runtime. During Java code execution, C->D reference is added. D is not an archived heap object. If the reference of B->C is updated to point to a new C instance, the original archived C instance is no longer reachable and becomes dormant again, and D is not reachable either from that point and can be garbage collected. G1 currently does not update the reference from the archived C to D when D is GC’ed, so the dormant C would contain a stale reference. That is not an issue for normal executions (including production jobs, tools execution, etc), but may cause confusions and issues when an agent is walking the heap regions.

One possible solution of handling it: After marking live objects in the open archive heap regions (ORAC), do another iteration of the OARC objects and clear out-going references from objects that are not 'live'. That approach does not require tracking the OARC object state transition (dormant->live->dormant), and is guaranteed to clear all stale references.

[1] Caching Java Heap Objects (https://wiki.openjdk.java.net/display/HotSpot/Caching+Java+Heap+Objects)