There is a little know debug agent flag called USE_ITERATE_THROUGH_HEAP. It's value is 0x1 and can be specified on the command line using the debugflags=0x1 option. It causes the debug agent to use the JVMTI IterateThroughHeap function rather than use FollowReferences. These APIs are used in support of the VirtualMachine.InstanceCounts command.

Support for USE_ITERATE_THROUGH_HEAP has been broken since at least 2007 due to the following code, which is only executed when using USE_ITERATE_THROUGH_HEAP:

            error = JVMTI_FUNC_PTR(jvmti,ForceGarbageCollection)(jvmti);
            if ( error != JVMTI_ERROR_NONE ) {

                /* Setup callbacks, just need object callback */
                heap_callbacks.heap_iteration_callback = &cbObjectCounter;

                /* Iterate through entire heap, tagged classes only */
                error = JVMTI_FUNC_PTR(jvmti,IterateThroughHeap)
                              (jvmti, JVMTI_HEAP_FILTER_CLASS_UNTAGGED,
                               NULL, &heap_callbacks, &data);

            }

The error check is backwards. It should be "==", not "!=". As a result, if you set the USE_ITERATE_THROUGH_HEAP flag, VirtualMachine.InstanceCounts will always return 0 matches. At a minimum I found the following two tests failed as a result:

com/sun/jdi/InstancesTest.java
com/sun/jdi/ReferrersTest.java

I then fixed the error check and these two tests started passing, but there is still at least one nsk/jdi test that fails due to the IterateThroughHeap solution returning some dead objects (which it tries to avoid by first calling ForceGarbageCollection, but this is not guaranteed to collect all garbage).

My conclusion is that we should simply remove the USE_ITERATE_THROUGH_HEAP support. It clearly is not being used by anyone, or they would have filed bugs against it. Also, it is not covered in the debug agent documentation, and is only included in the help output for debug builds (although it is functionally supported in release builds).

Also the debugflags option should be removed. It is only used to support USE_ITERATE_THROUGH_HEAP. Removing it does introduce some risk that user scripts could start failing, but I think this is extremely unlikely given it was only ever used in support of USE_ITERATE_THROUGH_HEAP, and debugflags is also not in the docs and only in the help output for the debug builds.