The JDWP spec mentions nothing about DisableCollection and EnableCollection tracking the depth or nesting of the commands. This means that EnableCollection should enable collection no matter how many DisableCollection commands were done before it. This is how the debug agent used to work before JDK-8255987. Now the commands' nesting level is tracked, meaning that if there are two DisableCollection commands, you need two EnableCollection commands to re-enable collection.

Another issue is that support for DisableCollection/EnableCollection nesting was intermixed with the support for having VM.Suspend disable collection on all objects (which was the main purpose of JDK-8255987). As a result, if you do a VM.Suspend and then do a DisableCollection on an ObjectReference, that object can now be collected, even though the spec says it should not be during a VM.Suspend.

To fix both of these issues, node->strongCount should go back to being a boolean (node->isStrong). Also, separate flags should be maintained to indicate if the reference is strong due to a DisableCollection and/or due to VM.Suspend. We need a flag for each, because it's possible that both can be true at the same time. When node->isStrong is true, if there is an EnableCollection it only gets set false if there is no current VM.Suspend. Likewise was if VM.Resume is done, it only gets set false if there is no outstanding DisableCollection.

There should be no need for maintaining a count anymore since we aren't suppose to for DisableCollection/EnableCollection, and there is no need to for VM.Suspend/Resume, since it only calls commonRef_pinAll() code when the suspendAllCount is changed to/from 0.

Please also note JDK-8269232, which is also introduced by this nesting level counting, but is being fixed in more direct manner to keep the changes simple. However, any fix for this CR will end up replacing the changes for JDK-8269232.

The fix for this issue will also fix JDK-8258071.