G1: Limit the amount of remembered set scrubbing

During the cleanup operation, that is part of G1's concurrent marking algorithm, the RSets of all the regions in the heap are scrubbed. Basically entries coming from dead regions and dead objects are removed.

This has a positive effect on RSet scanning times during the following maixed GCs (since there are little or no extraneous entries an a regions RSet). The negative of this operation is that it can be very expensive (depending upon # of regions and the region size). For regions which are mostly full we incur the penalty of scrubbing the RSet without really reducing the number of entries by much.

This behavior was seen with a customer application:

2012-06-13T10:32:49.662-0700: 72269.812: [GC remark 72269.825: [GC ref-proc, 0.0008720 secs], 0.0980720 secs]
 [Times: user=0.37 sys=0.00, real=0.10 secs]
2012-06-13T10:32:49.779-0700: 72269.929: [GC cleanup 37679M->37655M(55296M), 1.0483700 secs]
 [Times: user=4.17 sys=0.01, real=1.05 secs]
2012-06-13T10:32:50.829-0700: 72270.978: [GC concurrent-cleanup-start]
2012-06-13T10:32:50.829-0700: 72270.978: [GC concurrent-cleanup-end, 0.0000620]

At the end of the cleanup pause few (if any) regions were being freed and the pause was taking a huge amount of time.

Instrumentation indicated that the almost all of the time of the cleanup pause was being taken up by RSet scrubbing.

It might be to reduce the amount of RSet scrubbing by not scrubbing (or at least reducing the amount) the RSets of regions that we are not going to collect during the next mixed GC phase. We would let the RSets of these regions grow until they will be collected during a mixed GC phase and then we will fully scrub their RSets.