Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8043766

CMM Testing: 8u40 Decommit auxiliary data structures



    • gc
    • b11



        Test checking that native memory is returning to the system.

        ---- Details from Thomas Schatzl -----

        Main testing should be in the area of
        a) not breaking existing functionality - the feature is completely
        transparent to the VM and
        b) memory usage reductions.

        What the feature does:
        - uncommits parts of the Java heap and auxiliary data according to the
        current sizing policy (e.g. HeapMin/MaxFreeRatio) at full GCs.

        Full GCs are the only point in time where the heap is decreased.

        The release (uncommit) of this data is based on regions. I.e. the unit
        of memory freeing is regions, when a region is freed, all its associated
        auxiliary data should go away as required.

        There is one complication in this matter, and that is that the unit of
        commit/uncommit is always OS memory pages, but an OS page worth of
        auxiliary data may cover multiple regions. If one of these regions is
        still not uncommitted, the auxiliary data cannot be freed either.

        This ratio between OS memory page size and region size is not fixed too,
        both OS memory page size and region size can vary depending on flags
        (UseLargePages etc).

        The change is G1 only too.

        So, what are the relevant memory consumers:
         - Java heap: split into disjoint regions, given G1RegionSize (range 1M
        to 32M, power of two increments); can be managed either with "OS page
        size" or large pages (-XX:+UseLargePages active).

         - Auxiliary data:
           contents of auxiliary data always map directly to a disjoint
        contiguous section of the Java heap. Their memory is *always* managed in
        native page size.

           - Two mark bitmaps, where each bit maps to an area of
        MinObjAlignmentInBytes * BitsPerByte bits.
        MinObjAlignmentInBytes can vary from 8 (default) to 256 iirc (actually I
        cannot find a bound, but it must be a power of two).

        This amounts to ~3% of heap space by default; if 4k pages are used (e.g.
        on x86), a single page spans 256k of heap (i.e. only a fraction of a
        region in any case) - but with increasing MinObjAlignmentInBytes a
        single page may span multiple regions.

          - Three "tables" (card table, offset table, hot card cache count)
        where in each element of them spans an area of 512 bytes. This is
        hardcoded at this time.

        The hot card cache count table is only used if it is enabled
        (G1ConcRSLogCacheSize is larger than zero).

        E.g. with 4k pages, each of them span 2 MB of heap.

        In total they make up 0.6% of the Java heap.

        Current status:
        - there is a prototype available (for creating tests) that manages the
        Java heap, the mark bitmaps, and one of the mentioned tables.

        So I think you can already start working on tests.

        Planned date that this feature to be finished and in the repos is 9th of
        June. It may be delayed slightly because we did not plan my vacation (I
        talked to Paul about this already) into that time.

        I will start getting reviews for the first changes next week.

        Difficulties I see here:
        - getting reliable information of actual uncommitted memory. I.e. it is
        not only important that the VM reports that the heap shrank, but actual
        memory usage (e.g. RSS size or whatever) adjusted correctly (i.e. that
        the commit/uncommit calls are made correctly).
        Also somehow the amount of memory committed/uncommitted what the VM
        thinks should probably be checked.

        - there is a fairly large amount of possible combinations of settings
        that should be checked.

        - getting the VM to reliably uncommit a particular area. Possibly best
        achievable using large object allocations.


          Issue Links



                azakharov Andrey Zakharov (Inactive)
                dfazunen Dmitry Fazunenko (Inactive)
                0 Vote for this issue
                5 Start watching this issue