There are a few non-resizable ResourceHashtables in heapShared.cpp and archiveHeapWriter.cpp. When a lot of heap objects are being written, these tables take a long time to look up.
The fix is to change these tables to ResizeableResourceHashtable (similar to what's being done in archiveBuilder.cpp, which writes the CDS metadata objects).
Example: test/hotspot/jtreg/runtime/cds/appcds/LotsOfClasses.java
Before: the test takes about 40 seconds to execute. Writing the CDS heap takes about 16.5 seconds:
[15.912s][info ][cds ] Heap range = [0x00000000e0c00000 - 0x0000000100000000]
...
[30.442s][info ][cds ] Size of heap region = 34771256 bytes, 721739 objects, 42058 roots, 27655 native ptrs
[32.438s][info ][cds ] oopmap = 5133128 ... 8692814 ( 59% ... 100% = 40%)
[32.438s][info ][cds ] ptrmap = 2176368 ... 2595628 ( 50% ... 59% = 9%)
After: the test takes 27 seconds to execute. Writing the CDS heap takes about 3.9 seconds:
[16.118s][info ][cds ] Heap range = [0x00000000e0c00000 - 0x0000000100000000]
...
[19.506s][info ][cds ] Size of heap region = 34771248 bytes, 721751 objects, 42056 roots, 27656 native ptrs
[20.084s][info ][cds ] oopmap = 5133214 ... 8692812 ( 59% ... 100% = 40%)
[20.084s][info ][cds ] ptrmap = 2176388 ... 2595671 ( 50% ... 59% = 9%)
The fix is to change these tables to ResizeableResourceHashtable (similar to what's being done in archiveBuilder.cpp, which writes the CDS metadata objects).
Example: test/hotspot/jtreg/runtime/cds/appcds/LotsOfClasses.java
Before: the test takes about 40 seconds to execute. Writing the CDS heap takes about 16.5 seconds:
[15.912s][info ][cds ] Heap range = [0x00000000e0c00000 - 0x0000000100000000]
...
[30.442s][info ][cds ] Size of heap region = 34771256 bytes, 721739 objects, 42058 roots, 27655 native ptrs
[32.438s][info ][cds ] oopmap = 5133128 ... 8692814 ( 59% ... 100% = 40%)
[32.438s][info ][cds ] ptrmap = 2176368 ... 2595628 ( 50% ... 59% = 9%)
After: the test takes 27 seconds to execute. Writing the CDS heap takes about 3.9 seconds:
[16.118s][info ][cds ] Heap range = [0x00000000e0c00000 - 0x0000000100000000]
...
[19.506s][info ][cds ] Size of heap region = 34771248 bytes, 721751 objects, 42056 roots, 27656 native ptrs
[20.084s][info ][cds ] oopmap = 5133214 ... 8692812 ( 59% ... 100% = 40%)
[20.084s][info ][cds ] ptrmap = 2176388 ... 2595671 ( 50% ... 59% = 9%)