I found few issue investigating RSS memory increase when UseDynamicNumberOfCompilerThreads is used (default currently).

One is small idle time for C2 and C1 (100 and 500 ms) which leads to cycles of remove/add compiler threads (example from TestOverloadCompileQueues.java test):

Added initial compiler thread C2 CompilerThread0
Added initial compiler thread C1 CompilerThread0
Added compiler thread C1 CompilerThread1 (available memory: 13537MB, available profiled code cache: 115MB)
Added compiler thread C1 CompilerThread2 (available memory: 13537MB, available profiled code cache: 115MB)
Added compiler thread C1 CompilerThread3 (available memory: 13537MB, available profiled code cache: 115MB)
Added compiler thread C1 CompilerThread4 (available memory: 13537MB, available profiled code cache: 115MB)
Added compiler thread C1 CompilerThread5 (available memory: 13537MB, available profiled code cache: 115MB)
Added compiler thread C2 CompilerThread1 (available memory: 14222MB, available non-profiled code cache: 115MB)
Removing compiler thread C2 CompilerThread1 after 137 ms idle time
Removing compiler thread C1 CompilerThread5 after 515 ms idle time
Removing compiler thread C1 CompilerThread4 after 517 ms idle time
Added compiler thread C2 CompilerThread1 (available memory: 14221MB, available non-profiled code cache: 115MB)
Added compiler thread C2 CompilerThread2 (available memory: 14221MB, available non-profiled code cache: 115MB)
Added compiler thread C2 CompilerThread3 (available memory: 14221MB, available non-profiled code cache: 115MB)
Removing compiler thread C2 CompilerThread3 after 121 ms idle time
Added compiler thread C2 CompilerThread3 (available memory: 14187MB, available non-profiled code cache: 115MB)

The second is that we allocate only 2 compiler threads during VM initialization and add more later during compilation requests as you see in the example above.

Both these lead to memory fragmentation and increase in RSS.

I suggest to increase number of threads creating during VM initialization.
Switch off UseDynamicNumberOfCompilerThreads if CICompilerCount is small (4?).
Keep ratio 1 C1 vs 2 C2 threads when we allocate them (the example shows that we allocate too many C1 threads).
Increase idle times before we remove threads.
We need to experiment to get the best number and time.