We have noticed increased deoptimizations and recompilations due to more frequent code cache flushing, during migration from JDK8 to JDK11. As a result, applications experienced latency and CPU usage regressions.

The aggressive flushing behavior is likely introduced in JDK-8046809. The new option -XX:StartAggressiveSweepingAt is not fully respected, because NMethodSweeper::notify() should use MAX2(start_threshold, 1.1) instead of MIN2.

This behavior can be reproduced with DaCapo's tradesoap benchmark, which could use ~20MB of code cache:
$ java -Xms2g -Xmx2g -XX:ReservedCodeCacheSize=40m -XX:StartAggressiveSweepingAt=<N> -Xlog:codecache+sweep*=debug -XX:+UnlockDiagnosticVMOptions -XX:+PrintMethodFlushingStatistics -jar dacapo-9.12-MR1-bach.jar tradesoap -n 5 -t 16

The log shows that code cache is flushed when the usage is far below (100 - StartAggressiveSweepingAt)%, which is 90% by default.

Counting lines with "CodeCache flushing" is the number of completed flushes (NMethodSweeper::sweep_code_cache()).
"Total number of full sweeps" at the end of the output is the number of attempted sweeps (NMethodSweeper::possibly_sweep()), i.e. number of times the sweeper thread is woken up.

Without the fix:
StartAggressiveSweepingAt | completed_flushes | attempted_sweeps
2 | 418 | 12768
10 (default) | 405 | 12907
50 | 419 | 11933

With the fix:
StartAggressiveSweepingAt | completed_flushes | attempted_sweeps
2 | 3 | 3
10 (default) | 6 | 6
50 | 315 | 315