ADDITIONAL SYSTEM INFORMATION :
Using Corretto 11 distribution. Although, I suspect this is a bug in upstream OpenJDK.

A DESCRIPTION OF THE PROBLEM :
After updating from 11.0.16 to 11.0.17 (specifically 11.0.17.8.1), occasionally, the JVM will start using more and more system memory in an unbounded fashion, until eventually the system runs out of memory.

REGRESSION : Last worked in version 11

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
I don't really know. In one application, this happens pretty consistently on 11.0.17, a couple of other applications, i've seen this happen on a couple of times. However, I haven't been able to identify under what exact situation this happens.

On the application where this happens consistently downgrading to 11.0.16 fixed the memory leak.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The JVM should not leak memory.
ACTUAL -
The JVM leaks memory, and the amount of resident memory used by the JVM increases without bound even though a maximum heap size is set, and the application does not make use of JNI, or other obvious uses of off-heap memory.

CUSTOMER SUBMITTED WORKAROUND :
Downgrade to 11.0.16 or earlier.

FREQUENCY : occasionally