HotSpot leaking memory in long-running requests

FULL PRODUCT VERSION :
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

FULL OS VERSION :
Ubuntu 14.04
Linux andrew-VirtualBox 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

EXTRA RELEVANT SYSTEM CONFIGURATION :
Ubuntu 14.04 Guest running in VirtualBox on OSX 10.11.5 host.
No other changes to standard Ubuntu configuration.

A DESCRIPTION OF THE PROBLEM :
When running a web application for a long period of time, and that application makes a large number of long-runing requests (> 1s), it will begin to leak memory.

Looking at the application with JConsole, all the following pools are limited, and none are growing:

* Heap Space
* Meta Space
* Code Cache
* Compressed Class Space

As such, I'm not entirely sure where all the extra memory is going. Looking at the application with JCmd and NMT, I see that over time, the Class/Malloc area grows. That is, given the following partial output from jcmd <pid> VM.native_memory summary:

- Class (reserved=23075KB, committed=15651KB)
                            (classes #2182)
                            (malloc=547KB #6210)
                            (mmap: reserved=22528KB, committed=15104KB)

The "classes" number remains static, the "mmap" reserved & committed numbers remain static, but the "malloc" number grows over time, and does not decrease.

I have also observed that, after setting MALLOC_ARENAS_MAX to 2, when running on a system with glibc >= 2.16, that the malloc arenas seem to fragment over time, and that, after a couple of days, the amount of native heap space reported by pmap grows. For the sample application I have provided below, I have seen the *native* (*not* Java!) heap space grow to over 200MB used. This seems... incorrect.

THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: No

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes

REGRESSION. Last worked in version 7u80

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Check out the sample project from GitHub onto a machine with JRE 1.8.0_101.
cd into nacroleptic
Run "mvn clean package"
Run java -jar target/narcoleptic.jar
cd into ../devourer
Run "mvn clean package"
Run ./run.sh

Download Apache JMeter 3.0 from http://jmeter.apache.org/download_jmeter.cgi
Extract the archive and start with the "bin/jmeter.sh" script.

Open the file "load_test.jmx" with JMeter.

Start the JMeter tests with CTRL+R (CMD+R on OSX)

Leave for *at least* 30 minutes, then come back and observe the amount of memory used by the process. It will have increased beyond what would be expected. Leave for several hours, maybe a couple of days. Memory usage will continue to increase.

EXPECTED VERSUS ACTUAL BEHAVIOR :
Expected behaviour: Memory usage of the application remains stable.

Actual behaviour: Memory usage of the application increases over time, without stopping. It may slow, and *appear* to stop, but given a large enough time span, it will *alwasy* continue to increase.
ERROR MESSAGES/STACK TRACES THAT OCCUR :
N/A - application does not crash unless placed inside a cgroup (for example) with a hard memory limit.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
Please find the source code at: https://github.com/ipsi/memory-testing

This is the smallest test case I have been able to find, and I've been looking quite hard.
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Revert to Java 7
Disable HotSpot completely with -Xint
Set the MALLOC_ARENAS_MAX environment variable to 2, which slows the memory growth, but does not halt it.