Some NIO operations need to use a temporary direct buffer (or buffers). Those buffers are cached in a ThreadLocal to avoid repeated calls to malloc/free. When such a buffer is replaced in the cache (typically with a larger one), the old buffer is explicitly reclaimed. Unfortunately, when a thread exits, any buffers that are still cached are not explicitly reclaimed. We instead have to wait for the GC to find them unreachable so that they are reclaimed when their cleaners are called.

The above is not a huge issue in most cases. However, we've had an issue with one of our services that does churn through Threads (which live long enough for their Thread objects + cached buffers to be moved to the old generation) and also does CMS cycles very, very infrequently (like once every few days). As a result, the old gen grows (slowly) with unreachable direct buffers that are holding on to unused direct memory which cause a nasty native memory leak.

It would be helpful to explicitly reclaim any direct buffers in a thread's cache when a thread exists. This introduces an interesting challenge in that the thread has to notify NIO that it's exiting so that NIO can reclaim any cached buffers. I propose to do that by introducing exit hooks to ThreadLocals. This will of course open a huge can of worms. :-) But, we can structure the initial version so that the exit hooks can only be registered within java.base to avoid having to update the ThreadLocal public API.

I'll post a patch with this change to get feedback on it.

(More on this topic in JDK-4469299)