jcmd to trim the glibc heap

The glibc is somewhat notorious for retaining released C Heap memory: calling free(3) returns memory to the glibc, and most libc variants will return at least a portion of it back to the Operating System, but the glibc often does not.

This depends on the granularity of the allocations and a number of other factors, but we found that many small allocations in particular may cause the process heap segment (hence RSS) to get bloaty. This can cause the VM to not recover from C-heap usage spikes.

The glibc offers an API, "malloc_trim", which can be used to cause the glibc to return free'd memory back to the Operating System.

This may cost performance, however, and therefore I hesitate to call malloc_trim automatically. That may be an idea for another day.

Instead of an automatic trim I propose to add a jcmd which allows to manually trigger a libc heap trim. Such a command would have two purposes:

- when analyzing cases of high memory footprint, it allows to distinguish "real" footprint, e.g. leaks, from a cases where the glibc just holds on to memory
- as a stop gap measure it allows to release pressure from a high footprint scenario.
- Note that this command also helps with analyzing libc peaks which had nothing to do with the VM - e.g. peaks created by customer code which just happens to share the same process as the VM. Such memory does not even have to show up in NMT.

In a way, this would be similar to the GC.run jcmd.

I propose to introduce this command for Linux only. Other OSes (apart maybe AIX) do not seem to have this problem, but Linux is arguably important enough in itself to justify a Linux specific jcmd.