JMX memory leak

This is logged by Oracle Weblogic team
This is tracked in BugDB
BUG 9502049 - STRESS BPM - MEMORY LEAK DETECTED.
BUG 9527439 - STRESS MEMORY LEAK IN SERVERNOTIFFORWARDER

repro.jar repro test case attached

The WL developer who reported this issue had been dialoging with
###@###.###
and the analysis from the above BUGDB bugs follows
*****************************************************
Email to Eamonn
---------------

Eamonn,

I believe we are running into a memory leak in a long running JMX connection.
I am wondering if it is a known issue. The details are as follows

- WLS has an admin server and managed servers. We have implemented a
federated model where the admin server delegates requests to the managed
servers. The admin server has a long running JMX connection to each managed
server.

- a user adds a notification listener on the adminserver side of the
connection (client side)

- ArrayNotificationBuffer registers a listener for every object in the
MBeanServer on the managed server

       for (ObjectName name : names)
           addBufferListener(name);

- ArrayNotificationBuffer registers a listener to be notified about new
MBeans added to the managed server

          addNotificationListener(MBeanServerDelegate.DELEGATE_NAME,
                                   creationListener, creationFilter, null);


- ServerNotifForwarder keeps a listenerMap of notification listeners (server
side)

          listenerMap.put(nn, set);

- it does not appear that these listeners entries are removed unless the
connection is closed. The ArrayNotificationBuffer does not remove them, it
does not appear that they are removed if a MBean is unregistered, and the
code prevents them from being removed if the MBean is not registered

      if (name != null && !name.isPattern()) {
           if (!mbeanServer.isRegistered(name)) {
               throw new InstanceNotFoundException("The MBean " + name +
                   " is not registered.");
           }
       }

So it appears if you have a number of short lived MBeans, then the
listenerMap will continue to grow until the connection is closed.

My questions are:

- do you agree this is an issue with the JMX implementation? Is this a known
issue?

- is there anything we can do to workaround the issue? [Somehow cycling the
connections is a fairly big change to make and we are at the very end of our
release cycle.]

- is there a mechanism for patching the JMX implementation?

If you are not the right person, please let me know who to contact.

Thanks,

Peter


*** PBOWER 03/26/10 08:47 am ***
Response from Éamonn

Hi Peter,

Yes, I am still the right person to contact with technical questions about
JMX. The former JMX team is no longer working on it full time but we are
still available to help with urgent problems.

I believe the behaviour you describe is deliberate. As I recall, our
reasoning was that clients do not need to have an actual network connection
open to be valid. For example, the connector could be using an datagram
protocol or the like. Such a client should nevertheless receive
notifications, and since it cannot distinguish between, on the one hand, an
MBean that was unregistered and then re-registered between two of its
transient connections, and on the other, an MBean that remained there the
whole time, it would not know in the former case that it needed to re-add its
listeners to the new MBean. The downside is the behaviour you observe when
many MBeans come and go, and the client (typically a federating client) adds
a listener automatically to each of them.

The code you cite, that tests whether the MBean is registered before removing
the listenerMap entry, does not seem correct to me. I think it should throw
InstanceNotFoundException only if there is no entry in the listenerMap. It
would choose between InstanceNotFoundException and ListenerNotFoundException
based on whether the MBean exists. That would mean that you could remove a
listener "ghost" from an MBean that has already been unregistered. It is a
slight deformation of JMX semantics but does not seem unreasonable.

Unfortunately, since the JMX API is part of the JDK, the only way to patch it
is as part of a JDK update release. We could set the wheels in motion for
this if you think it would help, but it would be months before you would see
the patch, and, even then, only customers who had updated to the very latest
JDK 6 would be able to take advantage of it.

If the problem is very critical, then the best workaround I can think of
would be to abuse reflection to stick your hand into the motor and tweak
things while it's running. :-/ I'm thinking you could have a thread that
accesses ServerNotifForwarder and thence listenerMap directly (via
Field.setAccessible), and that periodically purges listenerMap of entries for
MBeans that no longer exist. It's very nasty, and it requires your code to be
privileged, but I think it would work. (I would suggest failing safe, though,
so if you don't find ServerNotifForwarder or listenerMap on whatever JRE you
are running on, you just ignore the problem.) Alternatively, as you suggest,
you could cycle your connections periodically.

Both delegation and notifications are much improved in JMX 2.0, and you would
have a much better solution there, but unfortunately that has been deferred
from JDK 7 so you won't be able to use it any time soon.
Regards,

@ Éamonn McManus · JMX Spec Lead · http://weblogs.java.net/blog/emcmanus
***************************************************