-
Bug
-
Resolution: Fixed
-
P2
-
5.0
-
b54
-
generic
-
generic
1) I create a cascading agent
2) I stop the connector server in the subagent
3) I do a queryNames in the master agent
=> dead lock.
I suspect the root cause is contained in the 3 threads below - see attached
file for a complete thread stack dump.
What causes the dead lock seems to be that the cascading agent receives
a JMXConnectionNotification.FAILED notification, and tries to call
"removeNotificationListener" on the failed connector within the notification
handle.
[exec] "MainThread" prio=5 tid=0x000e7ef8 nid=0x9 in Object.wait() [f177e000..f177fc30]
[exec] at java.lang.Object.wait(Native Method)
[exec] - waiting on <0xf1fa0278> (a [I)
[exec] at java.lang.Object.wait(Object.java:429)
[exec] at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.restart(ClientCommunicatorAdmin.java:82)
[exec] - locked <0xf1fa0278> (a [I)
[exec] at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.gotIOException(ClientCommunicatorAdmin.java:34)
[exec] at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1410)
[exec] at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getMBeanInfo(RMIConnector.java:1030)
[exec] at com.sun.jdmk.remote.cascading.proxy.CascadingProxy.getMBeanInfo(CascadingProxy.java:357)
[exec] at com.sun.jmx.mbeanserver.DynamicMetaDataImpl.getMBeanInfo(DynamicMetaDataImpl.java:67)
[exec] at com.sun.jmx.mbeanserver.BaseMetaDataImpl.getMBeanClassName(BaseMetaDataImpl.java:157)
[exec] at com.sun.jmx.mbeanserver.MetaDataImpl.getMBeanClassName(MetaDataImpl.java:167)
[exec] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.filterListOfObjects(DefaultMBeanServerInterceptor.java:1493)
[exec] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.queryMBeansImpl(DefaultMBeanServerInterceptor.java:490)
[exec] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.queryNames(DefaultMBeanServerInterceptor.java:533)
[exec] at com.sun.jmx.mbeanserver.JmxMBeanServer.queryNames(JmxMBeanServer.java:600)
[exec] at CascadingServiceTest.listMBeans(CascadingServiceTest.java:81)
[exec] "Thread-34" daemon prio=5 tid=0x001abf58 nid=0x4b in Object.wait() [f057f000..f057fc30]
[exec] at java.lang.Object.wait(Native Method)
[exec] - waiting on <0xf1fa0278> (a [I)
[exec] at java.lang.Object.wait(Object.java:429)
[exec] at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.restart(ClientCommunicatorAdmin.java:82)
[exec] - locked <0xf1fa0278> (a [I)
[exec] at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.access$500(ClientCommunicatorAdmin.java:16)
[exec] at com.sun.jmx.remote.internal.ClientCommunicatorAdmin$Checker.run(ClientCommunicatorAdmin.java:173)
[exec] at java.lang.Thread.run(Thread.java:534)
[exec] "Thread-35" daemon prio=5 tid=0x001acb80 nid=0x4c in Object.wait() [eff7e000..eff7fc30]
[exec] at java.lang.Object.wait(Native Method)
[exec] - waiting on <0xf1fa0278> (a [I)
[exec] at java.lang.Object.wait(Object.java:429)
[exec] at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.restart(ClientCommunicatorAdmin.java:82)
[exec] - locked <0xf1fa0278> (a [I)
[exec] at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.gotIOException(ClientCommunicatorAdmin.java:34)
[exec] at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1410)
[exec] at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.removeNotificationListener(RMIConnector.java:1256)
[exec] at com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent.cleanup(ProxyCascadingAgent.java:1052)
[exec] - locked <0xf1f9f5a0> (a com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent)
[exec] at com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent.stop(ProxyCascadingAgent.java:598)
[exec] - locked <0xf1f9f5a0> (a com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent)
[exec] at com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent.handleJMXConnectionNotification(ProxyCascadingAgent.java:746)
[exec] - locked <0xf1f9f5a0> (a com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent)
[exec] at com.sun.jdmk.remote.cascading.CascadingAgent$ConnectionListener.handleNotification(CascadingAgent.java:212)
[exec] at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:221)
[exec] at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:184)
[exec] at javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:396)
[exec] at javax.management.remote.rmi.RMIConnector.access$1200(RMIConnector.java:119)
[exec] at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.doStart(RMIConnector.java:1554)
[exec] at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.restart(ClientCommunicatorAdmin.java:106)
[exec] at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.gotIOException(ClientCommunicatorAdmin.java:34)
[exec] at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1410)
[exec] at javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1290)
[exec] at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:420)
[exec] at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:318)
[exec] at java.lang.Thread.run(Thread.java:534)
2) I stop the connector server in the subagent
3) I do a queryNames in the master agent
=> dead lock.
I suspect the root cause is contained in the 3 threads below - see attached
file for a complete thread stack dump.
What causes the dead lock seems to be that the cascading agent receives
a JMXConnectionNotification.FAILED notification, and tries to call
"removeNotificationListener" on the failed connector within the notification
handle.
[exec] "MainThread" prio=5 tid=0x000e7ef8 nid=0x9 in Object.wait() [f177e000..f177fc30]
[exec] at java.lang.Object.wait(Native Method)
[exec] - waiting on <0xf1fa0278> (a [I)
[exec] at java.lang.Object.wait(Object.java:429)
[exec] at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.restart(ClientCommunicatorAdmin.java:82)
[exec] - locked <0xf1fa0278> (a [I)
[exec] at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.gotIOException(ClientCommunicatorAdmin.java:34)
[exec] at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1410)
[exec] at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getMBeanInfo(RMIConnector.java:1030)
[exec] at com.sun.jdmk.remote.cascading.proxy.CascadingProxy.getMBeanInfo(CascadingProxy.java:357)
[exec] at com.sun.jmx.mbeanserver.DynamicMetaDataImpl.getMBeanInfo(DynamicMetaDataImpl.java:67)
[exec] at com.sun.jmx.mbeanserver.BaseMetaDataImpl.getMBeanClassName(BaseMetaDataImpl.java:157)
[exec] at com.sun.jmx.mbeanserver.MetaDataImpl.getMBeanClassName(MetaDataImpl.java:167)
[exec] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.filterListOfObjects(DefaultMBeanServerInterceptor.java:1493)
[exec] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.queryMBeansImpl(DefaultMBeanServerInterceptor.java:490)
[exec] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.queryNames(DefaultMBeanServerInterceptor.java:533)
[exec] at com.sun.jmx.mbeanserver.JmxMBeanServer.queryNames(JmxMBeanServer.java:600)
[exec] at CascadingServiceTest.listMBeans(CascadingServiceTest.java:81)
[exec] "Thread-34" daemon prio=5 tid=0x001abf58 nid=0x4b in Object.wait() [f057f000..f057fc30]
[exec] at java.lang.Object.wait(Native Method)
[exec] - waiting on <0xf1fa0278> (a [I)
[exec] at java.lang.Object.wait(Object.java:429)
[exec] at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.restart(ClientCommunicatorAdmin.java:82)
[exec] - locked <0xf1fa0278> (a [I)
[exec] at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.access$500(ClientCommunicatorAdmin.java:16)
[exec] at com.sun.jmx.remote.internal.ClientCommunicatorAdmin$Checker.run(ClientCommunicatorAdmin.java:173)
[exec] at java.lang.Thread.run(Thread.java:534)
[exec] "Thread-35" daemon prio=5 tid=0x001acb80 nid=0x4c in Object.wait() [eff7e000..eff7fc30]
[exec] at java.lang.Object.wait(Native Method)
[exec] - waiting on <0xf1fa0278> (a [I)
[exec] at java.lang.Object.wait(Object.java:429)
[exec] at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.restart(ClientCommunicatorAdmin.java:82)
[exec] - locked <0xf1fa0278> (a [I)
[exec] at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.gotIOException(ClientCommunicatorAdmin.java:34)
[exec] at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1410)
[exec] at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.removeNotificationListener(RMIConnector.java:1256)
[exec] at com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent.cleanup(ProxyCascadingAgent.java:1052)
[exec] - locked <0xf1f9f5a0> (a com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent)
[exec] at com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent.stop(ProxyCascadingAgent.java:598)
[exec] - locked <0xf1f9f5a0> (a com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent)
[exec] at com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent.handleJMXConnectionNotification(ProxyCascadingAgent.java:746)
[exec] - locked <0xf1f9f5a0> (a com.sun.jdmk.remote.cascading.proxy.ProxyCascadingAgent)
[exec] at com.sun.jdmk.remote.cascading.CascadingAgent$ConnectionListener.handleNotification(CascadingAgent.java:212)
[exec] at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:221)
[exec] at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:184)
[exec] at javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:396)
[exec] at javax.management.remote.rmi.RMIConnector.access$1200(RMIConnector.java:119)
[exec] at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.doStart(RMIConnector.java:1554)
[exec] at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.restart(ClientCommunicatorAdmin.java:106)
[exec] at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.gotIOException(ClientCommunicatorAdmin.java:34)
[exec] at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1410)
[exec] at javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1290)
[exec] at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:420)
[exec] at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:318)
[exec] at java.lang.Thread.run(Thread.java:534)