If the Map supplied to JMXConnectorServerFactory.newJMXConnectorServer contains an entry for jmx.remote.message.connection.server, and if jmxremote_optional.jar is in the classpath, then the factory will return a JMXMP connector server, even if the JMXServiceURL specifies another protocol. It doesn't matter what the type of the entry is, although if it is not a MessageConnectionServer then the connector will get a ClassCastException when start() is called.

The problem is that the JMXConnectorServerProvider for JMXMP does not test that the URL it is given has jmxmp as its protocol. This provider gets called for any URL, because of the magic provider entry in jmxremote_optional.jar. We never noticed this problem before, because if there is no jmx.remote.message.connection.server entry in the Map, the provider's attempt to construct a JMXMPConnectorServer fails with this exception:

java.net.MalformedURLException: Unknown protocol: rmi
at com.sun.jmx.remote.socket.SocketConnectionServer.<init>(SocketConnectionServer.java:51)
at javax.management.remote.jmxmp.JMXMPConnectorServer.completeEnv(JMXMPConnectorServer.java:169)
at javax.management.remote.jmxmp.JMXMPConnectorServer.<init>(JMXMPConnectorServer.java:159)
at com.sun.jmx.remote.protocol.jmxmp.ServerProvider.newJMXConnectorServer(ServerProvider.java:35)
at javax.management.remote.JMXConnectorServerFactory.getConnectorServerAsService(JMXConnectorServerFactory.java:204)
at javax.management.remote.JMXConnectorServerFactory.newJMXConnectorServer(JMXConnectorServerFactory.java:289)

ANY exception thrown during the construction of the JMXConnectorServer causes the provider logic to look for the next provider (this is itself rather questionable). So we end up with the correct provider for RMI in the usual case. But more or less by accident.