FULL PRODUCT VERSION :
java version " 1.7.0_21 "
Java(TM) SE Runtime Environment (build 1.7.0_21-b12)
Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Mac OS X
Darwin Kernel Version 11.4.2: Thu Aug 23 16:25:48 PDT 2012; root:xnu-1699.32.7~1/RELEASE_X86_64 x86_64

A DESCRIPTION OF THE PROBLEM :
jconsole cannot connect to a JVM using the

service:jmx:rmi://host:port/jndi/rmi://host:port/jmxrmi

connection syntax if the server's JMX is configured to use SSL. The root exception jconsole displays is

java.rmi.ConnectIOException: non-JRMP server at remote endpoint

This happens even with the truststore location and password set on the java command line.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Compile the JMXClient and JMXServer classes (attached).

Launch the server using

java ... jmxserver.JMXServer portNumber

specifying a valid keystore and truststore using -Djavax.net.ssl.keyStore, -Djavax.net.ssl.keyStorePassword, -Djavax.net.ssl.trustStore and -Djavax.net.ssl.trustStorePassword.

Then launch the client using

java ... jmxclient.JMXClient portNumber

specifying a valid truststore.

The client connects successfully and immediate disconnects. The server reports the two operations.

But if you use jconsole instead of the attached client (and specify the truststore) jconsole fails to connect with the stack trace attached.

Using jvisualvm instead of jconsole works.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
jconsole would successfully connect to the secured server, as happens with the attached client and with jvisualvm.
ACTUAL -
jconsole throws the attached stack trace.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: non-JRMP server at remote endpoint]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:357)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
at sun.tools.jconsole.ProxyClient.tryConnect(ProxyClient.java:368)
at sun.tools.jconsole.ProxyClient.connect(ProxyClient.java:314)
at sun.tools.jconsole.VMPanel$2.run(VMPanel.java:295)
Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: non-JRMP server at remote endpoint]
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203)
at javax.naming.InitialContext.lookup(InitialContext.java:411)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1924)
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1891)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:274)
... 4 more
Caused by: java.rmi.ConnectIOException: non-JRMP server at remote endpoint
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:248)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:340)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:114)
... 9 more

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
// ==============================
// JMXClient
package jmxclient;

import java.io.IOException;
import java.net.MalformedURLException;
import java.util.HashMap;
import java.util.Map;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;
import javax.rmi.ssl.SslRMIClientSocketFactory;

public class JMXClient {

    /**
     * @param args the command line arguments
     * args: host [port [username [password]]]
     *
     * Defaults:
     * host: localhost
     * port: 8686
     * username: (none)
     * password: (none)
     */
    public static void main(String[] args) {
        try {
            new JMXClient().run(args);
        } catch (Exception ex) {
            ex.printStackTrace();
            System.exit(1);
        }
    }
    
    private void run(final String[] args) throws MalformedURLException, IOException {
        String username = null;
        String pw = null;
        String host = " localhost " ;
        String port = " 8686 " ;
        if (args.length > 1) {
            host = args[0];
            if (args.length > 2) {
                port = args[1];
                if (args.length > 3) {
                    username = args[2];
                    if (args.length > 4) {
                        pw = args[3];
                    }
                }
            }
        }
        final Map<String,Object> env = new HashMap<String,Object>();
        
        if (username != null) {
            final String[] credentials = new String[2];
            credentials[0] = username;
            credentials[1] = pw;
            env.put(JMXConnector.CREDENTIALS, credentials);
        }
        env.put( " com.sun.jndi.rmi.factory.socket " , new SslRMIClientSocketFactory());
        final JMXServiceURL jmxURL = new JMXServiceURL( " service:jmx:rmi:// " + host + " : " + port + " /jndi/rmi:// " + host + " : " + port + " /jmxrmi " );
        final JMXConnector jmxConnector = JMXConnectorFactory.connect(jmxURL, env);
        System.err.println( " Connected using " + jmxURL.toString());
        System.out.println( " Connection ID is " + jmxConnector.getConnectionId());
        jmxConnector.close();
    }
}
// =====================================================


//=====================================================
// JMXServer

package jmxserver;

import java.lang.management.ManagementFactory;
import java.rmi.registry.LocateRegistry;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;
import javax.management.MBeanServer;
import javax.management.Notification;
import javax.management.NotificationListener;
import javax.management.remote.JMXConnectorServer;
import javax.management.remote.JMXConnectorServerFactory;
import javax.management.remote.JMXServiceURL;
import javax.management.remote.rmi.RMIConnectorServer;
import javax.rmi.ssl.SslRMIClientSocketFactory;
import javax.rmi.ssl.SslRMIServerSocketFactory;

public class JMXServer {

    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) {
        try {
            new JMXServer().run(args);
        } catch (Exception ex) {
            ex.printStackTrace();
            System.exit(1);
        }
    }
    
    private void run(final String[] args) throws Exception {
        final Map<String,Object> env = new HashMap<String,Object>();
        final int port = Integer.valueOf(args[0]);
        final SslRMIClientSocketFactory csf = new SslRMIClientSocketFactory();
        final SslRMIServerSocketFactory ssf = new SslRMIServerSocketFactory();
        LocateRegistry.createRegistry(port, csf, ssf);
        env.put(RMIConnectorServer.RMI_SERVER_SOCKET_FACTORY_ATTRIBUTE, ssf);
        env.put(RMIConnectorServer.RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE, csf);
        env.put( " com.sun.jndi.rmi.factory.socket " , new SslRMIClientSocketFactory());
        JMXServiceURL jmxSrvUrl = new JMXServiceURL( " service:jmx:rmi://localhost: " + args[0] + " /jndi/rmi://localhost: " + args[0] + " /jmxrmi " );
        
        MBeanServer mbServer = ManagementFactory.getPlatformMBeanServer();
        JMXConnectorServer jmxServer = JMXConnectorServerFactory.newJMXConnectorServer(jmxSrvUrl, env, mbServer);
        jmxServer.addNotificationListener(new Listener(), null, null);
        jmxServer.start();
    }
    
    private class Listener implements NotificationListener {

        @Override
        public void handleNotification(Notification notification, Object handback) {
            System.err.println((new Date()).toString() + " " + notification.getSequenceNumber() + " . " + notification.toString());
        }
        
    }
}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Use jvisualvm with its MBean plug-in.