RMI unmarshalling errors in ClientNotifForwarder cause silent failure

FULL PRODUCT VERSION :
java version "1.6.0_18"
Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
Java HotSpot(TM) 64-Bit Server VM (build 16.0-b13, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Linux machine 2.6.18-92.1.6.el5 #1 SMP Wed Jun 25 13:45:47 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux


A DESCRIPTION OF THE PROBLEM :
likely related to the symptoms of
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4972794

when a new client is started up and connects via jmx remote to a server on an older version of the code, the following is seen on console:

ClientNotifForwarder NotifFetcher-run
SEVERE: Failed to fetch notification, stopping thread. Error is: java.rmi.UnmarshalException:
[elided, more in "Error Messages" section]

The code version skew (mismatched enums) is the cause, however the defect is that the JMX remote client just stops working. the ClientNotifForwarder just stops running without any means of signaling the code relying on it.
No exception is thrown, so a Thread.UncaughtExceptionHandler won't pick this up (we had one set, and didn't see it trigger, which started this whole analysis.)

The connection isn't "lost" so a JMXConnector.addConnectionNotificationListener() listener won't get any JMXConnectionNotifications for jmx.remote.connection.closed, jmx.remote.connection.failed or even jmx.remote.connection.notifs.lost (because ClientNotifForwarder.NotifFetcher.fetchOneNotif() is not called on the IOException catch)

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
in directory a: start server on version1

$ cd a/ && java -cp . server server

in directory b: start client on version2

$ cd b/ && java -cp . server client

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
notification of error in the client, exception, something, anything.
ACTUAL -
$ java -cp . server client
 [ large error stack on console outside of my code's control ]
happy!


ERROR MESSAGES/STACK TRACES THAT OCCUR :
Mar 17, 2010 1:37:57 PM ClientNotifForwarder NotifFetcher-run
SEVERE: Failed to fetch notification, stopping thread. Error is: java.rmi.UnmarshalException: error unmarshalling return; nested exception is:
java.io.InvalidObjectException: enum constant CONSTANT1 does not exist in class server$ConfigKey
java.rmi.UnmarshalException: error unmarshalling return; nested exception is:
java.io.InvalidObjectException: enum constant CONSTANT1 does not exist in class server$ConfigKey
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:173)
at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl_Stub.fetchNotifications(Unknown Source)
at javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1306)
at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:554)
at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:437)
at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:418)
at com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:88)
Caused by: java.io.InvalidObjectException: enum constant CONSTANT1 does not exist in class server$ConfigKey
at java.io.ObjectInputStream.readEnum(ObjectInputStream.java:1704)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1667)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1323)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at sun.rmi.server.UnicastRef.unmarshalValue(UnicastRef.java:306)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:155)
... 7 more
Caused by: java.lang.IllegalArgumentException: No enum const class server$ConfigKey.CONSTANT1
at java.lang.Enum.valueOf(Enum.java:196)
at java.io.ObjectInputStream.readEnum(ObjectInputStream.java:1702)
... 25 more


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import java.lang.management.*;
import java.rmi.registry.*;
import javax.management.*;
import javax.management.remote.*;
/*
apply the following diff for version2:
39c39
< public enum ConfigKey { CONSTANT1, CONSTANT2; }
---
> public enum ConfigKey { CONSTANT3, CONSTANT2; }
54c54
< key = ConfigKey.CONSTANT1;
---
> key = ConfigKey.CONSTANT3;
*/
public class server {
    public static void main(String[] argv) throws Exception {
        int serverPort = 12345;
        JMXServiceURL serverUrl = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://localhost:"+serverPort+"/test");
        ObjectName name = new ObjectName("test","foo","bar");
        if ( "server".equalsIgnoreCase( argv[0] ) ) {
            LocateRegistry.createRegistry(serverPort);
            MBeanServer jmxServer = ManagementFactory.getPlatformMBeanServer();
            SteMBean bean = new Ste();
            jmxServer.registerMBean( bean, name );
            JMXConnectorServer jmxConnector = JMXConnectorServerFactory.newJMXConnectorServer(serverUrl, null, jmxServer);
            jmxConnector.start();
            System.err.println("listening on "+serverUrl);
        } else {
            JMXConnector jmxConnector = JMXConnectorFactory.connect(serverUrl);
            jmxConnector.addConnectionNotificationListener(new NotificationListener(){
                public void handleNotification(Notification notification,Object handback) {
                    System.err.println("no!"+notification);
                }
            }, null, null );
            MBeanServerConnection jmxServer = jmxConnector.getMBeanServerConnection();
            
            jmxServer.addNotificationListener( name, new NotificationListener(){
                    public void handleNotification(Notification notification,Object handback){
                        System.err.println("got:"+notification);
                    }}, null, null );

            jmxServer.invoke( name, "foo", new Object[]{}, new String[]{} );
            
            Thread.sleep(4000L);
            System.err.println("happy!");
        }
    }
    public enum ConfigKey { CONSTANT1, CONSTANT2; }
    public interface SteMBean {
        public void foo();
    }
    public static class Ste extends NotificationBroadcasterSupport implements SteMBean {
        private long count = 0;
        public void foo() {
            sendNotification( new TestNotification("test", this, count++) );
            System.err.println("foo");
        }
    }
    public static class TestNotification extends Notification {
        private ConfigKey key;
        public TestNotification(String type, Object source, long sequenceNumber) {
            super(type,source,sequenceNumber);
            key = ConfigKey.CONSTANT1;
        }
    }
}
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
i have combed the openjdk source (this is still present in jdk7) and have been unable to find a point to hook on to get notification of the failure.

widening the catch at

http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/eae6e9ab2606/src/share/classes/com/sun/jmx/remote/internal/ClientNotifForwarder.java line 597 or 600, or calling fetchOneNotif() again at the end of the if (!shouldStop()) block at line 604 would at least notifiy the client that something was awry.