RMI - passing object by value results in Unmarshalling exception (JDK 1.3.1_01)

Customer Problem Description:
-------------------------------

java version "1.3.1_01"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.1_01)
Java HotSpot(TM) Client VM (build 1.3.1_01, mixed mode)
======================================================================
We are working on upgrading a java distributed application software which uses RMI
extensively. The upgrade is from jdk1.1.8 to jdk1.3.1_01. The server is on
Solaris platform and the client is developed using Swing.

The application has almost all the business objects designed as customized swing model objects, which will attach itself to various swing views in realizing the GUI.

java RMI is used as a distributed protocol in this application. The concept of passing objects by value is very much used for passing these business objects
back and forth between the server and the client. In production the application
runs on jre1.1.8.

With the upgraded source, I was able to logon (just passes system class objects such as String, HashTable etc.). But when I stared using the application (the business objects are passed), first it gave unmarshalling exception due to ClassNotFound. The client is on Windows2000 and the server is on solaris(SunOS 5.6). The classpath was set correctly. We have bundled all the classes pertaining to the application in a single jar file.

Client Side workaround:
-----------------------
As a workaround at client side(Windows 2000)I tried assigning the jar file to bootclasspath by usinthe non-standard option of the
jvm -Xbootclasspath/p:storeref.jar.
For all usecases where the objects move from server to the client, this workaround seem to work fine. But for save use-cases the objects move from client to the server,which results in Unmarshalling exception due to classnotfound exception.

I tried the same workarounds with the server also. It doesn't seem to work.

This bug is quiet sporadic, rarely it works and mostly it is not. It is not
consistent across the application.
Also sometimes the exception is 'Unmarshalling exception due to
Connection reseet by peer'. This also happens rarely.

BugId: 4349670 may be related but customer has the following comments
wrt this bug.

We have all our classes bundled in a single jar file. This jar file has
remote classes, GUI classes and RMI parameter classes. So we assign the jar
file to the classpath while running the client OR server( ie, java
-classpath storeref.jar .....). Since all the classes required are available
in the classpath, I presume there should not be any problem in loading
classes. The question of loading classes from remote codebase will not arise
at all. Everything is available in classpath.
In production (jdk1.1.8) it runs fine this way. Nevertheless, the option of
deploying the jar file in a http server is not a recommended option for us,
since it is cumbersome to sortout between local classes, RMI parameter
classes and Remote classes.
  I have tried reproducing the bug with a sample test rmi client/server.
Also, I tried the sample code given in bug id 4349670. Both are working
fine. So I think, it is very difficult to recreate the bug off from the
actual application.
  I have pasted the rmi trace below. Please let me know if this helps or
do you have any other way of debugging?
+++++++++++++++++++++++++++++++++++++++

RMI Trace:
---------
Here is the rmi trace taken with the option
-Djava.rmi.server.logCalls=TRUE.

Mon Nov 19 16:36:00 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
[10.15.171.65: acn.manager.ACNServer[0]: java.lang.Object validateField(java.lang.Class,java.lang.Object,java.lang.Object)]

Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
[10.15.171.65: acn.manager.ACNServer[0]: java.lang.Object validateField(java.lang.Class,java.lang.Object,java.lang.Object)]

====================================starts here================================

Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
[10.15.171.65: acn.manager.ACNServer[0]: java.lang.Object save(java.lang.Class,
java.lang.Object, java.lang.Object)]

Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
[10.15.171.65] exception: java.rmi.UnmarshalException: error unmarshalling
arguments;nested exception is:
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
java.lang.ClassNotFoundException: acn/business/WaitListener
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-
10.15.171.65:java.lang.ClassNotFoundException: acn/business/WaitListener

Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.lang.Class.forName0(Native Method)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.lang.Class.forName(Class.java:120)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at javax.swing.event.EventListenerList.readObject(EventListenerList.java:250)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.lang.reflect.Method.invoke(Native Method)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.invokeObjectReader(ObjectInputStream.java:2214)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.inputObject (ObjectInputStream.java:1411)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:386)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.inputClassFields(ObjectInputStream.java:2263)

Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:519)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.inputObject(ObjectInputStream.java:1412)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:386)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.inputArray(ObjectInputStream.java:1142)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:374)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.inputClassFields(ObjectInputStream.java:2263)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:519)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.inputObject(ObjectInputStream.java:1412)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:386)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.inputClassFields(ObjectInputStream.java:2263)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:519)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.inputObject(ObjectInputStream.java:1412)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:386)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.inputClassFields(ObjectInputStream.java:2263)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:519)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.inputObject(ObjectInputStream.java:1412)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:386)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:236)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.util.Hashtable.readObject(Hashtable.java:792)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.lang.reflect.Method.invoke(NativeMethod)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.invokeObjectReader(ObjectInputStream.java:2214)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.inputObject(ObjectInputStream.java:1411)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:386)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.inputClassFields(ObjectInputStream.java:2263)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:519)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.inputObject(ObjectInputStream.java:1412)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:386)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:236)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at acn.manager.ACNServer_Skel.dispatch(ACNServer_Skel.java:145)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at sun.rmi.server.UnicastServerRef.oldDispatch(UnicastServerRef.java:319)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:187)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at sun.rmi.transport.Transport$1.run(Transport.java:152)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.security.AccessController.doPrivileged(Native Method)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at sun.rmi.transport.Transport.serviceCall(Transport.java:148)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:465)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:706)
Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(3)-10.15.171.65:
at java.lang.Thread.run(Thread.java:484)

==================================ends here===================================

Mon Nov 19 16:36:01 CST 2001:RMI:RMI TCP Connection(4)-10.15.171.65:[10.15.171.6
5: acn.manager.ACNServer[0]: java.lang.Object load(java.lang.Class,
java.lang.Object, java.lang.Object)]
Mon Nov 19 16:38:37 CST 2001:RMI:RMI TCP Connection(5)-138.108.77.72:[138.108.77
.72: sun.rmi.transport.DGCImpl[0:0:0, 2]: java.rmi.dgc.Lease dirty(java.rmi.serv
er.ObjID[], long, java.rmi.dgc.Lease)]

Mon Nov 19 16:39:29 CST 2001:RMI:RMI TCP Connection(6)-10.15.171.65:[10.15.171.6
5: sun.rmi.transport.DGCImpl[0:0:0, 2]: java.rmi.dgc.Lease dirty(java.rmi.server
.ObjID[], long, java.rmi.dgc.Lease)]
+++++++++++++++++++++++++++++++++++++++

As I mentioned above, this bug is inconsistent. I
have attached here two stack traces. One, when it works
fine(normaltrace.txt) and other, when it is not working(exceptiontrace.txt).



I compared these two traces, I see that for the normal operation the trace
shows "sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:241)".
For exception, the corresponding trace is showing
"sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:187)". I
think the problem is in dispatch::UnicastServerRef. I think this is a
good starting point to fix this problem.