Corba application fails w/ org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 203 completed: No

Beginning with Java 1.5.0 customers report to run into CORBA exceptions of the
following kind:

Mar 30, 2007 1:12:12 PM com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl <init>
WARNING: "IOP00410201: (COMM_FAILURE) Connection failure: socketType: IIOP_CLEAR_TEXT; hostname: 47.142.120.39; port: 37220"
org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 201 completed: No
        at com.sun.corba.se.impl.logging.ORBUtilSystemException.connectFailure(ORBUtilSystemException.java:2172)
        at com.sun.corba.se.impl.logging.ORBUtilSystemException.connectFailure(ORBUtilSystemException.java:2193)
        at com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.<init>(SocketOrChannelConnectionImpl.java:205)
        at com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.<init>(SocketOrChannelConnectionImpl.java:218)
        at com.sun.corba.se.impl.transport.SocketOrChannelContactInfoImpl.createConnection(SocketOrChannelContactInfoImpl.java:101)
        at com.sun.corba.se.impl.protocol.CorbaClientRequestDispatcherImpl.beginRequest(CorbaClientRequestDispatcherImpl.java:152)
        at com.sun.corba.se.impl.protocol.CorbaClientDelegateImpl.request(CorbaClientDelegateImpl.java:118)
        at org.omg.CORBA.portable.ObjectImpl._request(ObjectImpl.java:431)
        at com.nortelnetworks.fwcomp_if.common.coseventcomm._PushConsumerStub.push(_PushConsumerStub.java:18)
        at com.nortelnetworks.fwcomp.server.corbabase.es.ProxyPushSupplierImpl.forwardEvent(ProxyPushSupplierImpl.java:231)
        at com.nortelnetworks.fwcomp.common.events.notification.EFQ.run(EFQ.java:181)
 Caused by: java.net.ConnectException: Connection timed out
        at sun.nio.ch.Net.connect(Native Method)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:464)
        at java.nio.channels.SocketChannel.open(SocketChannel.java:146)
        at com.sun.corba.se.impl.transport.DefaultSocketFactoryImpl.createSocket(DefaultSocketFactoryImpl.java:60)
        at com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.<init>(SocketOrChannelConnectionImpl.java:188)
        ... 8 more


 After some repetitions of the warning, the application sees the following error:

Mar 30 13:30:56
 org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 203 completed: No
 at com.sun.corba.se.impl.logging.ORBUtilSystemException.writeErrorSend(ORBUtilSystemException.java:2231)
 at com.sun.corba.se.impl.logging.ORBUtilSystemException.writeErrorSend(ORBUtilSystemException.java:2249)
 at com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.sendWithoutLock(SocketOrChannelConnectionImpl.java:1042)
 at com.sun.corba.se.impl.encoding.BufferManagerWriteStream.sendFragment(BufferManagerWriteStream.java:82)
 at com.sun.corba.se.impl.encoding.BufferManagerWriteStream.sendMessage(BufferManagerWriteStream.java:96)
 at com.sun.corba.se.impl.encoding.CDROutputObject.finishSendingMessage(CDROutputObject.java:144)
 at com.sun.corba.se.impl.protocol.CorbaMessageMediatorImpl.finishSendingRequest(CorbaMessageMediatorImpl.java:247)
 at com.sun.corba.se.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete1(CorbaClientRequestDispatcherImpl.java:342)
 at com.sun.corba.se.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete(CorbaClientRequestDispatcherImpl.java:323)
 at com.sun.corba.se.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:129)
 at org.omg.CORBA.portable.ObjectImpl._invoke(ObjectImpl.java:457)
 at com.nortelnetworks.fwmgmt_if.common.SnmpIf._AccessorStub.get(_AccessorStub.java:218)
 at com.nortelnetworks.fwmgmt.server.adaptation.SNMP.SNMPSession$1.sendRequest(SNMPSession.java:544)
 at com.nortelnetworks.fwmgmt.server.adaptation.SNMP.SNMPSession.sendRequest(SNMPSession.java:1269)
 at com.nortelnetworks.fwmgmt.server.adaptation.SNMP.SNMPSession.get(SNMPSession.java:550)
 at com.nortelnetworks.fwmgmt.server.adaptation.SNMP.SNMPSession.get(SNMPSession.java:410)
 at com.nortelnetworks.fwmgmt.server.adaptation.SNMP.SNMPAdaptor.get(SNMPAdaptor.java:206)
 at com.nortelnetworks.mg9kem.server.adaptation.base.Mg5000Adaptor.get(Mg5000Adaptor.java:1326)
 at com.nortelnetworks.fwmgmt.server.adaptation.SNMP.SNMPAdaptor.get(SNMPAdaptor.java:188)
 at com.nortelnetworks.mg9kem.server.adaptation.base.Mg5000Adaptor.get(Mg5000Adaptor.java:1311)
 at com.nortelnetworks.fwmgmt.server.adaptation.SNMP.SNMPAdaptor.get(SNMPAdaptor.java:167)
 at com.nortelnetworks.mg9kem.server.adaptation.base.Mg5000Adaptor.get(Mg5000Adaptor.java:1297)
 at com.nortelnetworks.mg9kem.server.adaptation.base.Mg5000NeAdaptor.getEntPhysicalEntry(Mg5000NeAdaptor.java:2348)
 at com.nortelnetworks.mg9kem.server.em.logicalequipment.Mg5kNe.getEntPhysicalEntry(Mg5kNe.java:4484)
 at com.nortelnetworks.mg9kem.server.em.cards.CardOverloadIface.auditData(CardOverloadIface.java:757)
 at com.nortelnetworks.mg9kem.server.em.physicalequipment.Mg5kIntelligentCard$CardAuditNoRetry.execute(Mg5kIntelligentCard.java:7135)
 at com.nortelnetworks.fwmgmt.server.dataaudit.task.AuditTask$PausableThread.run(AuditTask.java:177)
 Caused by: java.nio.channels.ClosedChannelException
 at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:125)
 at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:294)
 at com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.write(SocketOrChannelConnectionImpl.java:715)
 at com.sun.corba.se.impl.encoding.CDROutputObject.writeTo(CDROutputObject.java:174)
 at com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.sendWithoutLock(SocketOrChannelConnectionImpl.java:992)
 ... 24 more
 

 The application needs to be restarted in order to cure the problem.

 This behaviour of the application was never seen with 1.4.2.


 Customer found out the following:

 Class com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.java
 uses a different send method depending on type of Socket ( nio vs the
 old type)

 (i) Customer wrote two custom socket factories, one based on new IO,
     the other on the old socket mechanism (i.e without SocketChannels).
     Customer added a way to get a handle to all the client sockets that
     were created by these factories, so that they could be closed at
     will.
 (ii) Customer used each of these factories in their application using
     -Dcom.sun.CORBA.transport.ORBSocketFactoryClass property and then
     closed the sockets after some time. Customer found the following :
     (a) With nio sockets, the ORB keeps attempting to reuse the old
         (closed) socket ad infinitum and throws the COMM_FAILURE with
         minor code 203 - this is the behavior seen in their application.
     (b) With the old sockets, the COMM_FAILURE happens once after the
         socket is closed, and then a new one is created to replace it,
         and subsequent calls are successful.

 In JDK 1.5, the default socket factory is nio based, so this would
 explain why dropped/closed TCP connections have not given us any
 trouble in the past.

 Customer provided the following testcase:

 Please find attached the custom socket factory that customer used to
 demonstrate the behavior: NortelSocketFactory.java

 It can be used with any corba client (no third party software required).
 Customer ran their corba client with the following commandline switches
 (i)
   -Dcom.sun.CORBA.transport.ORBSocketFactoryClass=NortelSocketFactory
   -DUseNio=true
 In (i), the client never recovers.
 (ii)
   -Dcom.sun.CORBA.transport.ORBSocketFactoryClass=NortelSocketFactory
   -DUseNio=false
 In (ii) , the client recovers automatically.