deadlock in the orb

This is only a referal to a bug described in 4752117

Customer Problem Description:
------------------------------

During regression tests, Nortel is seeing the following exceptions
during 195th iteration of a client process connecting to ORBD.

We are testing the one single client connecting to the server.The test
is just login/logout to the server from client. Every time when a
client login to the server, it queries the orb to get a remote object,
after it gets the remote object, it release the Context object it
created, so I don't think it holds the connection very long. But our
test shows, it stopped on iteration 195, and threw the following
exception. Note that, it runs fine till 195th iteration.


Iteration 195 of 1000:
Test exception occurred:
org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 201 completed: No
        at
com.sun.corba.se.internal.iiop.ConnectionTable.getConnection(ConnectionTable
.java:161)
        at
com.sun.corba.se.internal.iiop.ConnectionTable.getConnection(ConnectionTable
.java:65)
        at
com.sun.corba.se.internal.iiop.GIOPImpl.getConnection(GIOPImpl.java:67)
        at
com.sun.corba.se.internal.corba.ClientDelegate.createRequest(ClientDelegate.
java:652)
        at
com.sun.corba.se.internal.corba.ClientDelegate.createRequest(ClientDelegate.
java:594)
        at
com.sun.corba.se.internal.corba.ClientDelegate.request(ClientDelegate.java:8
86)
        at org.omg.CORBA.portable.ObjectImpl._request(ObjectImpl.java:431)
        at
org.omg.CosNaming._NamingContextStub.resolve(_NamingContextStub.java:242)
        at com.sun.jndi.cosnaming.CNCtx.callResolve(CNCtx.java:440)
        at com.sun.jndi.cosnaming.CNCtx.lookup(CNCtx.java:492)
        at com.sun.jndi.cosnaming.CNCtx.lookup(CNCtx.java:470)
        at javax.naming.InitialContext.lookup(InitialContext.java:347)
        at
com.nortel.cdma.gsf.service.ClientServiceManagerBase.connect(ClientServiceMa
nagerBase.java:935)
        at
com.nortel.cdma.design_only.service.session.SessionServiceClientTestApp.runT
est(SessionServiceClientTestApp.java:228)
        at
com.nortel.cdma.design_only.service.session.SessionServiceClientTestApp.runA
pplication(SessionServiceClientTestApp.java:129)
        at com.nortel.cdma.gsf.util.Application$1.run(Application.java:383)
        at java.lang.Thread.run(Thread.java:536)
SessionServiceClient: At least one test failed!
SessionServiceClient Terminated.
Client terminated.


We try the same testing three times so far, every time it dies on 195
connections. Here is our code on client side to do the connection.

=============================================================
    InetAddress clientHostIp = InetAddress.getLocalHost();
    Context namingContext = null;
    Object hostMaster = null;

    DebugLogger.logTrace("Connecting client (host="+
       clientHostIp+
       ",loginId="+loginId+
       ",password="+password+")...");

    namingContext =
      NamingServerUtilities.getNamingContext(serverHostname,serverPortId);

    DebugLogger.logTrace("Looking up
"+HostMasterOSSClientInterface.BOUND_NAME+
" using "+namingContext+"...");

    hostMaster = namingContext.lookup
      (HostMasterOSSClientInterface.BOUND_NAME);

    namingContext.close();

    DebugLogger.logTrace("Host Master found for "+serverHostname);

    connection = connect(loginId, password, hostMaster, clientHostIp);

============================================================================
===

We have increased the file descriptors and high water mark to 480,
but that doesn't seem to help.

orbd ... -Dcom.sun.CORBA.ORAHighWaterMark=480

> rlim_fd_max/D
rlim_fd_max:
rlim_fd_max: 4096
           
> rlim_fd_cur/D
rlim_fd_cur:
rlim_fd_cur: 1024