I distilled the offending code in the socketInThread
class. I takes different number of threads for different servers to force
the Socket ctor to throw an exception. On our slowest server it took only
25+ threads without any delay at all (Thread.sleep(5000); line commented
out). On Netscape Enterprise Server 3.6 it took more than 360 threads, each
thread being put to sleep for 5 sec before terminating (to imitate the
processing delay). You should be able to reproduce this behaviour by
adjusting the number of created threads and their sleep time. The stack
trace in case of failure and the class code are presented below. Any help
in dealing with this problem will be greatly appreciated. Vlad.

java.net.NoRouteToHostException: Connection timed out
        at java.net.PlainSocketImpl.doConnect(Compiled Code)
        at java.net.PlainSocketImpl.connectToAddress(Compiled Code)
        at java.net.PlainSocketImpl.connect(Compiled Code)
        at java.net.Socket.<init>(Compiled Code)
        at java.net.Socket.<init>(Compiled Code)
        at socketInThread.run(Compiled Code)
        at java.lang.Thread.run(Compiled Code)

import java.io.*;
import java.net.*;
import java.util.*;
 
public class socketInThread implements Runnable
{
   public static void main(String argv[])
   {
      int numThreads = 1;
 
      if (argv.length > 1)
      {
         System.out.println ("Usage: java socketInThread [numThreads]");
      }
      else if (argv.length == 1)
      {
         try
         {
            numThreads = Integer.parseInt(argv[0]);
         }
         catch (Exception e)
         {
             e.printStackTrace();
         }
 
         if (numThreads < 1) numThreads = 1;
         System.out.println ("No. of Threads: " + numThreads);
         socketInThread t = new socketInThread(numThreads);
      }
      else
      {
         System.out.println ("No. of Threads: " + numThreads);
         socketInThread t = new socketInThread();
      }
   }
 
   public socketInThread()
   {
      this(1);
   }
 
   public socketInThread(int nofThreads)
   {
      try
      {
         for (int i = 0; i < nofThreads; ++i)
         {
             (new Thread(this)).start();
         }
      }
      catch (Exception e)
      {
         e.printStackTrace();
      }
   }
 
   public void run()
   {
      try
      {
         Socket socket = new Socket("inet_address", portNo);
// these values come from an INI file
         System.out.println("new Socket SUCCESS...");
         Thread.sleep(5000);
         socket.close();
         socket = null;
      }
      catch (Exception e)
      {
         System.out.println("Failed to create Socket");
         e.printStackTrace();
      }
   }
 
}




There are number of workaround that have been tried to be implemented but they are not satisfactory. JDK 1.1.8_09a


saida.memon@East 2000-04-28 customer emailed:
Farida,
My machine is 600 Mhz, I duplicated the problem on port 80 after only 364
sockets, I had not encountered even once 'file descriptors' error, and we're
in process of installation of the latest jdk. I will let you know the
results as soon as I run the tests.


saida.memon@East 2000-05-02

(1) The problem is pretty clear now. Once the Socket object is created, it
behaves as expected - blocks on busy server indefinitely or for the timeout
specified. However, the Socket ctor does not. We would expect it to keep
trying to connect to the server until the server times out with ETIMEDOUT
message. That does not happen. The ctor may try a few times, but after a
few seconds it times out independently of the server. This is easily
repeatable on a slow server or a server with a limited pending connections
queue, e.g. Netscape server that defaults to 128 pending connections. Here
I'm talking about the value of the backlog parameter to the listen(socketFD,
backlog) system call that specifies the upper limit on the size of the
queue. On my server I can easily force the Socket ctor to timeout by
reducing that value to the notorious default of 5. The behavior that we'd
like to see is that the Socket ctor accepted a timeout argument that could
have a default value. We should be able to communicate through this
argument the desired timeout value to the ctor.

(2) We've achieved some success with jdk1.1.8_10 (Solaris 2.5.1). By
increasing backlog to a big number (up to 2048) we were able to run to
completion up to 1000 threads in real life application. We had worse luck
with jdk1.2.2_5a on the same version of Solaris. The ctor timeouts returned
with a vengeance. Is it possible that we need to upgrade OS to be able to
use jdk1.2.2_5a successfully?

(3) To summarize, eventually we want a Socket ctor that will allow us to
specify the desired timeout. In the meantime we need to know how the Socket
ctor negotiates a connection to the server internally, so we could make an
intelligent decision on how to deal with possible multithreading problems in
our applications. I would be satisfied with a ctor pseudo code showing the
logical steps in creating a Socket object.



saida.memon@East 2000-05-02


Farida,
The engineers are correct and I have the book and I've read it. However, in
real world situation I have to deal with real world servers and real world
networks that I have very little control over, if any. That's why I need to
understand why the Socket ctor does not wait for the server to come up with
ETIMEDOUT. In my view the ctor's business is to keep trying to connect
until explicitly told to stop by the server or by the application developer.
This is obviously the case with the present implementation. It assumes too
much on its own. Any way, can somebody just look into the source code for
it and describe what it is doing, so we could make an intelligent work
around its less than acceptable behavior. As Sun does not provide the
source code, I obviously can't do it myself. Judging by the Java forum I'm
not the first developer trapped in this problem.

The result of showrev -p follows:

Patch: 104266-01 Obsoletes: Packages: SUNWcsu
Patch: 104613-01 Obsoletes: Packages: SUNWcsu
Patch: 103612-06 Obsoletes: 103615-04, 103654-01 Packages: SUNWcsu,
SUNWarc, SUNWnisu
Patch: 103663-02 Obsoletes: Packages: SUNWcsu, SUNWhea
Patch: 103600-14 Obsoletes: 103609-02 Packages: SUNWcsu, SUNWcsr
Patch: 103591-07 Obsoletes: Packages: SUNWcsu, SUNWcsr
Patch: 104317-01 Obsoletes: Packages: SUNWcsu
Patch: 104334-01 Obsoletes: Packages: SUNWcsu
Patch: 104692-01 Obsoletes: Packages: SUNWcsu
Patch: 103663-15 Obsoletes: 103683-01 Packages: SUNWcsu, SUNWcsr, SUNWhea
Patch: 103594-16 Obsoletes: 105520-01 Packages: SUNWcsu, SUNWcsr, SUNWnisu
Patch: 103612-47 Obsoletes: 103615-04, 103654-01 Packages: SUNWcsu,
SUNWcsr, SUNWarc, SUNWscpu, SUNWfns, SUNWnisu, SUNWsra, SUNWsutl
Patch: 103630-13 Obsoletes: Packages: SUNWcsu, SUNWcsr
Patch: 103582-18 Obsoletes: Packages: SUNWcsu, SUNWcsr
Patch: 103627-05 Obsoletes: 103606-02, 105069-01 Packages: SUNWcsu,
SUNWcsr, SUNWarc, SUNWbtool, SUNWhea, SUNWtoo, SUNWxcu4
Patch: 103640-24 Obsoletes: 103591-09, 103658-02, 103920-05, 103600-18,
103609-02 Packages: SUNWcsu, SUNWcsr, SUNWkvm, SUNWcar, SUNWhea
Patch: 103680-02 Obsoletes: Packages: SUNWcsu
Patch: 103696-04 Obsoletes: Packages: SUNWcsu, SUNWcsr
Patch: 103817-03 Obsoletes: Packages: SUNWcsu
Patch: 103934-08 Obsoletes: Packages: SUNWcsu, SUNWcsr, SUNWhea
Patch: 104220-03 Obsoletes: Packages: SUNWcsu
Patch: 104331-07 Obsoletes: Packages: SUNWcsu
Patch: 104433-09 Obsoletes: Packages: SUNWcsu, SUNWcsr
Patch: 104654-05 Obsoletes: Packages: SUNWcsu
Patch: 104708-15 Obsoletes: 104710-01, 103766-02 Packages: SUNWcsu,
SUNWhea, SUNWssadv, SUNWssaop
Patch: 104736-04 Obsoletes: Packages: SUNWcsu
Patch: 104776-02 Obsoletes: Packages: SUNWcsu, SUNWarc
Patch: 104968-02 Obsoletes: Packages: SUNWcsu
Patch: 103603-09 Obsoletes: Packages: SUNWcsu
Patch: 103622-12 Obsoletes: Packages: SUNWcsu, SUNWcsr, SUNWhea
Patch: 103690-09 Obsoletes: Packages: SUNWcsu
Patch: 103981-16 Obsoletes: Packages: SUNWcsu, SUNWcsr
Patch: 104650-02 Obsoletes: Packages: SUNWcsu
Patch: 104935-01 Obsoletes: Packages: SUNWcsu
Patch: 104958-01 Obsoletes: Packages: SUNWcsu
Patch: 103738-08 Obsoletes: Packages: SUNWcsu
Patch: 104956-04 Obsoletes: Packages: SUNWcsu
Patch: 104960-01 Obsoletes: Packages: SUNWcsu
Patch: 105050-01 Obsoletes: Packages: SUNWcsu, SUNWscpu
Patch: 105092-01 Obsoletes: Packages: SUNWcsu
Patch: 104463-03 Obsoletes: Packages: SUNWcsu, SUNWxcu4
Patch: 104490-05 Obsoletes: 103847-02 Packages: SUNWcsu
Patch: 104818-01 Obsoletes: Packages: SUNWcsu, SUNWnisu
Patch: 104820-01 Obsoletes: Packages: SUNWcsu
Patch: 104822-01 Obsoletes: Packages: SUNWcsu, SUNWarc
Patch: 105016-01 Obsoletes: Packages: SUNWcsu, SUNWarc
Patch: 103612-51 Obsoletes: 103615-04, 103654-01 Packages: SUNWcsu,
SUNWcsr, SUNWarc, SUNWscpu, SUNWfns, SUNWnisu, SUNWsra, SUNWsutl
Patch: 104255-01 Obsoletes: Packages: SUNWcsu
Patch: 104166-04 Obsoletes: Packages: SUNWcsu
Patch: 105675-03 Obsoletes: Packages: SUNWcsu
Patch: 106563-04 Obsoletes: 104433-09 Packages: SUNWcsu, SUNWcsr, SUNWarc,
SUNWhea
Patch: 103630-14 Obsoletes: Packages: SUNWcsu, SUNWcsr
Patch: 103582-20 Obsoletes: Packages: SUNWcsu, SUNWcsr
Patch: 103594-19 Obsoletes: 105520-01 Packages: SUNWcsu, SUNWcsr, SUNWnisu
Patch: 103603-11 Obsoletes: Packages: SUNWcsu
Patch: 103622-14 Obsoletes: Packages: SUNWcsu, SUNWcsr, SUNWhea
Patch: 103640-29 Obsoletes: 103591-09, 103658-02, 103920-05, 103600-18,
103609-02, 104317-01, 106623-01, 103612-51, 103615-04, 103654-01, 104447-01,
107239-01 Packages: SUNWcsu, SUNWcsr, SUNWkvm, SUNWcar, SUNWarc, SUNWscpu,
SUNWfns, SUNWhea, SUNWnisu, SUNWsra, SUNWsutl
Patch: 103627-11 Obsoletes: 103606-02, 105069-01 Packages: SUNWcsu,
SUNWcsr, SUNWarc, SUNWbtool, SUNWhea, SUNWtoo, SUNWxcu4
Patch: 103690-12 Obsoletes: Packages: SUNWcsu
Patch: 103696-05 Obsoletes: Packages: SUNWcsu, SUNWcsr
Patch: 103699-02 Obsoletes: Packages: SUNWcsu
Patch: 103934-09 Obsoletes: Packages: SUNWcsu, SUNWcsr, SUNWhea
Patch: 104266-02 Obsoletes: Packages: SUNWcsu
Patch: 104650-03 Obsoletes: Packages: SUNWcsu
Patch: 104708-16 Obsoletes: 104710-01, 103766-02 Packages: SUNWcsu,
SUNWhea, SUNWssadv, SUNWssaop
Patch: 103738-12 Obsoletes: Packages: SUNWcsu
Patch: 105784-02 Obsoletes: Packages: SUNWcsu, SUNWarc
Patch: 103640-32 Obsoletes: 103591-09, 103658-02, 103920-05, 103600-18,
103609-02, 104317-01, 106623-01, 103612-51, 103615-04, 103654-01, 104447-01,
107239-01, 104915-10, 104594-03 Packages: SUNWcsu, SUNWcsr, SUNWkvm,
SUNWcar, SUNWarc, SUNWscpu, SUNWfns, SUNWhea, SUNWnisu, SUNWsra, SUNWsutl
Patch: 103627-13 Obsoletes: 103606-02, 105069-01 Packages: SUNWcsu,
SUNWcsr, SUNWarc, SUNWbtool, SUNWhea, SUNWtoo, SUNWxcu4
Patch: 104893-01 Obsoletes: Packages: SUNWcsr
Patch: 103597-04 Obsoletes: Packages: SUNWcsr
Patch: 104283-04 Obsoletes: Packages: SUNWcsr
Patch: 105004-10 Obsoletes: 104604-04 Packages: SUNWcsr, SUNWcar, SUNWhea
Patch: 105299-01 Obsoletes: Packages: SUNWcsr
Patch: 105352-01 Obsoletes: Packages: SUNWcsr
Patch: 104560-05 Obsoletes: Packages: SUNWcsr
Patch: 104893-02 Obsoletes: Packages: SUNWcsr
Patch: 105004-11 Obsoletes: 104604-04 Packages: SUNWcsr, SUNWcar, SUNWhea
Patch: 105299-02 Obsoletes: Packages: SUNWcsr
Patch: 104628-05 Obsoletes: Packages: SUNWcsr
Patch: 104795-02 Obsoletes: Packages: SUNWkvm
Patch: 104595-08 Obsoletes: Packages: SUNWkvm
Patch: 104489-08 Obsoletes: Packages: SUNWtltk, SUNWtltkd
Patch: 103566-39 Obsoletes: 103508-01 Packages: SUNWxwfnt, SUNWxwplt,
SUNWxwopt, SUNWxwslb
Patch: 103566-40 Obsoletes: 103508-01 Packages: SUNWxwfnt, SUNWxwplt,
SUNWxwopt, SUNWxwslb
Patch: 103566-45 Obsoletes: 103508-01 Packages: SUNWxwfnt, SUNWxwplt,
SUNWxwopt, SUNWxwslb
Patch: 104338-02 Obsoletes: Packages: SUNWxwplt, SUNWxwslb
Patch: 104010-01 Obsoletes: Packages: SUNWvolu
Patch: 104841-03 Obsoletes: Packages: SUNWvolu
Patch: 106382-01 Obsoletes: Packages: SUNWvolu
Patch: 104841-05 Obsoletes: Packages: SUNWvolu
Patch: 103901-11 Obsoletes: Packages: SUNWolrte, SUNWolinc, SUNWolslb
Patch: 104533-04 Obsoletes: Packages: SUNWolrte, SUNWolslb
Patch: 106662-01 Obsoletes: Packages: SUNWolrte, SUNWolslb
Patch: 106663-01 Obsoletes: Packages: SUNWolrte
Patch: 103901-12 Obsoletes: Packages: SUNWolrte, SUNWolinc, SUNWolslb
Patch: 104533-05 Obsoletes: Packages: SUNWolrte, SUNWolslb
Patch: 104976-03 Obsoletes: Packages: SUNWoldcv, SUNWoldst
Patch: 104976-04 Obsoletes: Packages: SUNWoldcv, SUNWoldst
Patch: 103461-28 Obsoletes: Packages: SUNWmfrun
Patch: 103461-33 Obsoletes: Packages: SUNWmfrun
Patch: 103558-14 Obsoletes: Packages: SUNWadmap, SUNWadmc, SUNWsadml
Patch: 105701-02 Obsoletes: Packages: SUNWadmap
Patch: 103558-15 Obsoletes: Packages: SUNWadmap, SUNWadmc, SUNWsadml
Patch: 106224-01 Obsoletes: Packages: SUNWoldst
Patch: 104093-07 Obsoletes: Packages: SUNWoldst
Patch: 104918-01 Obsoletes: Packages: SUNWoldst
Patch: 104977-01 Obsoletes: Packages: SUNWoldst
Patch: 106529-05 Obsoletes: 101242-13 Packages: SUNWlibC
Patch: 103879-04 Obsoletes: Packages: SUNWkcsrt
Patch: 103879-05 Obsoletes: Packages: SUNWkcsrt, SUNWkcspg
Patch: 104854-02 Obsoletes: Packages: SUNWdoc
Patch: iss_man_dhcp_sparc-01 Obsoletes: Packages: SUNWman
Patch: 103948-02 Obsoletes: Packages: SUNWaccu
Patch: 104816-01 Obsoletes: Packages: SUNWaccu
Patch: 107524-01 Obsoletes: Packages: SUNWaccu
Patch: 104516-03 Obsoletes: Packages: SUNWapppu
Patch: 103959-08 Obsoletes: Packages: SUNWscpu, SUNWlpr, SUNWlpu, SUNWlps
Patch: 103959-10 Obsoletes: Packages: SUNWscpu, SUNWlpr, SUNWlpu, SUNWlps
Patch: 103866-05 Obsoletes: Packages: SUNWbcp
Patch: 104873-04 Obsoletes: 103911-01 Packages: SUNWbnuu
Patch: 104824-01 Obsoletes: Packages: SUNWfac
Patch: 103743-01 Obsoletes: Packages: SUNWfns
Patch: 104605-08 Obsoletes: Packages: SUNWhea
Patch: 104212-13 Obsoletes: Packages: SUNWhmdu, SUNWhmd
Patch: 104246-08 Obsoletes: Packages: SUNWhmdu, SUNWhmd
Patch: 103686-02 Obsoletes: Packages: SUNWnisu
Patch: 105165-02 Obsoletes: Packages: SUNWnisu
Patch: 104475-02 Obsoletes: Packages: SUNWnisu
Patch: 105344-01 Obsoletes: Packages: SUNWtoo
Patch: 103900-01 Obsoletes: Packages: SUNWowbcp
Patch: 105251-01 Obsoletes: Packages: SUNWowbcp
Patch: 104995-01 Obsoletes: Packages: SUNWolimt
Patch: 105790-10 Obsoletes: 103796-19 Packages: SUNWffb, SUNWffbcf,
SUNWffbxg, SUNWffbw
Patch: 105790-22 Obsoletes: 103796-19 Packages: SUNWffb, SUNWffbcf,
SUNWffbxg, SUNWffbw
Patch: 105789-03 Obsoletes: 103796-19 Packages: SUNWvlxil
Patch: 105789-08 Obsoletes: 103796-19 Packages: SUNWvlxil
Patch: 103801-07 Obsoletes: Packages: SUNWsprot, SUNWxcu4t