Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8037403

"SocketException: Invalid argument" under heavy concurrency when SO_TIMEOUT > 0

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Incomplete
    • Icon: P4 P4
    • None
    • 7u51
    • core-libs
    • x86
    • os_x

      FULL PRODUCT VERSION :
      java version "1.7.0_60-ea"
      Java(TM) SE Runtime Environment (build 1.7.0_60-ea-b07)
      Java HotSpot(TM) 64-Bit Server VM (build 24.60-b09, mixed mode)

      ADDITIONAL OS VERSION INFORMATION :
      Mac OS 10.9.2

      A DESCRIPTION OF THE PROBLEM :
      On the Mac, there seems to be an issue with Socket.setSoTimeout(...) that manifests under heavy load.

      When I open up 1000+ simultaneous connections with setSoTimeout(30000), many of them will fail almost immediately with the following stack trace:

      java.net.SocketException: Invalid argument
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:152)
      at java.net.SocketInputStream.read(SocketInputStream.java:122)
      at java.net.SocketInputStream.read(SocketInputStream.java:210)
      at loadster.core.utils.net.SocketTimeoutTest$Client.run(SocketTimeoutTest.java:73)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:744)

      However, when I do the exact same thing and setSoTimeout(0), all of the connections complete successfully every time.

      Note that even though the issue is related to setSoTimeout(0), the sockets are NOT timing out. They typically fail within milliseconds.

      Please see the attached test case to reproduce the issue.

      STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
      Run the attached test case.

      In short, here is what it does:

      1. Opens a local server socket and listens for incoming connections.
      2. Fires off 1250 client sockets concurrently that connect to the server socket.
      3. Each connection is handled in a separate thread by the server.
      4. When SO_TIMEOUT is set, many of the clients fail when they first try to read.

      EXPECTED VERSUS ACTUAL BEHAVIOR :
      EXPECTED -
      It's expected that all of the client sockets would connect and read successfully.
      ACTUAL -
      Many of the client sockets fail immediately with "SocketException: Invalid argument" even though they have not timed out.

      ERROR MESSAGES/STACK TRACES THAT OCCUR :
      java.net.SocketException: Invalid argument
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:152)
      at java.net.SocketInputStream.read(SocketInputStream.java:122)
      at java.net.SocketInputStream.read(SocketInputStream.java:210)
      at loadster.core.utils.net.SocketTimeoutTest$Client.run(SocketTimeoutTest.java:73)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:744)

      REPRODUCIBILITY :
      This bug can be reproduced always.

      ---------- BEGIN SOURCE ----------
      import java.io.InputStream;
      import java.io.OutputStream;
      import java.net.InetSocketAddress;
      import java.net.ServerSocket;
      import java.net.Socket;
      import java.util.concurrent.ExecutorService;
      import java.util.concurrent.Executors;
      import java.util.concurrent.TimeUnit;
      import java.util.concurrent.atomic.AtomicInteger;

      /**
       * Test to demonstrate "java.net.SocketException: Invalid argument" under heavy concurrency when SO_TIMEOUT is set.
       *
       * When SO_TIMEOUT=0 all connections are successful.
       * When SO_TIMEOUT=30000 (or anything but 0) many will fail almost immediately.
       *
       * This test fails in 1.7.0_60-ea and 1.7.0_51.
       */
      public class SocketTimeoutTest {

          //private static final int SO_TIMEOUT = 0; // THIS WILL WORK
          private static final int SO_TIMEOUT = 30000; // THIS WILL NOT WORK

          private static final int CONNECTIONS = 1250;

          private AtomicInteger successCount = new AtomicInteger(0);
          private AtomicInteger failureCount = new AtomicInteger(0);

          public static void main(String[] args) throws Exception {
              new SocketTimeoutTest().runLotsOfSockets();
          }

          public void runLotsOfSockets() throws Exception {
              ServerSocket serverSocket = new ServerSocket(8000, CONNECTIONS);
              ExecutorService clientPool = Executors.newFixedThreadPool(CONNECTIONS);

              // Fire off a bunch of clients
              for (int i = 0; i < CONNECTIONS; i++) {
                  clientPool.execute(new Client());
              }

              // Run the server and dispatch enough threads to handle incoming connections
              for (int i = 0; i < CONNECTIONS; i++) {
                  new Thread(new Server(serverSocket.accept())).start();
              }

              // Wait up to 10 seconds for all clients to finish
              clientPool.shutdown();
              clientPool.awaitTermination(10, TimeUnit.SECONDS);

              System.out.println("With SO_TIMEOUT=" + SO_TIMEOUT + ", success count is " + successCount + ", failure count is " + failureCount);

              serverSocket.close();
          }

          /**
           * Simple client thread that sends a single byte, reads a single byte response, then closes the connection.
           */
          public class Client implements Runnable {
              public void run() {
                  try {
                      Socket socket = new Socket();
                      socket.setSoTimeout(SO_TIMEOUT);
                      socket.connect(new InetSocketAddress("localhost", 8000));

                      InputStream input = socket.getInputStream();
                      OutputStream output = socket.getOutputStream();

                      try {
                          output.write("x".getBytes());
                          int b = input.read();

                          successCount.addAndGet(1);
                      } finally {
                          output.close();
                          input.close();
                          socket.close();
                      }
                  } catch (Exception e) {
                      e.printStackTrace();
                      failureCount.addAndGet(1);
                  }
              }
          }

          /**
           * Simple server thread that echoes a single byte, then closes the connection.
           */
          public class Server implements Runnable {
              private Socket socket;

              public Server(Socket socket) {
                  this.socket = socket;
              }

              public void run() {
                  try {
                      InputStream input = socket.getInputStream();
                      OutputStream output = socket.getOutputStream();

                      try {
                          output.write(input.read());
                      } finally {
                          input.close();
                          output.close();
                          socket.close();
                      }
                  } catch (Exception e) {
                      e.printStackTrace();
                  }
              }
          }
      }

      ---------- END SOURCE ----------

      CUSTOMER SUBMITTED WORKAROUND :
      Set SO_TIMEOUT to zero. Unfortunately this is not an option in many cases.

      It may be related but is NOT the same thing as this bug: https://bugs.openjdk.java.net/browse/JDK-8024710

            Unassigned Unassigned
            webbuggrp Webbug Group
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: