Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8316580

HttpClient with StructuredTaskScope does not close when a task fails

XMLWordPrintable

    • b18
    • x86_64
    • os_x
    • Verified

      ADDITIONAL SYSTEM INFORMATION :
      MacBook Pro 2019 - 2,4 GHz 8-Core Intel Core i9
      macOs Ventura 13.5.2 (22G91) - latest
      JDK 21 x64 from https://www.oracle.com/java/technologies/downloads/#jdk21-mac - latest GA 20.9.2023
      OpenJDK 21 build 35 from https://jdk.java.net/21/

      A DESCRIPTION OF THE PROBLEM :
      I created a simple Http GET performance testing tool using HttpClient with StructuredTaskScope.

      When the tool sends over 65 concurrent requests without delay to a localhost port which does not have anything listening, HttpClient is not able to release locks and the program does not exit. With low amount of requests this happens randomly and increasing request count makes the issue happen always. If there is something responding from the port, the tool exists normally.

      If I check the thread dump, the main thread is stuck on
      jdk.internal.net.http.HttpClientFacade.close

      "main" #1 [10243] prio=5 os_prio=31 cpu=401.70ms elapsed=4.47s tid=0x00007fdde1809c00 nid=10243 in Object.wait() [0x000070000eec7000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait0(java.base@21/Native Method)
      - waiting on <0x000000043f656430> (a jdk.internal.net.http.HttpClientImpl$SelectorManager)
      at java.lang.Object.wait(java.base@21/Object.java:366)
      at java.lang.Thread.join(java.base@21/Thread.java:2072)
      - locked <0x000000043f656430> (a jdk.internal.net.http.HttpClientImpl$SelectorManager)
      at java.lang.Thread.join(java.base@21/Thread.java:2200)
      at jdk.internal.net.http.HttpClientImpl.awaitTermination(java.net.http@21/HttpClientImpl.java:628)
      at java.net.http.HttpClient.close(java.net.http@21/HttpClient.java:900)
      at jdk.internal.net.http.HttpClientFacade.close(java.net.http@21/HttpClientFacade.java:192)
      at test.examples.HttpGetBurstBugWithStructuredScope.runBurst(HttpGetBurstBugWithStructuredScope.java:24)
      at test.examples.HttpGetBurstBugWithStructuredScope.main(HttpGetBurstBugWithStructuredScope.java:15)

      If I use CountdownLatch to manage exiting, the program does not get stuck.

      STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
      Run the HttpGetBurstBugWithStructuredScope. Adjust requestCount to test for the threshold where error happens or does not happen. Ensure that using an url with a working server exists the tool normally.

      EXPECTED VERSUS ACTUAL BEHAVIOR :
      EXPECTED -
      HttpClient should be able to close normally.
      ACTUAL -
      HttpGetBurstBugWithStructuredScope does not exit. main thread shows HttpClientFacade.close() in thread dump

      ---------- BEGIN SOURCE ----------
      package test.examples;

      import java.net.URI;
      import java.net.http.HttpClient;
      import java.net.http.HttpRequest;
      import java.net.http.HttpResponse;
      import java.time.Duration;
      import java.util.concurrent.ExecutionException;
      import java.util.concurrent.Executors;
      import java.util.concurrent.StructuredTaskScope;

      public class HttpGetBurstBugWithStructuredScope {

          public static void main(String[] args) {
              new HttpGetBurstBugWithStructuredScope().runBurst(
                      "http://localhost:62057/greet",
                      200
              );
          }

          void runBurst(String url, int reqCount) {
              final var dest = URI.create(url);
              try (final var virtualThreadExecutor = Executors.newVirtualThreadPerTaskExecutor()) {
                  try (final var httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(1)).executor(virtualThreadExecutor).build()) {
                      try (final var scope = new StructuredTaskScope.ShutdownOnFailure()) {
                          launchAndProcessRequests(scope, httpClient, reqCount, dest);
                      } finally {
                          System.out.println("StructuredTaskScope closed");
                      }
                  } finally {
                      System.out.println("HttpClient closed");
                  }
              } finally {
                  System.out.println("ThreadExecutor closed");
              }
          }

          private static void launchAndProcessRequests(
                  StructuredTaskScope.ShutdownOnFailure scope,
                  HttpClient httpClient,
                  int reqCount,
                  URI dest) {
              for (int counter = 0; counter < reqCount; counter++) {
                  scope.fork(() ->
                          getUrlAndAssert200(httpClient, dest)
                  );
              }
              try {
                  scope.join();
              } catch (InterruptedException e) {
                  throw new RuntimeException("scope.join() was interrupted", e);
              }
              try {
                  scope.throwIfFailed();
              } catch (ExecutionException e) {
                  throw new RuntimeException("something threw an exception in StructuredTaskScope", e);
              }
          }

          private static String getUrlAndAssert200(HttpClient httpClient, URI url) {
              final var response = executeRequest(httpClient, url);
              String res = response.body();
              int statusCode = response.statusCode();
              if (statusCode != 200) {
                  throw new RuntimeException(url.toString() + " returned status " + statusCode);
              }
              return res;
          }

          private static HttpResponse<String> executeRequest(HttpClient httpClient, URI url) {
              try {
                  var request = HttpRequest.newBuilder(url).GET().build();
                  return httpClient.send(request, HttpResponse.BodyHandlers.ofString());
              } catch (InterruptedException e) {
                  throw new RuntimeException(e);
              } catch (Exception e) {
                  throw new RuntimeException(e);
              }
          }
      }

      ---------- END SOURCE ----------

      CUSTOMER SUBMITTED WORKAROUND :
      package test.examples;

      import java.net.URI;
      import java.net.http.HttpClient;
      import java.net.http.HttpRequest;
      import java.net.http.HttpResponse;
      import java.time.Duration;
      import java.util.concurrent.CountDownLatch;
      import java.util.concurrent.ExecutionException;
      import java.util.concurrent.ExecutorService;
      import java.util.concurrent.Executors;

      public class HttpGetBurstBug {

          public static void main(String[] args) {
              new HttpGetBurstBug().runBurst(
                      "http://localhost:62057/greet",
                      2000
              );
          }

          void runBurst(String url, int reqCount) {
              final var dest = URI.create(url);
              try {
                  try (final var virtualThreadExecutor = Executors.newVirtualThreadPerTaskExecutor()) {
                      try (final var httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(1)).executor(virtualThreadExecutor).build()) {
                          launchAndProcessRequests(virtualThreadExecutor, httpClient, reqCount, dest);
                      }
                  } finally {
                      System.out.println("HttpClient closed");
                  }
              } finally {
                  System.out.println("Successfully closed");
              }
          }

          private static void launchAndProcessRequests(
                  ExecutorService virtualThreadExecutor,
                  HttpClient httpClient,
                  int reqCount,
                  URI dest) {
              final var latch = new CountDownLatch(reqCount);
              for (int counter = 0; counter < reqCount; counter++) {
                  virtualThreadExecutor.execute(() -> {
                              try {
                                  getUrlAndAssert200(httpClient, dest);
                              } catch (Exception e) {
                                  System.err.println("http get failed: " + e.getMessage());
                              } finally {
                                  latch.countDown();
                              }
                          }
                  );
              }
              try {
                  latch.await();
                  System.out.println("latch.await() completed");
              } catch (InterruptedException e) {
                  throw new RuntimeException("latch.await() was interrupted", e);
              }
          }

          private static String getUrlAndAssert200(HttpClient httpClient, URI url) {
              final var response = executeRequest(httpClient, url);
              String res = response.body();
              int statusCode = response.statusCode();
              if (statusCode != 200) {
                  throw new RuntimeException(url.toString() + " returned status " + statusCode);
              }
              return res;
          }

          private static HttpResponse<String> executeRequest(HttpClient httpClient, URI url) {
              try {
                  var request = HttpRequest.newBuilder(url).GET().build();
                  return httpClient.send(request, HttpResponse.BodyHandlers.ofString());
              } catch (InterruptedException e) {
                  throw new RuntimeException(e);
              } catch (Exception e) {
                  throw new RuntimeException(e);
              }
          }
      }


      FREQUENCY : always


            dfuchs Daniel Fuchs
            webbuggrp Webbug Group
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: