Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8361091

HttpClient connection leak on massive concurrent connections with Http2

XMLWordPrintable

    • generic
    • generic

      ADDITIONAL SYSTEM INFORMATION :
      openjdk 21.0.7 2025-04-15 LTS
      OpenJDK Runtime Environment Corretto-21.0.7.6.1 (build 21.0.7+6-LTS)
      OpenJDK 64-Bit Server VM Corretto-21.0.7.6.1 (build 21.0.7+6-LTS, mixed mode, sharing)

      Reproduced on MacOS and Amazon Corretto

      ProductName: macOS
      ProductVersion: 15.4.1
      BuildVersion: 24E263

      A DESCRIPTION OF THE PROBLEM :
      When opening connections concurrently using the HttpClient, most of the connection objects are orphaned from the Http2Client but are kept alive by the *openedConnections* Set stored in the HttpClientImpl.

      This has the side-effect of open file descriptors repeatedly growing on the system, as the Http2Connection objects and their FileDescriptors are being kept alive.

      As per previous recommendations our production server has a single HttpClient instance which we reuse across requests.

      When inspecting the heap dump of a running process I see
      * Many Http2Connection objects with *finalStream* set to true.
      * The Http2ClientImpl "connections" pool is empty.
      * The HttpClientImpl "openConnections" set has the same number of entries as the number of Http2Connection objects
      * The PlainHttpConnection "closed" value is false and "connected" value is true

      This issue compounds on a running system and we need to periodically shut it down to drop the number of file descriptors

      This may be similar to this bug report: https://bugs.openjdk.org/browse/JDK-8326498

      REGRESSION : Last worked in version 17

      STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
      * Create a http client
      * Make 64 requests concurrently to the same domain (or URL)
      * Wait for requests to be done and go into and leave the process running (Thread.sleep forever)

      * Wait until the connections should have reasonably been terminated

      * Open visualVM and connect to the process
      * Run a GC to be sure and then a Heap dump
      * Look for "Http2Connection" and inspect the "finalStream" value of each
      * Look for "Http2ClientImpl" and inspect the "connections" value (this will likely be empty)
      * Look for "PlainHttpConnection" and inspect the "closed" and "connected" values

      EXPECTED VERSUS ACTUAL BEHAVIOR :
      EXPECTED -
      After a while I would expect the "Http2Connection" count to drop to zero
      ACTUAL -
      The "Http2Connection" count stays the same (in my case over 100 due to the server redirecting in the below example where 64 requests are made).

      ---------- BEGIN SOURCE ----------
      import java.net.URI;
      import java.net.http.HttpClient;
      import java.net.http.HttpRequest;
      import java.net.http.HttpResponse;
      import java.time.Duration;
      import java.util.ArrayList;
      import java.util.concurrent.CompletableFuture;
      import java.util.concurrent.ExecutionException;
      import java.util.concurrent.Executors;
      import java.util.stream.Collectors;

      public class Main {
          public static void main(String[] args) throws InterruptedException, ExecutionException {
              System.out.println("Hello world!");

              var executor = Executors.newWorkStealingPool();
              var httpClient = HttpClient.newBuilder()
                      .followRedirects(HttpClient.Redirect.ALWAYS)
                      .connectTimeout(Duration.ofSeconds(5))
                      .executor(executor)
                      .build();

              // todo enter a URL that is known to connect using HTTP2
              var url = "";
              var responses = new ArrayList<CompletableFuture<HttpResponse<String>>>();
              for (int i = 0; i < 64; i++) {
                  URI uri = URI.create(url);
                  var request = HttpRequest.newBuilder().GET()
                          .uri(uri)
      // .header("Authorization", "")
                          .build();
                  var response = httpClient.sendAsync(request, HttpResponse.BodyHandlers.ofString());
                  responses.add(response);
              }

              CompletableFuture.allOf(responses.toArray(new CompletableFuture[0])).get();
              System.out.println(responses.stream().map(r -> {
                  try {
                      return Integer.toString(r.get().statusCode());
                  } catch (InterruptedException e) {
                      throw new RuntimeException(e);
                  } catch (ExecutionException e) {
                      throw new RuntimeException(e);
                  }
              }).collect(Collectors.joining("\n")));

              while (true) {
                  // try and force gc to clear out references
                  System.gc();
                  Thread.sleep(5000);
              }
          }
      }

      ---------- END SOURCE ----------

            jpai Jaikiran Pai
            webbuggrp Webbug Group
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: