Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8308144

Uncontrolled memory consumption in SSLFlowDelegate.Reader

XMLWordPrintable

    • b25
    • generic
    • generic
    • Verified

        A DESCRIPTION OF THE PROBLEM :
        When using HttpClient to make requests to HTTPS resources, there is an issue where the entire file is being downloaded into memory without the ability to limit the buffer size.
        If the SSLEngine cannot decode the entire buffer due to the algorithm's blocking nature, it returns a decoded chunk of data and BUFFER_UNDERFLOW status, which leads to SSLFlowDelegate.Reader requesting more data despite the output queue being full.
        https://github.com/openjdk/jdk/blob/ad348a8cec50561d3e295b6289772530f541c6b1/src/java.net.http/share/classes/jdk/internal/net/http/common/SSLFlowDelegate.java#L470

        In some cases, we have observed a queue called SSLFlowDelegate.Reader.outputQ in our heap dumps with hundreds of thousands of byte-buffers.

        STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
        Make http-request without body processing for some large file over https and add system properties -Djdk.internal.httpclient.debug=true.
        After that check output for lines like "Adding 16384 to outputQ queue"

        EXPECTED VERSUS ACTUAL BEHAVIOR :
        EXPECTED -
        There should be just few such lines for http-header.
        ACTUAL -
        A lot of such lines:

        DEBUG: [HttpClient-1-Worker-0] [194ms] SSL Writer(SocketTube(1)) Adding 476 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [248ms] SSL Writer(SocketTube(1)) Adding 42 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [248ms] SSL Writer(SocketTube(1)) Adding 6 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [250ms] SSL Writer(SocketTube(1)) Adding 45 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [273ms] SSL Writer(SocketTube(1)) Adding 129 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [297ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [309ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [310ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [312ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [313ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [313ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [319ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [321ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [322ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [323ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [324ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [324ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [325ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [330ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [335ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [336ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [336ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [336ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [337ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [337ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [338ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [340ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [340ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [341ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [348ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [352ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [353ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [353ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [353ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [354ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [354ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [354ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [355ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [355ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [356ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [361ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [363ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [366ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [367ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [367ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [368ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [368ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [369ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [369ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [370ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [370ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [371ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [375ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [378ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [379ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [380ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [381ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [382ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [382ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [383ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [383ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [384ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [384ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [385ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [390ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [390ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [391ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [392ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [395ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [395ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [396ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [396ms] SSL Reader(SocketTube(1)) Adding 16384 to outputQ queue
        DEBUG: [HttpClient-1-Worker-0] [396ms] SSL Reader(SocketTube(1)) Adding 14959 to outputQ queue

        ---------- BEGIN SOURCE ----------
        import java.net.URI;
        import java.net.http.HttpClient;
        import java.net.http.HttpRequest;
        import java.net.http.HttpResponse;

        public class Main {
            public static void main(String[] args) throws InterruptedException {
                try (var client = HttpClient.newHttpClient()) {
                    var request = HttpRequest.newBuilder()
                            .uri(URI.create("https://nginx.org/download/nginx-1.23.4.tar.gz"))
                            .build();
                    client.sendAsync(request, HttpResponse.BodyHandlers.ofInputStream());
                    Thread.sleep(1000);
                    System.exit(0);
                }
            }
        }

        ---------- END SOURCE ----------

        FREQUENCY : always


              jpai Jaikiran Pai
              webbuggrp Webbug Group
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated:
                Resolved: