My colleague David Benjamin discovered the following issue:

We ran into an issue with the OpenJDK TLS 1.3 stack regarding KeyUpdates:

The logic around SSLReadCipher.atKeyLimit() and its callers tries to trigger a KeyUpdate of type update_requested when the peer has sent data exceeding OpenJDK's limit. This is checked roughly after every record in SSLSocketImpl and SSLEngineImpl.

However, as Section 4.6.3 of RFC 8446 describes, an arbitrarily large amount of data may come in before OpenJDK receives the corresponding KeyUpdate from the peer:

> Note that implementations may receive an arbitrary number of
> messages between sending a KeyUpdate with request_update set to
> "update_requested" and receiving the peer's KeyUpdate, because those
> messages may already be in flight.

This may happen because the link has a high bandwidth-delay product (BDP), or, in some protocols, the peer may not not be reading at all. For example, in a typical HTTP/1.1 server, the server will read the client's request, then switch to sending the response. While sending the response, it doesn't read from the link at all. If the response trips OpenJDK's key limit, the HTTP/1.1 server will not see the KeyUpdate until after it has finished sending the response.

However, we have observed that OpenJDK sends such a KeyUpdate in response to every record past the limit. This means that the peer gets a storm of KeyUpdates, when only one was needed. This can have several consequences:

The storm of KeyUpdates may trip the peer's DoS limits for the number of consecutive KeyUpdates and break the connection.
If the peer isn't reading at all, sufficiently many KeyUpdates will exceed the link's BDP and writing new data will block. From there, depending on how OpenJDK writes data, the TLS stack may either block indefinitely or buffer unbounded data in memory. These scenarios are, respectively, a deadlock or potential DoS vector.

Note that, although SSLReadCipher.atKeyLimit() sets keyLimitEnabled to false, that doesn't actually prevent a KeyUpdate storm. The only thing that checks keyLimitEnabled is the code to update keyLimitCountdown. atKeyLimit() still returns true at that point, so SSLSocketImpl, etc., will continue to send KeyUpdates.

Instead, once the stack has sent one limit-based KeyUpdate, it should stop sending more. Any subsequent ones will arrive after the first one anyway, so it won't do any good.