Authorization header is removed when a proxy Authenticator is set on HttpClient

ADDITIONAL SYSTEM INFORMATION :
Tested on Ubuntu Linux using

openjdk version "17.0.10" 2024-01-16
OpenJDK Runtime Environment (build 17.0.10+7-Ubuntu-122.04.1)
OpenJDK 64-Bit Server VM (build 17.0.10+7-Ubuntu-122.04.1, mixed mode, sharing)

A DESCRIPTION OF THE PROBLEM :
A customer of CloudBees CI, a product based on Jenkins, reported a critical functional regression when updating from Java 11 to 17. Investigation revealed that an HTTP request posting to a token request endpoint as part of an OpenID Connect authorization flow was missing the `Authorization: Bearer …` header which is of course mandatory for completing the flow. The regression was found to occur only when a systemwide HTTP proxy was defined, and furthermore only when this proxy required authentication; in this combination of circumstances, the Java 17 HTTP client silently drops the `Authorization` header (while the same code worked fine in Java 11).

Searching the web turned up JDK-8306745. The code in Jenkins is very similar to the unit test attached there. (https://github.com/jenkinsci/jenkins/pull/8990 touches the relevant code.) I believe that was closed improperly because it is mixing up two unrelated things: authentication to the proxy server and authentication to the origin server. Merely defining an `Authenticator` whose `getPasswordAuthentication` returns something for `RequestorType.PROXY` (suggesting a `Proxy-Authorization: Basic …`) should have no effect on an `Authorization` header (especially `Bearer`!) intended for the origin server. In fact merely calling `HttpClient.Builder.authenticator` with a no-op `Authenticator` that always returns `null` would silently strip any `Authorization` headers explicitly defined in requests, which seems surprising and counterintuitive.

The problem was especially perverse in the case reported by the CloudBees CI customer because the origin server in this case was explicitly listed among the no-proxy hosts, so `ProxySelector.select` was returning `Proxy.NO_PROXY`. (The proxy server was presumably required for unrelated origin servers on the Internet.) Thus calling both `.proxy` and `.authenticator` on the builder produces a client which is incapable of making authorized requests even when neither the proxy nor the authenticator were actually active.

The intent of the `CONTEXT_RESTRICTED` in https://github.com/openjdk/jdk/commit/028f2e14b3550dd1f2a358a2a0aaf83d4a824af0 (JDK-8213189) seems to have been to ensure that user code does not pass headers which are supposed to be managed by the library, which is sensible enough, but it is being applied indiscriminately here. I request that JDK-8306745 be reopened as a valid use case, so long as the authenticator is indeed limited to supplying `Proxy-Authorization`.

REGRESSION : Last worked in version 11.0.22

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
(supplied already in JDK-8306745)


CUSTOMER SUBMITTED WORKAROUND :
The suggested workaround

> don't use any authenticator, and explicitly handle 401/407 in the calling code

does not seem satisfactory. First of all, this essentially means reimplementing the logic of communicating with a proxy, which is something the library is supposed to handle for you. Worse, the explicit 407 handling would need to be present in each _user_ of the `HttpClient`. When the client builder is offered as a generic systemwide service to be used by a variety of code making distinct HTTP requests, this means it is impossible to configure an authenticated proxy systemwide and package that into a builder transparently, which seems to be the intent of the `.proxy` method.

The only other known workarounds are: stop using a proxy, or at least an authenticated one, if that is possible; switch to a third-party HTTP client (or perhaps the old `java.net.HttpURLConnection`); or go back to using Java 11 (since the new behavior cannot be disabled even on an emergency basis via `jdk.httpclient.allowRestrictedHeaders`).

FREQUENCY : always