-
Bug
-
Resolution: Incomplete
-
P4
-
None
-
11
-
x86_64
-
windows_10
-
Not verified
ADDITIONAL SYSTEM INFORMATION :
OS: windows 10
Java: JDK11 (Not OpenJDK)
A DESCRIPTION OF THE PROBLEM :
when i try to fix production issues, i figure out that httpclient built-in jdk cannot handle requests over 1000 normally, but apache http client can handle them all successfully
ERROR MESSAGES/STACK TRACES THAT OCCUR :
requests cannot be handled all successfully, but apache httpclient can
sometimes request timeout or null returned from jdk httpclient, sometimes, no error message thrown, but requests cannot be handled continously
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
create a spring boot project
add a controller, any name is ok
create a simeple api in that controller, code is as below
@RestController
public class UserController {
@PostMapping("/test")
public String test(HttpServletRequest request) throws InterruptedException {
return "port: " + request.getRemotePort();
}
}
create two test method, one is apache httpclient, one is jdk httpclient
public class HttpTest {
public static void main(String[] args) throws Exception{
long start = System.currentTimeMillis();
CountDownLatch c = new CountDownLatch(1000);
AtomicInteger ai = new AtomicInteger(0);
Set<String> exceptions = new HashSet<>();
for (int i = 0; i < c.getCount(); i++) {
post(c, ai, exceptions);
}
c.await();
System.err.println((System.currentTimeMillis() - start ) / 1000);
}
static void post(CountDownLatch c, AtomicInteger ai, Set<String> exceptions) throws Exception{
HttpClient httpClient = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(5))
.executor(new ThreadPoolExecutor(2,10,20, TimeUnit.SECONDS,
new LinkedBlockingQueue<>(5), Executors.defaultThreadFactory(), new ThreadPoolExecutor.CallerRunsPolicy()))
.build();
new Thread(() -> {
try {
long start = System.currentTimeMillis();
HttpRequest httpRequest = HttpRequest.newBuilder()
.uri(URI.create("http://localhost:8080/test?id=" + Math.random()))
// means read timeout
.timeout(Duration.ofSeconds(15))
.POST(HttpRequest.BodyPublishers.ofString(MapperUtil.getIgnoreUnkownParamOm().writeValueAsString(Maps.newHashMap())))
.header(HttpClientConstant.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
.version(HttpClient.Version.HTTP_2)
.build();
HttpResponse<String> response = httpClient.send(httpRequest, HttpResponse.BodyHandlers.ofString());
if (response.statusCode() != 200) {
System.out.println(response.statusCode());
ai.addAndGet(1);
} else {
System.err.println(Thread.currentThread().getId() + " request success");
}
c.countDown();
} catch (Exception e) {
c.countDown();
ai.addAndGet(1);
exceptions.add(e.getMessage());
}
}).start();
}
}
above is test code from jdk http client
public class HttpTest1 {
public static void main(String[] args) throws Exception{
long start = System.currentTimeMillis();
HttpClient client = init();
CountDownLatch c = new CountDownLatch(2000);
AtomicInteger ai = new AtomicInteger(0);
Set<String> exceptions = new HashSet<>();
for (int i = 0; i < c.getCount(); i++) {
post(client, c, ai);
}
c.await();
System.err.println("time: " + (System.currentTimeMillis() - start) / 1000);
}
static HttpClient init() throws Exception{
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
connectionManager.setMaxTotal(500);
connectionManager.setValidateAfterInactivity(100);
connectionManager.setDefaultMaxPerRoute(50);
HttpClient httpClient = HttpClients.custom()
.setConnectionManager(connectionManager)
.setKeepAliveStrategy(new ConnectionKeepAliveStrategy() {
@Override
public long getKeepAliveDuration(HttpResponse response, HttpContext context) {
HeaderElementIterator it = new BasicHeaderElementIterator
(response.headerIterator(HTTP.CONN_KEEP_ALIVE));
while (it.hasNext()) {
HeaderElement he = it.nextElement();
String param = he.getName();
String value = he.getValue();
if (value != null && param.equalsIgnoreCase
("timeout")) {
return Long.parseLong(value) * 1000;
}
}
return 60 * 1000;
}
})
.setDefaultRequestConfig(RequestConfig.custom().setStaleConnectionCheckEnabled(true).build()).build();
return httpClient;
}
static void post(HttpClient client, CountDownLatch c, AtomicInteger ai ) throws Exception{
new Thread(() -> {
try {
HttpPost post = new HttpPost("http://localhost:8080/test");
post.setEntity(new StringEntity(JSONObject.toJSONString(Maps.newHashMap())));
HttpResponse response = client.execute(post);
final HttpEntity entity = response.getEntity();
EntityUtils.consume(entity);
c.countDown();
if (response.getStatusLine().getStatusCode() != 200) {
ai.addAndGet(1);
} else{
System.err.println(Thread.currentThread().getId() + " request success");
}
} catch (Exception ex) {
c.countDown();
ai.addAndGet(1);
System.err.println(ex.getMessage());
}
}).start();
}
}
this is test code from apache http client
run the two test case, we can see that all requests, no matter number, can be handled successfully but jdk cannot
ACTUAL -
apache httpclient can handle them successfully
jdk httpclient cannot ,sometimes request timeout thrown, sometimes null returned, sometimes no error message thrown, but remaining requetss cannot be handled
---------- BEGIN SOURCE ----------
as steps to reproduce
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
swith httpclient implementation from jdk build in to apache
FREQUENCY : always
OS: windows 10
Java: JDK11 (Not OpenJDK)
A DESCRIPTION OF THE PROBLEM :
when i try to fix production issues, i figure out that httpclient built-in jdk cannot handle requests over 1000 normally, but apache http client can handle them all successfully
ERROR MESSAGES/STACK TRACES THAT OCCUR :
requests cannot be handled all successfully, but apache httpclient can
sometimes request timeout or null returned from jdk httpclient, sometimes, no error message thrown, but requests cannot be handled continously
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
create a spring boot project
add a controller, any name is ok
create a simeple api in that controller, code is as below
@RestController
public class UserController {
@PostMapping("/test")
public String test(HttpServletRequest request) throws InterruptedException {
return "port: " + request.getRemotePort();
}
}
create two test method, one is apache httpclient, one is jdk httpclient
public class HttpTest {
public static void main(String[] args) throws Exception{
long start = System.currentTimeMillis();
CountDownLatch c = new CountDownLatch(1000);
AtomicInteger ai = new AtomicInteger(0);
Set<String> exceptions = new HashSet<>();
for (int i = 0; i < c.getCount(); i++) {
post(c, ai, exceptions);
}
c.await();
System.err.println((System.currentTimeMillis() - start ) / 1000);
}
static void post(CountDownLatch c, AtomicInteger ai, Set<String> exceptions) throws Exception{
HttpClient httpClient = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(5))
.executor(new ThreadPoolExecutor(2,10,20, TimeUnit.SECONDS,
new LinkedBlockingQueue<>(5), Executors.defaultThreadFactory(), new ThreadPoolExecutor.CallerRunsPolicy()))
.build();
new Thread(() -> {
try {
long start = System.currentTimeMillis();
HttpRequest httpRequest = HttpRequest.newBuilder()
.uri(URI.create("http://localhost:8080/test?id=" + Math.random()))
// means read timeout
.timeout(Duration.ofSeconds(15))
.POST(HttpRequest.BodyPublishers.ofString(MapperUtil.getIgnoreUnkownParamOm().writeValueAsString(Maps.newHashMap())))
.header(HttpClientConstant.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
.version(HttpClient.Version.HTTP_2)
.build();
HttpResponse<String> response = httpClient.send(httpRequest, HttpResponse.BodyHandlers.ofString());
if (response.statusCode() != 200) {
System.out.println(response.statusCode());
ai.addAndGet(1);
} else {
System.err.println(Thread.currentThread().getId() + " request success");
}
c.countDown();
} catch (Exception e) {
c.countDown();
ai.addAndGet(1);
exceptions.add(e.getMessage());
}
}).start();
}
}
above is test code from jdk http client
public class HttpTest1 {
public static void main(String[] args) throws Exception{
long start = System.currentTimeMillis();
HttpClient client = init();
CountDownLatch c = new CountDownLatch(2000);
AtomicInteger ai = new AtomicInteger(0);
Set<String> exceptions = new HashSet<>();
for (int i = 0; i < c.getCount(); i++) {
post(client, c, ai);
}
c.await();
System.err.println("time: " + (System.currentTimeMillis() - start) / 1000);
}
static HttpClient init() throws Exception{
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
connectionManager.setMaxTotal(500);
connectionManager.setValidateAfterInactivity(100);
connectionManager.setDefaultMaxPerRoute(50);
HttpClient httpClient = HttpClients.custom()
.setConnectionManager(connectionManager)
.setKeepAliveStrategy(new ConnectionKeepAliveStrategy() {
@Override
public long getKeepAliveDuration(HttpResponse response, HttpContext context) {
HeaderElementIterator it = new BasicHeaderElementIterator
(response.headerIterator(HTTP.CONN_KEEP_ALIVE));
while (it.hasNext()) {
HeaderElement he = it.nextElement();
String param = he.getName();
String value = he.getValue();
if (value != null && param.equalsIgnoreCase
("timeout")) {
return Long.parseLong(value) * 1000;
}
}
return 60 * 1000;
}
})
.setDefaultRequestConfig(RequestConfig.custom().setStaleConnectionCheckEnabled(true).build()).build();
return httpClient;
}
static void post(HttpClient client, CountDownLatch c, AtomicInteger ai ) throws Exception{
new Thread(() -> {
try {
HttpPost post = new HttpPost("http://localhost:8080/test");
post.setEntity(new StringEntity(JSONObject.toJSONString(Maps.newHashMap())));
HttpResponse response = client.execute(post);
final HttpEntity entity = response.getEntity();
EntityUtils.consume(entity);
c.countDown();
if (response.getStatusLine().getStatusCode() != 200) {
ai.addAndGet(1);
} else{
System.err.println(Thread.currentThread().getId() + " request success");
}
} catch (Exception ex) {
c.countDown();
ai.addAndGet(1);
System.err.println(ex.getMessage());
}
}).start();
}
}
this is test code from apache http client
run the two test case, we can see that all requests, no matter number, can be handled successfully but jdk cannot
ACTUAL -
apache httpclient can handle them successfully
jdk httpclient cannot ,sometimes request timeout thrown, sometimes null returned, sometimes no error message thrown, but remaining requetss cannot be handled
---------- BEGIN SOURCE ----------
as steps to reproduce
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
swith httpclient implementation from jdk build in to apache
FREQUENCY : always