We have 2 APP's that one is server and other one is a client.
Server is on a weblogic (APP A) and client is on a spring-boot (on tomcat - APP B). When we are doing load test which A sends request to B. But after some point there are hundreads of Bind Exception errors for a time, than load test keep running normally, than Bind Exception, than normal etc.. If we do load test with higher TPS we get these exception more frequently. This is the scenario:
Load test keep running, no error, 300 TPS, message count 10.000
Load test keep running, no error, 300 TPS, message count 30.000
Bind Exception, 300 TPS, mesasge count 32.000
Load test keep running, no error, 300 TPS, message count 40.000
Load test keep running, no error, 300 TPS, message count 50.000
Bind Exception, 300 TPS, mesasge count 52.000
Load test keep running, no error, 300 TPS, message count 60.000
...
...
Error is:
org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://XXXXXXX:9090/api/8252": Cannot assign requested address; nested exception is java.net.BindException: Cannot assign requested address
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:666)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:613)
at org.springframework.web.client.RestTemplate.postForLocation(RestTemplate.java:355)
at com.ttech.tims.tes.pushws.impl.PushConsumerThread.tryToSendPushRequest(PushConsumerThread.java:207)
at com.ttech.tims.tes.pushws.impl.PushConsumerThread.pushMessage(PushConsumerThread.java:162)
at com.ttech.tims.tes.pushws.impl.PushConsumerThread.run(PushConsumerThread.java:350)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:454)
at sun.nio.ch.Net.connect(Net.java:446)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
at weblogic.socket.NIOSocketMuxer.newSocket(NIOSocketMuxer.java:432)
at weblogic.socket.NIOSocketMuxer.newSocket(NIOSocketMuxer.java:364)
at weblogic.socket.ChannelSocketFactory.createSocket(ChannelSocketFactory.java:98)
at weblogic.net.http.HttpClient.openServer(HttpClient.java:384)
at weblogic.net.http.HttpClient.openServer(HttpClient.java:511)
at weblogic.net.http.HttpClient.New(HttpClient.java:313)
at weblogic.net.http.HttpClient.New(HttpClient.java:292)
at weblogic.net.http.HttpURLConnection.connect(HttpURLConnection.java:295)
at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:78)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:652)
... 6 more
What i have tried so far:
Increased clients max thread number (in application.properties from 100 to 500 etc.)
Increased weblogics max concurrent thread count from 100 to 200 (300, 400 etc.) via console
Checked max open files for server and client.
Checked firewall between server and client.
Monitored with Java Mission Control and checked for Thread Blocking status. Threads are blocked because of log4j writing errors with 300TPS when exception occures. But this doesn't solve the problem why we have those exceptions. Decraesed log level from Info to Error.
No improvements so far, still getting BindExceptions after some point. Any suggestions?
Thanks EJP for suggestions. Here is how i solve the problem. Spring Rest template is waiting for unix to close the connections. For this reason after i reach the maximum socket connections, connections wait in TIME_WAIT state. When we send Apache ClientHttpRequestFactory as a factory for spring RestTemplate, than apache handles requests with its connection pool. Here is the implementation:
#Bean("apacheRequestFactory")
public ClientHttpRequestFactory createRequestFactory() {
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
// maximum connections in the pool
connectionManager.setMaxTotal(this.systemPropertyBean.getPushConsumerThreadCnt());
// maximum concurrent connection to the hosts is equal to the our push thread count.
connectionManager.setDefaultMaxPerRoute(this.systemPropertyBean.getPushConsumerThreadCnt());
RequestConfig config = RequestConfig.custom()
.setConnectTimeout(this.systemPropertyBean.getPushTimeoutMillis())// 3 sn
.setConnectionRequestTimeout(this.systemPropertyBean.getPushTimeoutMillis())
.setSocketTimeout(this.systemPropertyBean.getPushTimeoutMillis()).build(); // read timeout
/* the Connection Timeout (http.connection.timeout) – the time to establish the connection with the remote host
the Socket Timeout (http.socket.timeout) – the time waiting for data – after the connection was established; maximum time of inactivity between two data packets
the Connection Manager Timeout (http.connection-manager.timeout) – the time to wait for a connection from the connection manager/pool
*/
CloseableHttpClient httpClient = HttpClientBuilder.create().setConnectionManager(connectionManager).setDefaultRequestConfig(config).build();
return new HttpComponentsClientHttpRequestFactory(httpClient);
}
#Bean
public RestTemplate restTemplate() {
// I was using SimpleClientHttpRequestFactory before.
RestTemplate restTemplate = new RestTemplate(createRequestFactory());
restTemplate.getMessageConverters().add(new MappingJackson2HttpMessageConverter());
restTemplate.getMessageConverters().add(new StringHttpMessageConverter());
return restTemplate;
}
Related
I have a java program where I input about 500 proxies and 500 “tokens.” It will create a new websocket connection using a random proxy for each token for identification. The websocket returns a time in milliseconds (usually around 40000 which is 40 seconds) for how long I should wait before sending a “heartbeat” message to that connection. How could I do this without slowing down or crashing my pc?
I felt very confused after reading the Connection Management doc of the Apache HTTP components module, and also a few other resources on connection keep alive strategy and connection eviction policy.
There are a bunch of adjectives used in there to describe the state of a connection like stale, idle, available, expired and closed etc. There isn't a lifecycle diagram describing how a connection changes among these states.
My confusion mainly arose from below situation.
I set a ConnectionKeepAliveStrategy that provides a KeepAliveDuration of 5 seconds via below code snippet.
ConnectionKeepAliveStrategy keepAliveStrategy = ( httpResponse, httpContext ) -> {
HeaderElementIterator iterator =
new BasicHeaderElementIterator( httpResponse.headerIterator( HTTP.CONN_KEEP_ALIVE ) );
while ( iterator.hasNext() )
{
HeaderElement header = iterator.nextElement();
if ( header.getValue() != null && header.getName().equalsIgnoreCase( "timeout" ) )
{
return Long.parseLong( header.getValue(), 10) * 1000;
}
}
return 5 * 1000;
};
this.client = HttpAsyncClients.custom()
.setDefaultRequestConfig( requestConfig )
.setMaxConnTotal( 500 )
.setMaxConnPerRoute( 500 )
.setConnectionManager( this.cm )
.setKeepAliveStrategy( keepAliveStrategy )
.build();
The server I am talking to does support connections to be kept alive. When I printed out the pool stats of the connection manager after executing around ~200 requests asynchronously in a single batch, below info was observed.
Total Stats:
-----------------
Available: 139
Leased: 0
Max: 500
Pending: 0
And after waiting for 30 seconds (by then the keep-alive timeout had long been exceeded), I started a new batch of the same HTTP calls. Upon inspecting the connection manager pool stats, the number of available connections are is still 139.
Shouldn't it be zero since the keep-alive timeout had been reached? The PoolStats Java doc states that Available is "the number of idle persistent connections". Are idle persistent connections considered alive?
I think Apache HttpClient: How to auto close connections by server's keep-alive time is a close hit but hope some expert could give an insightful explanation about the lifecycle of a connection managed by PoolingHttpClientConnectionManager.
Some other general questions:
Does the default connection manager used in HttpAsyncClients.createdDefault() handle connection keep-alive strategy and connection eviction on its own?
What are the requirements/limitations that could call for implementing them on a custom basis? Will they contradict each other?
Documenting some of my further findings which might partially fulfill as an answer.
Whether using a ConnectionKeepAliveStrategy to set a timeout on the keep alive session or not, the connections will end up in the TCP state of ESTABLISHED, as inspected via netstat -apt. And I observed that they are automatically recycled after around 5 minutes in my Linux test environment.
When NOT using a ConnectionKeepAliveStrategy, upon a second request batch the established connections will be reused.
When using a ConnectionKeepAliveStrategy and its timeout has NOT been reached, upon a second request batch the established connections will be reused.
When using a ConnectionKeepAliveStrategy and its timeout has been exceeded, upon a second request batch, the established connections will be recycled into the TIME_WAIT state, indicating that client side has decided to close the connections.
This recycling can be actively exercised by performing connectionManager.closeExpiredConnections(); in a separate connection evicting thread, which will lead the connections into TIME_WAIT stage.
I think the general observation is that ESTABLISHED connections are deemed as Available by the connection pool stats, and the connection keep alive strategy with a timeout does put the connections into expiry, but it only takes effect when new requests are processed, or when we specifically instruct the connection manager to close expired connections.
TCP state diagram from Wikipedia for reference.
Using the new java.net.http package released with JDK 11, an HttpRequest has been assembled with a deliberately low response timeout:
HttpRequest.Builder builder = HttpRequest.newBuilder(getEndpointUri());
addRequestHeaders(builder);
builder.POST(HttpRequest.BodyPublishers.ofString(rawXml));
builder.timeout(Duration.ofMillis(1));
HttpRequest httpRequest = builder.build();
The aim is to test that HttpTimeoutException outcomes are handled correctly, but unexpectedly this response timeout value is leading to an HttpConnectionTimeoutException, which is being caught by this code:
try {
HttpResponse<InputStream> httpResponse = completableExchange.join();
} catch (CompletionException ce) {
if (ce.getCause() instanceof HttpConnectTimeoutException) {
System.out.println("Connection timeout occurred!");
} else {
throw ce;
}
}
This means that a response timeout is causing the code to act as though a connection timeout has occurred. To the best of my understanding, the connection timeout and response timeout should be separate concepts, which should be possible to catch and handle separately.
The stack trace attached to the HttpConnectionTimeoutException looks like this:
java.net.http.HttpConnectTimeoutException: HTTP connect timed out
at java.net.http/jdk.internal.net.http.ResponseTimerEvent.handle(ResponseTimerEvent.java:68)
at java.net.http/jdk.internal.net.http.HttpClientImpl.purgeTimeoutsAndReturnNextDeadline(HttpClientImpl.java:1248)
at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:877)
Caused by: java.net.ConnectException: HTTP connect timed out
at java.net.http/jdk.internal.net.http.ResponseTimerEvent.handle(ResponseTimerEvent.java:69)
... 2 more
Am I misunderstanding the timeout concepts? Does the HttpRequest timeout value simply provide an alternative to the default of the HttpClient timeout value? Is there a reliable way to catch connection and response timeout as distinct events?
For what it's worth, the Javadoc for HttpRequest.Builder.timeout(Duration) says the following:
Sets a timeout for this request. If the response is not received within the specified timeout then an HttpTimeoutException is thrown from HttpClient::send or HttpClient::sendAsync completes exceptionally with an HttpTimeoutException. The effect of not setting a timeout is the same as setting an infinite Duration, ie. block forever.
To make things confusing, HttpConnectionTimeoutException is a subclass of HttpTimeoutException so technically the contract of the timeout(Duration) method is being satisfied. But this seems unhelpful.
(Before you ask: yes, the value passed to HttpRequest.Builder.timeout(Duration) is the deciding factor in whether or not an exception is thrown. So the exception is not based on the connection timeout value being used to create the HttpClient instance.)
IIRC you will get a HttpConnectionTimeoutException if the connection is not connected at the time the timeout is raised, or if the connect timeout is raised before the connection has finished connecting.
When sending a request - the underlying connection might already be connected or not - depending on whether a suitable existing connection was found in the pool. The request timeout starts immediately - independently of the state of the underlying connection. If the underlying connection is not connected yet, and the request timeout expires before it gets connected, then you will get a HttpConnectionTimeoutException because the connection could not be connected within the time allocated for the response to the request to be delivered. You could see it as the request timeout clipping the connect timeout.
Do you have any specific use case in mind for distinguishing the two cases:
HttpConnectionTimeoutException is raised because the connection could not be connected within the time specified by the connection timeout,
HttpConnectionTimeoutException is raised because the request timeout expired before the connection could be connected?
I have to call a web service that takes 40 seconds to respond. I'm in JBoss-EAP 6.4 and I use this code:
IService svc = service.getBasicHttpBindingIService();
[...]
((BindingProvider) svc).getRequestContext().put("com.sun.xml.ws.connect.timeout",timeout)
[...]
svc.processMessage(req);
When the timeout is less then 20000 (20 seconds), I get the expected exception:
HTTP transport error: java.net.SocketTimeoutException: connect timed out
If the timeout is bigger than 20000, i get another exeption after 20 second:
HTTP transport error: java.net.ConnectException: Connection timed out: connect
So, I can't get the response from the server (40 seconds to respond), I always get the exception. What kind of timeout I have to increase?
Why do you want the timeout to be 40 seconds? There are two timeouts represented by "com.sun.xml.internal.ws.connect.timeout" and "com.sun.xml.internal.ws.request.timeout". You could have been connected to the server, but the request is taking long to process in which case, the latter setting would take effect. Note: I believe these settings are for JBoss (I could be wrong, cross check it).
I have an issue with one of my AMQ producers, I have networks issues with it, consequently it fails to send messages to the Q.
However, the server opens a connections to the Q machine and maintain them intact. The number of connections raises, the Q machine is suffocated and cannot serve the rest of its producers and consumers.
Here is the exception I catch in my "bad" producer:
Caused by: javax.jms.JMSException: Wire format negotiation timeout: peer did not send his wire format.
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1413)
at org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1478)
at org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527)
at com.queue.service.ActiveMQ.ActiveMQWriter.initSession(ActiveMQWriter.java:163)
... 4 more
Caused by: java.io.IOException: Wire format negotiation timeout: peer did not send his wire format.
at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:99)
at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1388)
... 7 more
Running netstat command on my "bad" producer gives me State ESTABLISHED for every failed request.
Running netstat command on my Q server gives me State ESTABLISHED from the "bad" producer, for every failed request.
Question: what parameter I need to modify in the producer and/or Q in order to "break" those connections after a configurable time?
The best solution is to use failover transport, so that your clients can try connecting again if the first attempt fails.
You can also try extending wire format negotiation period (default 10 sec).
You can do that by using wireFormat.maxInactivityDurationInitalDelay property on the connection URL in your client to increase to 30 sec timeout
ActiveMQConnectionFactory cf = new ActiveMQConnectionFactory("failover:(tcp://localhost:61616?wireFormat.maxInactivityDurationInitalDelay=30000)");
http://activemq.apache.org/javaxjmsjmsexception-wire-format-negociation-timeout-peer-did-not-send-his-wire-format.html