Application description :
I'm using Apache HTTP Async Client ( Version 4.1.1 ) Wrapped By Comsat's Quasar FiberHttpClient ( version 0.7.0 ) in order to run & execute a highly concurrent Java application that uses fibers to internally send http requests to multiple HTTP end-points
The Application is running on top of tomcat( however , fibers are used only for internal request dispatching. tomcat servlet requests are still handled the standard blocking way )
Each external request opens 15-20 Fibers internally , each fiber builds an HTTP request and uses the FiberHttpClient to dispatch it
I'm using a c44xlarge server ( 16 cores ) to test my application
The end-points i'm connecting to preempt keep-alive connections, meaning if I try to maintain by resusing sockets , conncetions get closed during requests execution attempts. Therefor , I disable connection recycling.
According to the above sections, here's the tunning for my fiber http client ( which of course I'm using a single instance of ):
PoolingNHttpClientConnectionManager connectionManager =
new PoolingNHttpClientConnectionManager(
new DefaultConnectingIOReactor(
IOReactorConfig.
custom().
setIoThreadCount(16).
setSoKeepAlive(false).
setSoLinger(0).
setSoReuseAddress(false).
setSelectInterval(10).
build()
)
);
connectionManager.setDefaultMaxPerRoute(32768);
connectionManager.setMaxTotal(131072);
FiberHttpClientBuilder fiberClientBuilder = FiberHttpClientBuilder.
create().
setDefaultRequestConfig(
RequestConfig.
custom().
setSocketTimeout(1500).
setConnectTimeout(1000).
build()
).
setConnectionReuseStrategy(NoConnectionReuseStrategy.INSTANCE).
setConnectionManager(connectionManager).
build();
ulimits for open-files are set super high ( 131072 for both soft and hard values )
Eden is set for 18GB , Total heap size is 24GB
OS Tcp stack is also well tuned :
kernel.printk = 8 4 1 7
kernel.printk_ratelimit_burst = 10
kernel.printk_ratelimit = 5
net.ipv4.ip_local_port_range = 8192 65535
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.optmem_max = 40960
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.netdev_max_backlog = 100000
net.ipv4.tcp_max_syn_backlog = 100000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 1
Problem description
Under low-medium load all is well , connections are leased , cloesd and the pool replenishes
Beyond some concurrency point , the IOReactor Threads ( 16 of them ) seem to stop functioning properly, prior to dying.
I've written a small thread to get the pool stats and print them each second. At around 25K leased connections , actual data is not sent anymore over the socket connections , The Pending stat clibms to a sky-rocketing 30K pending connection requests as well
This situation persists and basically renders the application useless. At some point the I/O Reactor threads die, not sure when and I haven't been able to catch the exceptions so far
lsofing the java process , I can see it has tens of thousands of file descriptors , almost all of them are in CLOSE_WAIT ( which makes sense , as the I/O reactor thread die/stop functioning and never get to actually closing them
During the time the application breaks, the server is not heavily overloaded/cpu stressed
Questions
I'm guessing I am reaching some sort of boundary somewhere , though I'm rather clueless as to what or where it may reside. Except from the following
Is it possible I'm reaching an OS port ( all applicative requests are originating from a single internal IP after all) limits and creates an error that sends IO Reactor threads to die ( something similar to open files limit errors ) ?
Forgot to answer this, but I got what's going on roughly a week after posting the question :
There was some sort of miss-configuration that caused the io-reactor to spawn with only 2 threads.
Even after providing more reactor threads, the issue persisted. It turns out that our outgoing requests were mostly SSL. Apache SSL connection handling propagates the core handling to the JVM's SSL facilities which simply - are not efficient enough for handling thousands of SSL connections requests per second. Being more specific, some methods inside SSLEngine(If I recall correctly) are synchronized. doing thread-dumps under high loads shows the IORecator threads blocking each-other while trying to open SSL connections.
Even trying to create a pressure release valve in the form of connection lease-timeout didn't work because the backlogs created were to large, rendering the application useless.
Offloading SSL outgoing requests handling to nginx performed even worse - because the remote endpoints are terminating the requests preemptively, SSL client session cache could not be used ( same goes for the JVM implementation ).
Wound up putting a semaphore in-front of the entire module, limiting the whole thing to ~6000 at any given moment, which solved the issue.
Related
I felt very confused after reading the Connection Management doc of the Apache HTTP components module, and also a few other resources on connection keep alive strategy and connection eviction policy.
There are a bunch of adjectives used in there to describe the state of a connection like stale, idle, available, expired and closed etc. There isn't a lifecycle diagram describing how a connection changes among these states.
My confusion mainly arose from below situation.
I set a ConnectionKeepAliveStrategy that provides a KeepAliveDuration of 5 seconds via below code snippet.
ConnectionKeepAliveStrategy keepAliveStrategy = ( httpResponse, httpContext ) -> {
HeaderElementIterator iterator =
new BasicHeaderElementIterator( httpResponse.headerIterator( HTTP.CONN_KEEP_ALIVE ) );
while ( iterator.hasNext() )
{
HeaderElement header = iterator.nextElement();
if ( header.getValue() != null && header.getName().equalsIgnoreCase( "timeout" ) )
{
return Long.parseLong( header.getValue(), 10) * 1000;
}
}
return 5 * 1000;
};
this.client = HttpAsyncClients.custom()
.setDefaultRequestConfig( requestConfig )
.setMaxConnTotal( 500 )
.setMaxConnPerRoute( 500 )
.setConnectionManager( this.cm )
.setKeepAliveStrategy( keepAliveStrategy )
.build();
The server I am talking to does support connections to be kept alive. When I printed out the pool stats of the connection manager after executing around ~200 requests asynchronously in a single batch, below info was observed.
Total Stats:
-----------------
Available: 139
Leased: 0
Max: 500
Pending: 0
And after waiting for 30 seconds (by then the keep-alive timeout had long been exceeded), I started a new batch of the same HTTP calls. Upon inspecting the connection manager pool stats, the number of available connections are is still 139.
Shouldn't it be zero since the keep-alive timeout had been reached? The PoolStats Java doc states that Available is "the number of idle persistent connections". Are idle persistent connections considered alive?
I think Apache HttpClient: How to auto close connections by server's keep-alive time is a close hit but hope some expert could give an insightful explanation about the lifecycle of a connection managed by PoolingHttpClientConnectionManager.
Some other general questions:
Does the default connection manager used in HttpAsyncClients.createdDefault() handle connection keep-alive strategy and connection eviction on its own?
What are the requirements/limitations that could call for implementing them on a custom basis? Will they contradict each other?
Documenting some of my further findings which might partially fulfill as an answer.
Whether using a ConnectionKeepAliveStrategy to set a timeout on the keep alive session or not, the connections will end up in the TCP state of ESTABLISHED, as inspected via netstat -apt. And I observed that they are automatically recycled after around 5 minutes in my Linux test environment.
When NOT using a ConnectionKeepAliveStrategy, upon a second request batch the established connections will be reused.
When using a ConnectionKeepAliveStrategy and its timeout has NOT been reached, upon a second request batch the established connections will be reused.
When using a ConnectionKeepAliveStrategy and its timeout has been exceeded, upon a second request batch, the established connections will be recycled into the TIME_WAIT state, indicating that client side has decided to close the connections.
This recycling can be actively exercised by performing connectionManager.closeExpiredConnections(); in a separate connection evicting thread, which will lead the connections into TIME_WAIT stage.
I think the general observation is that ESTABLISHED connections are deemed as Available by the connection pool stats, and the connection keep alive strategy with a timeout does put the connections into expiry, but it only takes effect when new requests are processed, or when we specifically instruct the connection manager to close expired connections.
TCP state diagram from Wikipedia for reference.
Javax mail version used 1.6.2
manually setting JavaMailSender
Timeout thing I tried with mail.smtp.timeout & mail.smtps.timeout.
And, I tried with both String & Integer value 3000.
String timeOut = "3000";
Properties pros = new Properties();
pros.put("mail.smtp.timeout", timeOut);
pros.put("mail.smtp.connectiontimeout", timeOut);
pros.put("mail.smtp.writetimeout", timeOut);
pros.put("mail.smtp.auth", "true");
pros.put("mail.smtp.starttls.enable", "true");
jmailSender.setJavaMailProperties(pros);
return jmailSender;
It's taking around 7 seconds without any fail.
Since by default is infinite, so most probably it is not setting somehow
Are any properties missing or something else?
The properties mail.smtp.connectiontimeout and `mail.smtps.connectiontimeout only apply while establishing the connection. It is not related to any timeouts during transport.
The properties mail.smtp.timeout and mail.smtps.timeout are related to the time blocked waiting for a read. This is related to reading SMTP response codes.
The properties mail.smtp.writetimeout and mail.smtps.writetimeout are related to writing chunks of data which can vary in size.
None of these timeouts represent a deadline for a single transaction of sending a mime message. What is happening is that there is no single action (connect, read, write) that is exceeding the 3000ms.
For example, connect could take 1000ms, followed by say 30 requests (write) and response parsing (reads) that take 100ms, and set of say 3 writes to send the message that take 1000ms each due to the speed of the network and size of the message. That is 1000 + (30 * 100) + (3 * 1000) = 7000ms total time but no single action exceeded the timeouts.
In a test environment
Set all timeouts to 3000.
Set connectimeout to 1 and test. You should see the connection fail.
Restart the test by setting it back to 3000 and set timeout to 1. You should see the reads fail.
Restart the test by setting it back to 3000 and set writetimeout to 1. You should see the transport fail.
If the test doesn't act this way you either haven't set the properties correctly (typo or smtp vs. smtps). Or you are really lucky to have such low latency.
I've been looking a bit to Flume's HttpSource internals, trying yo figure out how the Jetty server is used.
I've seen a single element list of Connectors is used; this Connector will listen for incoming Http connections on configured Http host and port. Then a Context is created for the root path, and a HttpServlet is added to this Context containing the logic to be executed when a connection is received. Finally, the Jetty server is started.
Connector[] connectors = new Connector[1];
if (sslEnabled) {
SslSocketConnector sslSocketConnector = new HTTPSourceSocketConnector(excludedProtocols);
...
connectors[0] = sslSocketConnector;
} else {
SelectChannelConnector connector = new SelectChannelConnector();
...
connectors[0] = connector;
}
connectors[0].setHost(host);
connectors[0].setPort(port);
srv.setConnectors(connectors);
try {
org.mortbay.jetty.servlet.Context root = new org.mortbay.jetty.servlet.Context(srv, "/", org.mortbay.jetty.servlet.Context.SESSIONS);
root.addServlet(new ServletHolder(new FlumeHTTPServlet()), "/");
HTTPServerConstraintUtil.enforceConstraints(root);
srv.start();
...
My question is, seen the above implementation: does such a Jetty server create a thread for each incoming Http connection? Or does a unique HttpServlet serve all the requests, one by one, sequentially?
Thanks for helping!
First of note: org.mortbay.jetty means you are using a very old version of Jetty. Probably Jetty 5 or Jetty 6. Those have been EOL (End of Life'd) way back in 2010 (and earlier).
Back in the Jetty 6 days, there was a ThreadPool that was used on-demand, and depending on your Connector type it would either result in a thread per connection (known as blocking connectors), or a thread per nio selection (in this case your 1 connections have many threads over the lifetime of the connection, but never more than 1 active per connection).
Starting with Jetty 8, and Servlet async, this threading model was refactored to favor async behavior of request processing more.
With Jetty 9, all blocking connectors were dropped in favor of supporting fully async processing of the request, its inputstreams, and its outputstreams.
The current model is for a ThreadPool of threads to be used, on demand, only when needed by a connection (this could be for processing of the request, or the response, or reading the request body content, or writing the response body content, or active websocket streaming, etc...)
This model is preferred for SPDY and HTTP/2 based support, where you have multiple requests per physical connection. But know that in those models its quite possible to have multiple active threads per physical connection, depending on behavior of your servlets.
Also, the web application itself can choose to spin up more threads for its own processing, such as via the servlet async processing behaviors, or to initiate outgoing requests to other services, or to process other tasks that are unrelated to a specific request / response context.
I have developed a standalone Javase client which performs an EJB Lookup to a remote server and executes its method.The Server application is in EJB 3.0
Under some strange magical but rare situations my program hangs indefinetly, on looking inside the issue it seems that while looking up the ejb on the server, I never get the response from the server and it also never times out.
I would like to know if there is a property or any other way through which we can setup the lookup time in client or at the server side.
There is a very nice article that discusses ORB configuration best practices at DeveloperWorks here. I'm quoting the three different settings that can be configured at client (you, while doing a lookup and executing a method at a remote server);
Connect timeout: Before the client ORB can even send a request to a server, it needs to establish an IIOP connection (or re-use an
existing one). Under normal circumstances, the IIOP and underlying TCP
connect operations should complete very fast. However, contention on
the network or another unforeseen factor could slow this down. The
default connect timeout is indefinite, but the ORB custom property
com.ibm.CORBA.ConnectTimeout (in seconds) can be used to change the
timeout.
Locate request timeout: Once a connection has been established and a client sends an RMI request to the server, then LocateRequestTimeout
can be used to limit the time for the CORBA LocateRequest (a CORBA
“ping”) for the object. As a result, the LocateRequestTimeout should
be less than or equal to the RequestTimeout because it is a much
shorter operation in terms of data sent back and forth. Like the
RequestTimeout, the LocateRequestTimeout defaults to 180 seconds.
Request timeout: Once the client ORB has an established TCP connection to the server, it will send the request across. However, it
will not wait indefinitely for a response, by default it will wait for
180 seconds. This is the ORB request timeout interval. This can
typically be lowered, but it should be in line with the expected
application response times from the server.
You can try the following code, which performs task & then waits at most the time specified.
Future<Object> future = executorService.submit(new Callable<Object>() {
public Object call() {
return lookup(JNDI_URL);
}
});
try {
Object result = future.get(20L, TimeUnit.SECONDS); //- Waiting for at most 20 sec
} catch (ExecutionException ex) {
logger.log(LogLevel.ERROR,ex.getMessage());
return;
}
Also, the task can be cancelled by future.cancel(true).
Remote JNDI uses the ORB, so the only option available is com.ibm.CORBA.RequestTimeout, but that will have an affect on all remote calls. As described in the 7.0 InfoCenter, the default value is 180 (3 minutes).
My server uses data from an internal web service to construct its response, on a per request basis. I'm using Apache HttpClient 4.1 to make the requests. Each initial request will result in about 30 requests to the web service. Of these, 4 - 8 will end up with sockets stuck in CLOSE_WAIT, which never get released. Eventually these stuck sockets exceed my ulimit and my process runs out of file descriptors.
I don't want to just raise my ulimit (1024), because that will just mask the problem.
The reason I've moved to HttpClient is that java.net.HttpUrlConnection was behaving the same way.
I have tried moving to a SingleClientConnManager per request, and calling client.getConnectionManager().shutdown() on it, but sockets still end up stuck.
Should I be trying to solve this so that I end up with 0 open sockets while there are no running requests, or should I be concentrating on request persistence and pooling?
For clarity I'm including some details which may be relevant:
OS: Ubuntu 10.10
JRE: 1.6.0_22
Language: Scala 2.8
Sample code:
val cleaner = Executors.newScheduledThreadPool(1)
private val client = {
val ssl_ctx = SSLContext.getInstance("TLS")
val managers = Array[TrustManager](TrustingTrustManager)
ssl_ctx.init(null, managers, new java.security.SecureRandom())
val sslSf = new org.apache.http.conn.ssl.SSLSocketFactory(ssl_ctx, SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER)
val schemeRegistry = new SchemeRegistry()
schemeRegistry.register(new Scheme("https", 443, sslSf))
val connection = new ThreadSafeClientConnManager(schemeRegistry)
object clean extends Runnable{
override def run = {
connection.closeExpiredConnections
connection.closeIdleConnections(30, SECONDS)
}
}
cleaner.scheduleAtFixedRate(clean,10,10,SECONDS)
val httpClient = new DefaultHttpClient(connection)
httpClient.getCredentialsProvider().setCredentials(new AuthScope(AuthScope.ANY), new UsernamePasswordCredentials(username,password))
httpClient
}
val get = new HttpGet(uri)
val entity = client.execute(get).getEntity
val stream = entity.getContent
val justForTheExample = IOUtils.toString(stream)
stream.close()
Test: netstat -a | grep {myInternalWebServiceName} | grep CLOSE_WAIT
(Lists sockets for my process that are in CLOSE_WAIT state)
Post comment discussion:
This code now demonstrates correct usage.
One needs to pro-actively evict expired / idle connections from the connection pool, as in the blocking I/O model connections cannot react to I/O events unless they are being read from / written to. For details see
http://hc.apache.org/httpcomponents-client-dev/tutorial/html/connmgmt.html#d4e631
I've marked oleg's answer as correct, as it highlights an important usage point about HttpClient's connection pooling.
To answer my specific original question, though, which was "Should I be trying to solve for 0 unused sockets or trying to maximize pooling?"
Now that the pooling solution is in place and working correctly the application throughput has increased by about 150%. I attribute this to not having to renegotiate SSL and multiple handshakes, instead reusing persistent connections in accordance with HTTP 1.1.
It is definitely worth working to utilize pooling as intended, rather than trying to hack around with calling ThreadSafeClientConnManager.shutdown() after each request etcetera. If, on the other hand, you were calling arbitrary hosts and not reusing routes the way I am you might easily find that it becomes necessary to do that sort of hackery, as the JVM might surprise you with the long life of CLOSE_WAIT designated sockets if you're not garbage collecting very often.
I had the same issue and solved it using the suggesting found here: here. The author touches on some TCP basics:
When a TCP connection is about to close, its finalization is negotiated by both parties. Think of it as breaking a contract in a civilized manner. Both parties sign the paper and it’s all good. In geek talk, this is done via the FIN/ACK messages. Party A sends a FIN message to indicate it wants to close the socket. Party B sends an ACK saying it received the message and is considering the demand. Party B then cleans up and sends a FIN to Party A. Party A responds with the ACK and everyone walks away.
The problem comes in
when B doesn’t send its FIN. A is kinda stuck waiting for it. It has
initiated its finalization sequence and is waiting for the other party
to do the same.
He then mentions RFC 2616, 14.10 to suggest setting up an http header to solve this issue:
postMethod.addHeader("Connection", "close");
Honestly, I don't really know the implications of setting this header. But it did stop CLOSE_WAIT from happening on my unit tests.