I have a RESTful service that works very fast. I am testing it on localhost. The client is using Spring REST template. I started by using a naive approach:
RestTemplate restTemplate = new RestTemplate(Collections.singletonList(new GsonHttpMessageConverter()));
Result result = restTemplate.postForObject(url, payload, Result.class);
When I make a lot of these requests, I am getting the following exception:
Caused by: org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://localhost:8080/myservice":No buffer space available (maximum connections reached?): connect; nested exception is java.net.SocketException: No buffer space available (maximum connections reached?): connect
This is caused by connections not being closed and hanging in TIME_WAIT state. The exception starts happening when the ephemeral ports are exhausted. Then the execution waits for the ports to be free again. I am seeing peak performance with long breaks. The rate I am getting is almost what I need, but of course, these TIME_WAIT connections are not good. Tested both on Linux (Ubuntu 14) and Windows (7), similar results at different times due to different ranges of the ports.
To fix this, I tried using an HttpClient with HttpClientBuilder from Apache Http Components library.
RestTemplate restTemplate = new RestTemplate(Collections.singletonList(new GsonHttpMessageConverter()));
HttpClient httpClient = HttpClientBuilder.create()
.setMaxConnTotal(TOTAL)
.setMaxConnPerRoute(PER_ROUTE)
.build();
restTemplate.setRequestFactory(new HttpComponentsClientHttpRequestFactory(httpClient));
Result result = restTemplate.postForObject(url, payload, Result.class);
With this client, I see no exceptions. The client is now using only a very limited number of ephemeral ports. But whatever settings I use (TOTAL and PER_ROUTE), I can't get the performance I need.
Using the netstat command, I see that there are not many connections done to the server. I tried setting the numbers to several thousands, but it seems the client never uses that much.
Is there anything I can do to improve the performance, without opening too many connections?
UPDATE: I've tried setting number of total and per route connections to 5000 and 2500 but it still looks like the client is not creating more than a hundred (judging from netstat -n | wc -l). The REST service is implemented using JAX-RS and running on Jetty.
UPDATE2: I have now tuned the server with some memory settings and I am getting really good throughput. The naive approach is still a bit faster, but I think it's just a little overhead of the pooling on client side.
Actually Spring Boot is not leaking connections. What you're seeing here is standard behavior of the Linux kernel (and every major OS). All sockets that are closed from the machine go to a TIME_WAIT state for some duration of time. This is to prevent the next socket that uses that ephemeral port from receiving packets that were actually intended for the previous socket on that port. The difference you're seeing between the two is a result of the connection pooling approaches each one takes.
More specifically, RestTemplate does not use connection pooling by default. This means every rest call opens a new local ephemeral port and a new connection to the server. If your service is very fast, it will blow through its available local port range in no time at all. With the Apache HttpClient, you are taking advantage of connection pooling. This will prevent your application from seeing the problem that you described. However, given that your service is able to respond faster than the Linux kernel is taking sockets out of TIME_WAIT, connection pooling will make your client slower no matter what you do (if it didn't slow anything down - then you'd run out of local ephemeral ports again).
While it's possible to enable TCP reuse in the Linux kernel, it's can get dangerous (packets can get delayed and you could get ephemeral ports receiving random packets they don't understand which could cause all kinds of problems). The solution here is to use connection pooling as you have in the second example, with sufficiently high numbers to achieve close to the performance you're looking for.
To help you tune your connection pool, you'll want to tweak the maxConnPerRoute and maxConnTotal parameters. maxConnPerRoute limits the number of connections that will be made to a single IP:Port pair, and maxTotal limits the number of total connections that will ever be opened. In your case, since it appears all requests are made to the same location, you could set them to the same (high) value.
Related
In my batch application when I am sending requests across a network using a Web Service and Java, after running about 30000 requests and receiving the responses, the program throws a java.net.connectexception connection timed out exception.
I am using WildFly, along with some Java code in the middle to configure the requests (Strings) before sending it across the network.
After research the possible reasons I found for this is that there is either a Firewall blocking my access, which can't be true since it ran 90% of the requests already.
I've also seen somewhere that says that I could have overloaded the server, although I'm not sure what that means exactly.
You have filled up the server's listen backlog queue. This is caused by creating new connections faster than the server can accept them. You should look into connection pooling at the client, and handling multiple requests per connection at the server.
I am running an API on 10 different servers, all of them are behind a firewall. I am using jetty 8 to serve all the http requests. The use case for this API is short lived connections.
A few month ago I started to get random Too many open file descriptors errors. These errors make the server completely unresponsive and I need to restart the jetty server in order to fix that. Today this happened 0-10 times a day depending on the traffic I am getting.
After some investigations, I noticed that I am exhausting the number of available connections because all of them are stuck in the TIME_WAIT state so I can't create new ones.
ss -s
TCP: 13392 (estab 1549, closed 11439, orphaned 9, synrecv 0, timewait *11438*/0), ports 932
On this example the number of connections in TIME_WAIT state is pretty low but it can go up to 50k.
I have been trying several kernel tweaks and I also tried to set the SO_LINGER timer to 1 second for jetty sockets. All these changes helped reduce the frequency but I am still getting errors regularly.
Also worth mentioning, I am receiving around 3k requests/second on each server and the cpu usage is very low. The bottleneck to scale my traffic today is this connection issue.
Does anyone have an idea of what I can do to handle that correctly ?
'Too many open file descriptors' is probably caused by a resource leak in your application.
The TIME_WAIT state is caused by being the end that first sends a close, instead of the end that first receives the close. You might want to reconsider your application protocol so that it is the client which closes first. This is not too hard to arrange. It falls out free if you use client-side connection pooling for example.
These two conditions are not related. The TIME_WAIT state can only occur on a port whose socket has already been closed. It does not cause 'too many open file descriptors' problems.
I am using Tomcat 7.0.32, with Java 1.7. I have a use case where my client is single threaded and sends request at a high pace. Now my problem is my server, processes all requests, this it does at the expense of high latency. This is quite obvious since the communication is not concurrent.
I know if my connector is BIO and i set max threads to 1, my accept count as 1, then if i send 3 concurrent request, it fails with Connection Refused. Which is expected.
However, if the client is single threaded, then the above does not apply, i can continue to send as many requests as i want. However the latency drops. This again is natural. Now i am trying to check if at all there are any server(i.e. Tomcat) configuration available which can influence the OS to refuse connections if the time to acquire connections is high.
Any ideas?
Thanks,
Vicky
This is quite obvious since the communication is not concurrent.
It's not obvious to me. I run a dozen highly concurrent Tomcats.
If you are suggesting that Tomcat is not a highly concurrent server you are mistaken. It is. You need to reexamine your observations and your assumptions.
I am creating a web application having a login page , where number of users can tries to login at same time. so here I need to handle number of requests at a time.
I know this is already implemented for number of popular sites like G talk.
So I have some questions in my mind.
"How many requests can a port handle at a time ?"
How many sockets can I(server) create ? is there any limitations?
For e.g . As we know when we implement client server communication using Socket programming(TCP), we pass 'a port number(unreserved port number)to server for creating a socket .
So I mean to say if 100000 requests came at a single time then what will be approach of port to these all requests.
Is he manitains some queue for all these requests , or he just accepts number of requests as per his limit? if yes what is handling request limit size of port ?
Summary:
I want to know how server serves multiple requests simultaneously?I don't know any thing about it. I know we are connection to a server via its ip address and port number that's it.
So I thought there is only one port and many request come to that port only via different clients so how server manages all the requests?
This is all I want to know. If you explain this concept in detail it would be very helpful. Thanks any way.
A port doesn't handle requests, it receives packets. Depending on the implementation of the server this packets may be handled by one or more processes / threads, so this is unlimited theoretically. But you'll always be limited by bandwith and processing performance.
If lots of packets arrive at one port and cannot be handled in a timely manner they will be buffered (by the server, the operating system or hardware). If those buffers are full, the congestion maybe handled by network components (routers, switches) and the protocols the network traffic is based on. TCP for example has some methods to avoid or control congestion: http://en.wikipedia.org/wiki/Transmission_Control_Protocol#Congestion_control
This is typically configured in the application/web server you are using. How you limit the number of concurrent requests is by limiting the number of parallel worker threads you allow the server to spawn to serve requests. If more requests come in than there are available threads to handle them, they will start to queue up. This is the second thing you typically configure, the socket back-log size. When the back-log is full, the server will start responding with "connection refused" when new requests come in.
Then you'll probably be restricted by number of File Descriptors your os supports (in case of *nix) or the number of simultaneous connections your webserver supports. The OS maximum on my machine seems to be 75000.
100,000 concurrent connections should be easily possible in Java if you use something like Netty.
You need to be able to:
Accept incoming connections fast enough. The NIO framework helps enormously here, which is what Netty uses internally. There is a smallish queue for incoming requests, so you need to be able to handle these faster than the queue can fill up.
Create connections for each client (this implies some memory overhead for things like connection info, buffers etc.) - you may need to tweak your VM settings to have enough free memory for all the connections
See this article from 2009 where they discuss achieving 100,000 concurrent connections with about 20% CPU usage on a quad-core server.
I am using Jersey 1.4, the ApacheHttpClient, and the Apache MultiThreadedHttpConnectionManager class to manage connections. For the HttpConnectionManager, I set staleCheckingEnabled to true, maxConnectionsPerHost to 1000 and maxTotalConnections to 1000. Everything else is default. We are running in Tomcat and making connections out to multiple external hosts using the Jersey client.
I have noticed that after after a short period of time I will begin to see sockets in a CLOSE_WAIT state that are associated with the Tomcat process. Some monitoring with tcpdump shows that the external hosts appear to be closing the connection after some time but it's not getting closed on our end. Usually there is some data in the socket read queue, often 24 bytes. The connections are using https and the data seems to be encrypted so I'm not sure what it is.
I have checked to be sure that the ClientRequest objects that get created are closed. The sockets in CLOSE_WAIT do seem to get recycled and we're not running out of any resources, at least at this time. I'm not sure what's happening on the external servers.
My question is, is this normal and should I be concerned?
Thanks,
John
This is likely to be a device such as the firewall or the remote server timing out the TCP session. You can analyze packet captures of HTTPS using Wireshark as described on their SSL page:
http://wiki.wireshark.org/SSL
The staleCheckingEnabled flag only issues the check when you go to actually use the connection so you aren't using network resources (TCP sessions) when they aren't needed.