We Keep Coding

Using Spring REST template, either creating too many connections or slow

I have a RESTful service that works very fast. I am testing it on localhost. The client is using Spring REST template. I started by using a naive approach:
RestTemplate restTemplate = new RestTemplate(Collections.singletonList(new GsonHttpMessageConverter()));
Result result = restTemplate.postForObject(url, payload, Result.class);
When I make a lot of these requests, I am getting the following exception:
Caused by: org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://localhost:8080/myservice":No buffer space available (maximum connections reached?): connect; nested exception is java.net.SocketException: No buffer space available (maximum connections reached?): connect
This is caused by connections not being closed and hanging in TIME_WAIT state. The exception starts happening when the ephemeral ports are exhausted. Then the execution waits for the ports to be free again. I am seeing peak performance with long breaks. The rate I am getting is almost what I need, but of course, these TIME_WAIT connections are not good. Tested both on Linux (Ubuntu 14) and Windows (7), similar results at different times due to different ranges of the ports.
To fix this, I tried using an HttpClient with HttpClientBuilder from Apache Http Components library.
RestTemplate restTemplate = new RestTemplate(Collections.singletonList(new GsonHttpMessageConverter()));
HttpClient httpClient = HttpClientBuilder.create()
.setMaxConnTotal(TOTAL)
.setMaxConnPerRoute(PER_ROUTE)
.build();
restTemplate.setRequestFactory(new HttpComponentsClientHttpRequestFactory(httpClient));
Result result = restTemplate.postForObject(url, payload, Result.class);
With this client, I see no exceptions. The client is now using only a very limited number of ephemeral ports. But whatever settings I use (TOTAL and PER_ROUTE), I can't get the performance I need.
Using the netstat command, I see that there are not many connections done to the server. I tried setting the numbers to several thousands, but it seems the client never uses that much.
Is there anything I can do to improve the performance, without opening too many connections?
UPDATE: I've tried setting number of total and per route connections to 5000 and 2500 but it still looks like the client is not creating more than a hundred (judging from netstat -n | wc -l). The REST service is implemented using JAX-RS and running on Jetty.
UPDATE2: I have now tuned the server with some memory settings and I am getting really good throughput. The naive approach is still a bit faster, but I think it's just a little overhead of the pooling on client side.

Actually Spring Boot is not leaking connections. What you're seeing here is standard behavior of the Linux kernel (and every major OS). All sockets that are closed from the machine go to a TIME_WAIT state for some duration of time. This is to prevent the next socket that uses that ephemeral port from receiving packets that were actually intended for the previous socket on that port. The difference you're seeing between the two is a result of the connection pooling approaches each one takes.
More specifically, RestTemplate does not use connection pooling by default. This means every rest call opens a new local ephemeral port and a new connection to the server. If your service is very fast, it will blow through its available local port range in no time at all. With the Apache HttpClient, you are taking advantage of connection pooling. This will prevent your application from seeing the problem that you described. However, given that your service is able to respond faster than the Linux kernel is taking sockets out of TIME_WAIT, connection pooling will make your client slower no matter what you do (if it didn't slow anything down - then you'd run out of local ephemeral ports again).
While it's possible to enable TCP reuse in the Linux kernel, it's can get dangerous (packets can get delayed and you could get ephemeral ports receiving random packets they don't understand which could cause all kinds of problems). The solution here is to use connection pooling as you have in the second example, with sufficiently high numbers to achieve close to the performance you're looking for.
To help you tune your connection pool, you'll want to tweak the maxConnPerRoute and maxConnTotal parameters. maxConnPerRoute limits the number of connections that will be made to a single IP:Port pair, and maxTotal limits the number of total connections that will ever be opened. In your case, since it appears all requests are made to the same location, you could set them to the same (high) value.

How many requests can handle a port at 'a' time

I am creating a web application having a login page , where number of users can tries to login at same time. so here I need to handle number of requests at a time.
I know this is already implemented for number of popular sites like G talk.
So I have some questions in my mind.
"How many requests can a port handle at a time ?"
How many sockets can I(server) create ? is there any limitations?
For e.g . As we know when we implement client server communication using Socket programming(TCP), we pass 'a port number(unreserved port number)to server for creating a socket .
So I mean to say if 100000 requests came at a single time then what will be approach of port to these all requests.
Is he manitains some queue for all these requests , or he just accepts number of requests as per his limit? if yes what is handling request limit size of port ?
Summary:
I want to know how server serves multiple requests simultaneously?I don't know any thing about it. I know we are connection to a server via its ip address and port number that's it.
So I thought there is only one port and many request come to that port only via different clients so how server manages all the requests?
This is all I want to know. If you explain this concept in detail it would be very helpful. Thanks any way.

A port doesn't handle requests, it receives packets. Depending on the implementation of the server this packets may be handled by one or more processes / threads, so this is unlimited theoretically. But you'll always be limited by bandwith and processing performance.
If lots of packets arrive at one port and cannot be handled in a timely manner they will be buffered (by the server, the operating system or hardware). If those buffers are full, the congestion maybe handled by network components (routers, switches) and the protocols the network traffic is based on. TCP for example has some methods to avoid or control congestion: http://en.wikipedia.org/wiki/Transmission_Control_Protocol#Congestion_control

This is typically configured in the application/web server you are using. How you limit the number of concurrent requests is by limiting the number of parallel worker threads you allow the server to spawn to serve requests. If more requests come in than there are available threads to handle them, they will start to queue up. This is the second thing you typically configure, the socket back-log size. When the back-log is full, the server will start responding with "connection refused" when new requests come in.

Then you'll probably be restricted by number of File Descriptors your os supports (in case of *nix) or the number of simultaneous connections your webserver supports. The OS maximum on my machine seems to be 75000.

100,000 concurrent connections should be easily possible in Java if you use something like Netty.
You need to be able to:
Accept incoming connections fast enough. The NIO framework helps enormously here, which is what Netty uses internally. There is a smallish queue for incoming requests, so you need to be able to handle these faster than the queue can fill up.
Create connections for each client (this implies some memory overhead for things like connection info, buffers etc.) - you may need to tweak your VM settings to have enough free memory for all the connections
See this article from 2009 where they discuss achieving 100,000 concurrent connections with about 20% CPU usage on a quad-core server.

Socket open and close on 1 sec or to hold open

I need to have periodically communication with plc ( on every 1 sec ), I send message and I receive message. I use Socket class for this communication. Do I need to every 1 sec to open connection ( socket=new Socket(ipaddress, port) ), send messagethen socket.close() and so on , or to hold socket opet all time ?

I'll assume you're talking about TCP sockets here...
Apart from the obvious inefficiencies involved in setting up a TCP connection every second you're also likely to end up accumulating sockets in TIME_WAIT (hopefully on your client).
I've written about TIME_WAIT and the problems it causes with regards to server scalability and stability here on my blog: http://www.serverframework.com/asynchronousevents/2011/01/time-wait-and-its-design-implications-for-protocols-and-scalable-servers.html
Given the rate that you are opening and closing sockets (once a second would result in 240 (60*4) sockets sitting in TIME_WAIT in the normal (4 minute) 2MSL TIME_WAIT period) this shouldn't prove too much of a problem assuming the TIME_WAIT sockets ARE ending up on the client and not on the server and assuming you're not connecting to lots of servers every second, but... If you have many clients connecting to your server every second and you are not making sure that your server doesn't accumulate sockets in TIME_WAIT state then you may limit your server's scalability.
The alternative is to hold the socket connection open and only reopen it if and when it gets disrupted. This may prove slightly more complex for you to initially program but pooling the connection in this way is likely to be considerably more efficient (when you DO need to send data you just send the data and don't need to go through the TCP handshake to set the connection up) and much more resource efficient on the client; you're not perpetually holding 240 sockets in TIME_WAIT...

Keeping the socket always connected will reduce network traffic and computation time of the client. However, if the server uses blocking I/O it may run out of connection threads if many clients are remaining connected. You will also have to deal with dropped connections due to timeout, network issues and server downtime.

You can have perpetually connected clients wherein the client is always connected to the server. But the performance of this approach depends on how the server is implemented. If it uses a threaded model (one thread per client connection), you might find yourself running out of resources when handling a lot of client connections. You should be good to go with the perpetual client approach if your server uses a event based approach for handling requests as long as the "computation" isn't long lived.
As always, benchmark based on your use cases and you should be good to go.

Sockets in CLOSE_WAIT from Jersey Client

I am using Jersey 1.4, the ApacheHttpClient, and the Apache MultiThreadedHttpConnectionManager class to manage connections. For the HttpConnectionManager, I set staleCheckingEnabled to true, maxConnectionsPerHost to 1000 and maxTotalConnections to 1000. Everything else is default. We are running in Tomcat and making connections out to multiple external hosts using the Jersey client.
I have noticed that after after a short period of time I will begin to see sockets in a CLOSE_WAIT state that are associated with the Tomcat process. Some monitoring with tcpdump shows that the external hosts appear to be closing the connection after some time but it's not getting closed on our end. Usually there is some data in the socket read queue, often 24 bytes. The connections are using https and the data seems to be encrypted so I'm not sure what it is.
I have checked to be sure that the ClientRequest objects that get created are closed. The sockets in CLOSE_WAIT do seem to get recycled and we're not running out of any resources, at least at this time. I'm not sure what's happening on the external servers.
My question is, is this normal and should I be concerned?
Thanks,
John

This is likely to be a device such as the firewall or the remote server timing out the TCP session. You can analyze packet captures of HTTPS using Wireshark as described on their SSL page:
http://wiki.wireshark.org/SSL
The staleCheckingEnabled flag only issues the check when you go to actually use the connection so you aren't using network resources (TCP sessions) when they aren't needed.

Socket Programming -Java - Many Clients One Socket Question(s)

Essentially Im trying to get many many java clients connect to a socket on my ColdFusion server (Using the Socket Gateway). However before i even start to code this, Im a little confused about sockets and their performance. First of all, are sockets meant for many(1000+) clients connecting to one socket (say port 2202) on one server? How is the performance if all there waiting for is basically a ping, or something such that when these clients receive this "ping" they can go get some new data.
Thanks,
Faisal Abid

Socket is identified by following tuple,
Source IP
Source Port
Dest IP
Dest Port
Protocol (TCP or UDP)
Even 1000 clients all connect to the same port (dest port), each will get its own socket. So you will have 1000 sockets open.
It's going to be tough to maintain 1000 sockets with blocking I/O, which usually means 1000 threads. You need to use NIO. We have a server written with Mina, which can handle 2000 connections at peak.

First of all, are sockets meant for
many(1000+) clients connecting to one
socket (say port 2202) on one server
Yes, your server will open a socket on port 2202, and 1000 client will connect to it.
Server open server socket, and client will open client socket, it different.
How is the performance if all there
waiting for is basically a ping, or
something such that when these clients
receive this "ping" they can go get
some new data
On server, you use getInputStream function to get data from client, and getOutputStream function to send data to Client.
Notice: you should use Thread to process each request of client

There is nothing wrong with many socket clients that use blocking I/O (you may have a look at this article for more information about that). However, there are another approaches that can better fit your needs here:
nio;
multicasting;

Hi connecting with 1000 clients at the same time using simple java socket programming is not the perfect way. The probelm is that in fedora linux default maximum no of file can be open is 1024 and in windows it is 2048 or something like that. so in your server side you will find more than 1000 files open and after that if client will go on increaing then you will find too many files opnened error. so the better way is to use non blocking socket programming (I mean to say to use SocketChannel) Using socket channel at the same time as per i have check we can connect with 20,000 clients without any problem.
So better use nio. there is a very good book from Oreilly publication for java nio.
I have used java nio (socket channel)
Thanks
Sunil Kumar Sahoo