We Keep Coding

java.net.Socket TCP keep-alive usage

How to use java.net.Socket setKeepAlive(boolean b) API?
I am hosting a simple server using Socket.
The clients can connect and send data. I will not close the connection unless the client sends end of stream. The client can continue to hold a connection for any length of time. Data transfer will be going on, but suddenly I see that there is a connection loss without any logs whatsoever. Once the client is restarted (not server), the transmission begins again.
I am not setting any timeout as by default it is infinity. I want to understand what setKeepAlive actually does so that I will be able to decide whether I need to set that to true.
Can someone help me understand the same? thanks.

As the documentation for setKeepAlive() says, it will enable (or disable) the SO_KEEPALIVE option on the socket.
When the keepalive option is set for a TCP socket and no data has been exchanged across the socket in either direction for 2 hours (NOTE: the actual value is implementation dependent), TCP automatically sends a keepalive probe to the peer. This probe is a TCP segment to which the peer must respond. One of three responses is expected: 1. The peer responds with the expected ACK. The application is not notified (since everything is OK). TCP will send another probe following another 2 hours of inactivity. 2. The peer responds with an RST, which tells the local TCP that the peer host has crashed and rebooted. The socket is closed. 3. There is no response from the peer. The socket is closed. The purpose of this option is to detect if the peer host crashes. Valid only for TCP socket: SocketImpl
Here is another reference explaining the SO_KEEPALIVE option.
Note that in networking, connections can be lost at any time for a myriad of reasons. If the connection traverses a NAT router, the entry in the NAT table could expire (when the connection is idle) and the connection is lost due to that. The client could cease to function, or be suspended (especially laptops and mobile devices), or a cable could be disconnected, or WiFi (or cellular) signal could be interfered with, or ... the list can go on. Your server needs to be written to cope gracefully with loss of connection.

I face the similar problem in my case
In JAVA Socket – TCP connections are managed on the OS level,
java.net.Socket does not provide any in-built function to set timeouts for keepalive packet on a per-socket level.
But we can enable keepalive option for java socket but it takes 2 hours 11 minutes (7200 sec) by default to process after a stale tcp connections.
This cause connection will be availabe for very long time before purge.
So we found some solution to use Java Native Interface (JNI) that call native code(c++) to configure these options.
****Windows OS****
In windows operating system keepalive_time & keepalive_intvl can be configurable but tcp_keepalive_probes cannot be change.By default, when a TCP socket is initialized sets the keep-alive timeout to 2 hours and the keep-alive interval to 1 second. The default system-wide value of the keep-alive timeout is controllable through the KeepAliveTime registry setting which takes a value in milliseconds.
On Windows Vista and later, the number of keep-alive probes (data retransmissions) is set to 10 and cannot be changed.
On Windows Server 2003, Windows XP, and Windows 2000, the default setting for number of keep-alive probes is 5. The number of keep-alive probes is controllable.
For windows Winsock IOCTLs library is used to configure the tcp-keepalive parameters.
int WSAIoctl(
SocketFD, // descriptor identifying a socket
SIO_KEEPALIVE_VALS, // dwIoControlCode
(LPVOID) lpvInBuffer, // pointer to tcp_keepalive struct
(DWORD) cbInBuffer, // length of input buffer
NULL, // output buffer
0, // size of output buffer
(LPDWORD) lpcbBytesReturned, // number of bytes returned
NULL, // OVERLAPPED structure
NULL // completion routine
);
Linux OS
Linux has built-in support for keepalive which is need to be enabling TCP/IP networking in order to use it. Programs must request keepalive control for their sockets using the setsockopt interface.
int setsockopt(int socket, int level, int optname,
const void *optval, socklen_t optlen)
Each client socket will be created using java.net.Socket. File descriptor ID for each socket will retrieve using java reflection.

Spymemcached and Connection Failures

I though Spymemcached does attempt to reestablish connection to the server when this connection get lost.
But I am seeing something different; wondering what I misunderstand or what I do wrong. Here is some sample code:
MemcachedClient c=new MemcachedClient(AddrUtil.getAddresses("server:11211"));
while(true)
try {
System.out.println(c.get("hatsts"));
Thread.sleep(10000);
} catch(Exception e) {
e.printStackTrace();
}
It runs initially without problem. Then I pull the network plug. Subsequently, the client detects a network failure and throws following exception:
net.spy.memcached.OperationTimeoutException: Timeout waiting for value
But then, when i re-establish the network, the client does not recover and continues throwing the exception; even after 5 min. I tried SpyMemcached 2.10.6 and 2.9.0.
What am I missing?

The problem here is that because you pulled the network cable the tcp socket on you client still thinks the connection is valid. The tcp keepalive varies from operating system to operating system and can be as high as 30 minutes. As a result the application (in this case Spymemcached) is not notified that the connection is no longer valid by the tcp layer and therefore Spymemcached doesn't try to reconnect.
The way Spymemcached detects this situation is by counting the amount of consecutive operation timeouts. The last time I checked the default value was 99. Once this many ops time out then Spymemcached will reconnect. You can change this value in the ConnectionFactory if you want to set it to some other value. There's a function called getContinuousTimeout() which is where the Spymemcached gets 99 from by default. You can construct your own ConnectionFactory with the ConnectionFactoryBuilder.
Hopefully this is enough information to answer your question and get you going in the right direction. If not let me know and I can add some more details.

Java outgoing TCP connection failover based on multiple DNS results

If I make a connection using new Socket("unit.domain.com", 100) and the unit.domain.com DNS record has multiple IP addresses in the A record.. In the event of a failed connection, Does Java automatically connect to one of the other addresses in the list like the browser does? or does that have to be implemented manually?

No!
Creating a socket via new Socket(String, int) results in a resolving like that
addr = InetAddress.getByName(hostname);
which is a shortcut for
return InetAddress.getAllByName(host)[0];
The address resolution is performed in the Socket c-tor.
If you have to reconnect (failover) use the result returned by InetAddress.getAllByName(host), randomize (or use round-robin) and connect to the necessary addresses.
Edit: also if you are going to need to connect with some likely failure, you'd be better off using connect method of the Socket class with a timeout. Also make sure you close even failed sockets (and esp. channels) since they may leak a FD on *Nix.

Forcing socket.connect to wait a specific time before it decides a connection is unavailable

I'm issuing a socket connection, using the following snippet
Socket socket = new Socket();
InetSocketAddress endPoint = new InetSocketAddress("localhost", 1234);
try
{
socket.connect(endPoint, 30000);
}
catch (IOException e)
{
e.printStackTrace();
// Logging
}
The endpoint it is trying to connect to is offline, what I want it to do is to attempt to connect, and using the 30000ms timeout, wait for that period of time before it concludes a result
Currently, that 30000 parameter doesn't seem to be applied, as from the timestamp on my logging it appears that it is determining within 1 second that a connection failed.
How can I force the connect to wait for a set amount of time before giving up?
13:13:57,685 6235 DEBUG [Thread-7] - Unable to connect to [localhost:1234]
13:13:58,685 7235 DEBUG [Thread-7] - Unable to connect to [localhost:1234]
13:13:59,695 8245 DEBUG [Thread-7] - Unable to connect to [localhost:1234]
13:14:00,695 9245 DEBUG [Thread-7] - Unable to connect to [localhost:1234]
EDIT : The API does state Connects this socket to the server with a specified timeout value. A timeout of zero is interpreted as an infinite timeout. The connection will then block until established or an error occurs. however it appears I'm not experiencing such behaviour, or am not catering to it, most likely the latter

What you're getting here is correct. connect won't sit on a socket waiting until it sees a server, it will attempt to connect and wait for a response. if there is nothing to connect to, it returns. if there is something to connect to, it will wait timeout seconds for a response and fail if none is received.

You need to distinguish among several possible exception conditions.
ConnectException with the text 'connection refused', which means the host was up and reachable and nothing was listening at the port. This happens very quickly and cannot be subjected to a timeout.
NoRouteToHostException: this indicates a connectivity issue. Again it happens immediately and cannot be subjected to a timeout.
UnknownHostException: the host names cannot be resolved via DNS. This happens immediately, or rather after a generally short DNS delay, and cannot be subjected to a timeout.
ConnectException with any other text: this can indicate a failure to respond by the target system. Usually happens when firewalls are present. Can be subjected to a timeout.
You are doing the correct thing by calling Socket.connect() with a timeout parameter. If you don't do this, or if you specify a zero timeout, the default system timeout is used, which is of the order of 60-75 seconds depending on the platform. This is contrary to the Javadoc's statement about an 'infinite timeout', which is not correct. Also you cannot increase the timeout beyond this limit via Socket.connect() witha a timeout parameter. Alternatively you can use java.nio socket channels in non-blocking mode with a select() to administer the timeout for you, but you still can't increase the timeout beyond the platform default via this or any other method.

When the timeout occurs, a SocketTimeoutException exception is thrown which you do not catch and log. The IOException is fired when "an error occurs during the connection". The timeout is never applied because there's an error beforehand.
Edit: Just to clarify: TCP/IP as a suite has many specifics that could prevent a packet from reaching it's desired outcome (a SYN/ACK packet). If a computer responds to your SYN packet by an informing your application that the port is closed (i.e. there's no application running/listening there), it would fire an exception telling you that it is impossible to connect to that port. If you wish to send and re-send SYN packets either way with the knowledge that an application will come online listening on that port, this is done on a different network layer (and, as far as I know, is not accessible with Java out-of-the-box).

Try scocket.setSoTimeout(timeout) before connecting.

Why would a "java.net.ConnectException: Connection timed out" exception occur when URL is up?

I'm getting a ConnectException: Connection timed out with some frequency from my code. The URL I am trying to hit is up. The same code works for some users, but not others. It seems like once one user starts to get this exception they continue to get the exception.
Here is the stack trace:
java.net.ConnectException: Connection timed out
Caused by: java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.Socket.connect(Socket.java:516)
at java.net.Socket.connect(Socket.java:466)
at sun.net.NetworkClient.doConnect(NetworkClient.java:157)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:365)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:477)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:214)
at sun.net.www.http.HttpClient.New(HttpClient.java:287)
at sun.net.www.http.HttpClient.New(HttpClient.java:299)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:796)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:748)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:673)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:840)
Here is a snippet from my code:
URLConnection urlConnection = null;
OutputStream outputStream = null;
OutputStreamWriter outputStreamWriter = null;
InputStream inputStream = null;
try {
URL url = new URL(urlBase);
urlConnection = url.openConnection();
urlConnection.setDoOutput(true);
outputStream = urlConnection.getOutputStream(); // exception occurs on this line
outputStreamWriter = new OutputStreamWriter(outputStream);
outputStreamWriter.write(urlString);
outputStreamWriter.flush();
inputStream = urlConnection.getInputStream();
String response = IOUtils.toString(inputStream);
return processResponse(urlString, urlBase, response);
} catch (IOException e) {
throw new Exception("Error querying url: " + urlString, e);
} finally {
IoUtil.close(inputStream);
IoUtil.close(outputStreamWriter);
IoUtil.close(outputStream);
}

Connection timeouts (assuming a local network and several client machines) typically result from
a) some kind of firewall on the way that simply eats the packets without telling the sender things like "No Route to host"
b) packet loss due to wrong network configuration or line overload
c) too many requests overloading the server
d) a small number of simultaneously available threads/processes on the server which leads to all of them being taken. This happens especially with requests that take a long time to run and may combine with c).

If the URL works fine in the web browser on the same machine, it might be that the Java code isn't using the HTTP proxy the browser is using for connecting to the URL.

The error message says it all: your connection timed out. This means your request did not get a response within some (default) timeframe. The reasons that no response was received is likely to be one of:
a) The IP/domain or port is incorrect
b) The IP/domain or port (i.e service) is down
c) The IP/domain is taking longer than your default timeout to respond
d) You have a firewall that is blocking requests or responses on whatever port you are using
e) You have a firewall that is blocking requests to that particular host
f) Your internet access is down
g) Your live-server is down i.e in case of "rest-API call".
Note that firewalls and port or IP blocking may be in place by your ISP

I'd recommend raising the connection timeout time before getting the output stream, like so:
urlConnection.setConnectTimeout(1000);
Where 1000 is in milliseconds (1000 milliseconds = 1 second).

try to do the Telnet to see any firewall issue
perform tracert/traceroute to find number of hops

I solved my problem with:
System.setProperty("https.proxyHost", "myProxy");
System.setProperty("https.proxyPort", "80");
or http.proxyHost...

Why would a “java.net.ConnectException: Connection timed out”
exception occur when URL is up?
Because the URLConnection (HttpURLConnection/HttpsURLConnection) is erratic. You can read about this here and here.
Our solution were two things:
a) set the ContentLength via setFixedLengthStreamingMode
b) catch any TimeoutException and retry if it failed.

This can be a IPv6 problem (the host publishes an IPv6 AAAA-Address and the users host thinks it is configured for IPv6 but it is actually not correctly connected). This can also be a network MTU problem, a firewall block, or the target host might publish different IP addresses (randomly or based on originators country) which are not all reachable. Or similliar network problems.
You cant do much besides setting a timeout and adding good error messages (especially printing out the hosts' resolved address). If you want to make it more robust add retry, parallel trying of all addresses and also look into name resolution caching (positive and negative) on the Java platform.

There is a possibility that your IP/host are blocked by the remote host, especially if it thinks you are hitting it too hard.

The reason why this happened to me was that a remote server was allowing only certain IP addressed but not its own, and I was trying to render the images from the server's URLs... so everything would simply halt, displaying the timeout error that you had...
Make sure that either the server is allowing its own IP, or that you are rendering things from some remote URL that actually exists.