We Keep Coding

Java HttpRequest timeout unexpectedly throwing HttpConnectTimeoutException

Using the new java.net.http package released with JDK 11, an HttpRequest has been assembled with a deliberately low response timeout:
HttpRequest.Builder builder = HttpRequest.newBuilder(getEndpointUri());
addRequestHeaders(builder);
builder.POST(HttpRequest.BodyPublishers.ofString(rawXml));
builder.timeout(Duration.ofMillis(1));
HttpRequest httpRequest = builder.build();
The aim is to test that HttpTimeoutException outcomes are handled correctly, but unexpectedly this response timeout value is leading to an HttpConnectionTimeoutException, which is being caught by this code:
try {
HttpResponse<InputStream> httpResponse = completableExchange.join();
} catch (CompletionException ce) {
if (ce.getCause() instanceof HttpConnectTimeoutException) {
System.out.println("Connection timeout occurred!");
} else {
throw ce;
}
}
This means that a response timeout is causing the code to act as though a connection timeout has occurred. To the best of my understanding, the connection timeout and response timeout should be separate concepts, which should be possible to catch and handle separately.
The stack trace attached to the HttpConnectionTimeoutException looks like this:
java.net.http.HttpConnectTimeoutException: HTTP connect timed out
at java.net.http/jdk.internal.net.http.ResponseTimerEvent.handle(ResponseTimerEvent.java:68)
at java.net.http/jdk.internal.net.http.HttpClientImpl.purgeTimeoutsAndReturnNextDeadline(HttpClientImpl.java:1248)
at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:877)
Caused by: java.net.ConnectException: HTTP connect timed out
at java.net.http/jdk.internal.net.http.ResponseTimerEvent.handle(ResponseTimerEvent.java:69)
... 2 more
Am I misunderstanding the timeout concepts? Does the HttpRequest timeout value simply provide an alternative to the default of the HttpClient timeout value? Is there a reliable way to catch connection and response timeout as distinct events?
For what it's worth, the Javadoc for HttpRequest.Builder.timeout(Duration) says the following:
Sets a timeout for this request. If the response is not received within the specified timeout then an HttpTimeoutException is thrown from HttpClient::send or HttpClient::sendAsync completes exceptionally with an HttpTimeoutException. The effect of not setting a timeout is the same as setting an infinite Duration, ie. block forever.
To make things confusing, HttpConnectionTimeoutException is a subclass of HttpTimeoutException so technically the contract of the timeout(Duration) method is being satisfied. But this seems unhelpful.
(Before you ask: yes, the value passed to HttpRequest.Builder.timeout(Duration) is the deciding factor in whether or not an exception is thrown. So the exception is not based on the connection timeout value being used to create the HttpClient instance.)

IIRC you will get a HttpConnectionTimeoutException if the connection is not connected at the time the timeout is raised, or if the connect timeout is raised before the connection has finished connecting.
When sending a request - the underlying connection might already be connected or not - depending on whether a suitable existing connection was found in the pool. The request timeout starts immediately - independently of the state of the underlying connection. If the underlying connection is not connected yet, and the request timeout expires before it gets connected, then you will get a HttpConnectionTimeoutException because the connection could not be connected within the time allocated for the response to the request to be delivered. You could see it as the request timeout clipping the connect timeout.
Do you have any specific use case in mind for distinguishing the two cases:
HttpConnectionTimeoutException is raised because the connection could not be connected within the time specified by the connection timeout,
HttpConnectionTimeoutException is raised because the request timeout expired before the connection could be connected?

How to re-establish lost KDB connection in Java

In my Java application I am establishing KDB connection using
private void initConnection() throws KException, IOException {
conn = new c(host, port);
conn.tz = TimeZone.getTimeZone(CONNECTION_TIMEZONE);
}
Then I use conn for inserting data asynchronously in tables successfully.
KDB server is restarted once in week and at that time as connection is reset by server I am getting an exception java.net.SocketException.
Using the above exception I can again re-establish lost connection but this is not a decent solution as I lose batch etc. and I don’t want connection exception to be thrown at the time of insertion.
Instead, is there a call-back/listener/Exception method in KDB API where API can inform java application upfront if a connection is closed by server / lost?

Why don't you just do a
SELECT 1;
on your connection.
If it's open it'll run.
If it's closed it should throw an error because of the failing network connection.
You'd run this test only in the moments that your kbd server is scheduled for restart.
Personally, on the scheduled moments of restarting the kbd database I'd dump the data into an sqlite database for holding, and then when the frame for rebooting is over and connection is re-established dump the contents of the sqlite database into the kbd database.

If connection is lost, I would expect conn object to throw IOException or its successor, when you try to hit KDB.
Ie, you should catch IOException and
invoke initConnection method to re-establish connection
re-execute query
Would be nice to set reasonable threshold of retries either. Otherwise, if KDb is down, the code will keep trying to connect to it.

This answer is related (and possibly adds on) to Anton Dovzhenko's answer above, but hopefully the below helps you as well.
If we take a look at kx's documentation for reconnecting to a q process automatically, we can observe the following code snippet from ReconnectionExample.java below.
//initiate reconnect loop (possibly within a catch block).
while (true) {
try {
System.err.println("Connection failed - retrying..");
//Wait a bit before trying to reconnect
Thread.sleep(5000);
qConnection = qConnFactory.getQConnection();
System.out.println("Connection re-established! Resuming..");
//Exit loop
break;
} catch (IOException | KException e1) {
//resume loop if it fails
continue;
}
…
}
That would be one way to try and reconnect; within a catch block. However, to answer your questions in greater detail:
To alert the rest of your application, do so in the catch block. One way to go about doing this is to invoke another call back method to alert that the connection is now down.
To reconnect, do this as well in your catch block as well. Be careful to only do this at a specified interval, though. The reason why I mention that is because when a Java application tries to reconnect, it will create a new c object.
This should help your application handle reconnecting more efficiently, and notifying any other dependencies appropriately.

Spymemcached and Connection Failures

I though Spymemcached does attempt to reestablish connection to the server when this connection get lost.
But I am seeing something different; wondering what I misunderstand or what I do wrong. Here is some sample code:
MemcachedClient c=new MemcachedClient(AddrUtil.getAddresses("server:11211"));
while(true)
try {
System.out.println(c.get("hatsts"));
Thread.sleep(10000);
} catch(Exception e) {
e.printStackTrace();
}
It runs initially without problem. Then I pull the network plug. Subsequently, the client detects a network failure and throws following exception:
net.spy.memcached.OperationTimeoutException: Timeout waiting for value
But then, when i re-establish the network, the client does not recover and continues throwing the exception; even after 5 min. I tried SpyMemcached 2.10.6 and 2.9.0.
What am I missing?

The problem here is that because you pulled the network cable the tcp socket on you client still thinks the connection is valid. The tcp keepalive varies from operating system to operating system and can be as high as 30 minutes. As a result the application (in this case Spymemcached) is not notified that the connection is no longer valid by the tcp layer and therefore Spymemcached doesn't try to reconnect.
The way Spymemcached detects this situation is by counting the amount of consecutive operation timeouts. The last time I checked the default value was 99. Once this many ops time out then Spymemcached will reconnect. You can change this value in the ConnectionFactory if you want to set it to some other value. There's a function called getContinuousTimeout() which is where the Spymemcached gets 99 from by default. You can construct your own ConnectionFactory with the ConnectionFactoryBuilder.
Hopefully this is enough information to answer your question and get you going in the right direction. If not let me know and I can add some more details.

Forcing socket.connect to wait a specific time before it decides a connection is unavailable

I'm issuing a socket connection, using the following snippet
Socket socket = new Socket();
InetSocketAddress endPoint = new InetSocketAddress("localhost", 1234);
try
{
socket.connect(endPoint, 30000);
}
catch (IOException e)
{
e.printStackTrace();
// Logging
}
The endpoint it is trying to connect to is offline, what I want it to do is to attempt to connect, and using the 30000ms timeout, wait for that period of time before it concludes a result
Currently, that 30000 parameter doesn't seem to be applied, as from the timestamp on my logging it appears that it is determining within 1 second that a connection failed.
How can I force the connect to wait for a set amount of time before giving up?
13:13:57,685 6235 DEBUG [Thread-7] - Unable to connect to [localhost:1234]
13:13:58,685 7235 DEBUG [Thread-7] - Unable to connect to [localhost:1234]
13:13:59,695 8245 DEBUG [Thread-7] - Unable to connect to [localhost:1234]
13:14:00,695 9245 DEBUG [Thread-7] - Unable to connect to [localhost:1234]
EDIT : The API does state Connects this socket to the server with a specified timeout value. A timeout of zero is interpreted as an infinite timeout. The connection will then block until established or an error occurs. however it appears I'm not experiencing such behaviour, or am not catering to it, most likely the latter

What you're getting here is correct. connect won't sit on a socket waiting until it sees a server, it will attempt to connect and wait for a response. if there is nothing to connect to, it returns. if there is something to connect to, it will wait timeout seconds for a response and fail if none is received.

You need to distinguish among several possible exception conditions.
ConnectException with the text 'connection refused', which means the host was up and reachable and nothing was listening at the port. This happens very quickly and cannot be subjected to a timeout.
NoRouteToHostException: this indicates a connectivity issue. Again it happens immediately and cannot be subjected to a timeout.
UnknownHostException: the host names cannot be resolved via DNS. This happens immediately, or rather after a generally short DNS delay, and cannot be subjected to a timeout.
ConnectException with any other text: this can indicate a failure to respond by the target system. Usually happens when firewalls are present. Can be subjected to a timeout.
You are doing the correct thing by calling Socket.connect() with a timeout parameter. If you don't do this, or if you specify a zero timeout, the default system timeout is used, which is of the order of 60-75 seconds depending on the platform. This is contrary to the Javadoc's statement about an 'infinite timeout', which is not correct. Also you cannot increase the timeout beyond this limit via Socket.connect() witha a timeout parameter. Alternatively you can use java.nio socket channels in non-blocking mode with a select() to administer the timeout for you, but you still can't increase the timeout beyond the platform default via this or any other method.

When the timeout occurs, a SocketTimeoutException exception is thrown which you do not catch and log. The IOException is fired when "an error occurs during the connection". The timeout is never applied because there's an error beforehand.
Edit: Just to clarify: TCP/IP as a suite has many specifics that could prevent a packet from reaching it's desired outcome (a SYN/ACK packet). If a computer responds to your SYN packet by an informing your application that the port is closed (i.e. there's no application running/listening there), it would fire an exception telling you that it is impossible to connect to that port. If you wish to send and re-send SYN packets either way with the knowledge that an application will come online listening on that port, this is done on a different network layer (and, as far as I know, is not accessible with Java out-of-the-box).

Try scocket.setSoTimeout(timeout) before connecting.

Why would a "java.net.ConnectException: Connection timed out" exception occur when URL is up?

I'm getting a ConnectException: Connection timed out with some frequency from my code. The URL I am trying to hit is up. The same code works for some users, but not others. It seems like once one user starts to get this exception they continue to get the exception.
Here is the stack trace:
java.net.ConnectException: Connection timed out
Caused by: java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.Socket.connect(Socket.java:516)
at java.net.Socket.connect(Socket.java:466)
at sun.net.NetworkClient.doConnect(NetworkClient.java:157)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:365)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:477)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:214)
at sun.net.www.http.HttpClient.New(HttpClient.java:287)
at sun.net.www.http.HttpClient.New(HttpClient.java:299)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:796)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:748)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:673)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:840)
Here is a snippet from my code:
URLConnection urlConnection = null;
OutputStream outputStream = null;
OutputStreamWriter outputStreamWriter = null;
InputStream inputStream = null;
try {
URL url = new URL(urlBase);
urlConnection = url.openConnection();
urlConnection.setDoOutput(true);
outputStream = urlConnection.getOutputStream(); // exception occurs on this line
outputStreamWriter = new OutputStreamWriter(outputStream);
outputStreamWriter.write(urlString);
outputStreamWriter.flush();
inputStream = urlConnection.getInputStream();
String response = IOUtils.toString(inputStream);
return processResponse(urlString, urlBase, response);
} catch (IOException e) {
throw new Exception("Error querying url: " + urlString, e);
} finally {
IoUtil.close(inputStream);
IoUtil.close(outputStreamWriter);
IoUtil.close(outputStream);
}

Connection timeouts (assuming a local network and several client machines) typically result from
a) some kind of firewall on the way that simply eats the packets without telling the sender things like "No Route to host"
b) packet loss due to wrong network configuration or line overload
c) too many requests overloading the server
d) a small number of simultaneously available threads/processes on the server which leads to all of them being taken. This happens especially with requests that take a long time to run and may combine with c).

If the URL works fine in the web browser on the same machine, it might be that the Java code isn't using the HTTP proxy the browser is using for connecting to the URL.

The error message says it all: your connection timed out. This means your request did not get a response within some (default) timeframe. The reasons that no response was received is likely to be one of:
a) The IP/domain or port is incorrect
b) The IP/domain or port (i.e service) is down
c) The IP/domain is taking longer than your default timeout to respond
d) You have a firewall that is blocking requests or responses on whatever port you are using
e) You have a firewall that is blocking requests to that particular host
f) Your internet access is down
g) Your live-server is down i.e in case of "rest-API call".
Note that firewalls and port or IP blocking may be in place by your ISP

I'd recommend raising the connection timeout time before getting the output stream, like so:
urlConnection.setConnectTimeout(1000);
Where 1000 is in milliseconds (1000 milliseconds = 1 second).

try to do the Telnet to see any firewall issue
perform tracert/traceroute to find number of hops

I solved my problem with:
System.setProperty("https.proxyHost", "myProxy");
System.setProperty("https.proxyPort", "80");
or http.proxyHost...

Why would a “java.net.ConnectException: Connection timed out”
exception occur when URL is up?
Because the URLConnection (HttpURLConnection/HttpsURLConnection) is erratic. You can read about this here and here.
Our solution were two things:
a) set the ContentLength via setFixedLengthStreamingMode
b) catch any TimeoutException and retry if it failed.

This can be a IPv6 problem (the host publishes an IPv6 AAAA-Address and the users host thinks it is configured for IPv6 but it is actually not correctly connected). This can also be a network MTU problem, a firewall block, or the target host might publish different IP addresses (randomly or based on originators country) which are not all reachable. Or similliar network problems.
You cant do much besides setting a timeout and adding good error messages (especially printing out the hosts' resolved address). If you want to make it more robust add retry, parallel trying of all addresses and also look into name resolution caching (positive and negative) on the Java platform.

There is a possibility that your IP/host are blocked by the remote host, especially if it thinks you are hitting it too hard.

The reason why this happened to me was that a remote server was allowing only certain IP addressed but not its own, and I was trying to render the images from the server's URLs... so everything would simply halt, displaying the timeout error that you had...
Make sure that either the server is allowing its own IP, or that you are rendering things from some remote URL that actually exists.