In the docs (Tomcat 7 Config), it is written:
The number of milliseconds this Connector will wait, after accepting a connection, for the request URI line to be presented. Use a value of -1 to indicate no (i.e. infinite) timeout. The default value is 60000 (i.e. 60 seconds) but note that the standard server.xml that ships with Tomcat sets this to 20000 (i.e. 20 seconds). Unless disableUploadTimeout is set to false, this timeout will also be used when reading the request body (if any).
When a client sends request to a server, it will take N milliseconds to establish a connection. If this N exceeds the connection timeout that is set on the client's end, the request will fail in the client as expected.
I'm not able to understand what Tomcat's connectionTimeout does differently. Specifically, what does "after accepting a connection, for the request URI line to be presented" means ?
The connectionTimeout is the limit of time after which the server will automatically close the connection with the client not the other way around. It is a way to limit the impact of a Denial Of Service Attack. Indeed a typical way to do a DOS Attack is to launch several requests on a given server, and each request will last forever making the server waits for nothing and filling up its pool of threads such that the Server won't be able to accept any new requests. Thanks to this timeout, after x milliseconds it will ignore the request considering it as a potential attack.
Here is an interesting discussion on globally the same subject that goes a little bit deeper.
Related
I have a very basic spring boot 2.2.4 application that queries a downstream system using webclient with a blocking call. I am not doing any configuration of the webclient (setting timeouts, etc.), just using it "out of the box".
What I find is that the response time of the webclient call is either below 3 seconds or precisely 45 seconds, which I find very strange. Why if the response is slow, it is always 45 seconds?
The only reference to the 45 seconds I could find comes from the Reactor Netty documentation:
4.6. Connection Pool
By default, the TCP client uses a “fixed” connection pool with 500 as the maximum number of the channels and 45s as the acquisition timeout. This means that the implementation creates a new channel if someone tries to acquire a channel but none is in the pool. When the maximum number of the channels in the pool is reached, new tries to acquire a channel are delayed until a channel is returned to the pool again. The implementation uses FIFO order for channels in the pool. By default, there is no idle time specified for the channels in the pool.
Does anybody have any suggestion as to why my slow webclient calls always take 45 seconds to complete?
This is happening because of the fixed pool size which is 500. If all these connections (channels) are currently in use by existing requests, a new request for a connection would be queued until one of the 500 connections becomes free. If WebClient isn't able to acquire a connection within 45 seconds for a new request (since all of the 500 channels are still blocked by existing requests), it fails with an AcquireTimeout Exception. I'm assuming the ones that finished before 3 seconds are success ones while the 45 second ones are failures. Depending on the Throughput of your application, you can adjust the pool size accordingly.
In the past, when I've seen delays like this on successful connections, it was caused like this:
You try to connect with a domain name, so the client calls DNS first to get an address;
DNS returns address records for both IPv4 and IPv6
The client tries IPv6 first, but IPv6 is not properly configured on the network, and this fails after the IPv6 connection timeout, which can be in the 45s range.
The client then tries IPv4, which succeeds.
Whether or not this happens to you depends on your OS+version, your java version, and the configuration of your network.
Try connecting with in IPv4 IP address, like http://192.168.10.1/the/rest. If it's always fast then you probably had a problem like above. Unfortunately you can't usually connect this way with HTTPS, but depending on your OS you could try stuffing the correct v4 address in /etc/hosts.
Look here for more: https://serverfault.com/questions/548777/how-to-prevent-delays-associated-with-ipv6-aaaa-records
I think this is related to this netty issue:
https://github.com/reactor/reactor-netty/issues/1012
I will have a look at that first now.
Thanks for all the replies!
We have a Java 8 application served by an Apache Tomcat 8 behind an Apache server, which is requesting multiple webservices in parallel using CXF. From time to time, there's one of them which lasts exactly 3 more seconds than the rest (which should be about 500ms only).
I've activated CXF debug, and now I have the place inside CXF where the 3 seconds are lost:
14/03/2018 09:20:49.061 [pool-838-thread-1] DEBUG o.a.cxf.transport.http.HTTPConduit - No Trust Decider for Conduit '{http://ws.webapp.com/}QueryWSImplPort.http-conduit'. An affirmative Trust Decision is assumed.
14/03/2018 09:20:52.077 [pool-838-thread-1] DEBUG o.a.cxf.transport.http.HTTPConduit - Sending POST Message with Headers to http://172.16.56.10:5050/services/quertServices Conduit :{http://ws.webapp.com/}QueryWSImplPort.http-conduit
As you could see, there're three seconds between these two lines. When the request is ok, it usually takes 0ms in between these two lines.
I've been looking into the CXF code, but no clue about the reason of these 3 secs...
The server application (which is also maintained by us), is served from another Apache Tomcat 6.0.49, which is behind another Apache server. The thing is that it seems that the server's Apache receives the request after the 3 seconds.
Anyone could help me?
EDIT:
We've monitored the server's send/received packages and it seems that the client's server is sending a negotiating package at the time it should, while the server is replying after 3 seconds.
These are the packages we've found:
481153 11:31:32 14/03/2018 2429.8542795 tomcat6.exe SOLTESTV010 SOLTESTV002 TCP TCP:Flags=CE....S., SrcPort=65160, DstPort=5050, PayloadLen=0, Seq=2858646321, Ack=0, Win=8192 ( Negotiating scale factor 0x8 ) = 8192 {TCP:5513, IPv4:62}
481686 11:31:35 14/03/2018 2432.8608381 tomcat6.exe SOLTESTV002 SOLTESTV010 TCP TCP:Flags=...A..S., SrcPort=5050, DstPort=65160, PayloadLen=0, Seq=436586023, Ack=2858646322, Win=8192 ( Negotiated scale factor 0x8 ) = 2097152 {TCP:5513, IPv4:62}
481687 11:31:35 14/03/2018 2432.8613607 tomcat6.exe SOLTESTV010 SOLTESTV002 TCP TCP:Flags=...A...., SrcPort=65160, DstPort=5050, PayloadLen=0, Seq=2858646322, Ack=436586024, Win=256 (scale factor 0x8) = 65536 {TCP:5513, IPv4:62}
481688 11:31:35 14/03/2018 2432.8628380 tomcat6.exe SOLTESTV010 SOLTESTV002 HTTP HTTP:Request, POST /services/consultaServices {HTTP:5524, TCP:5513, IPv4:62}
So, it seems is the server's Tomcat is the one which is blocked with something. Any clue?
EDIT 2:
Although that happened yesterday (the first server waiting 3s for the ack of the second), this is not the most common scenario. What it usually happens is what I described at the beginning (3 seconds between the two CXF's logs and the server receiving any request from the first one after 3 seconds.
There has been some times when the server (the one which receives the request), hangs for 3 seconds. For instance:
Server 1 sends 5 requests at the same time (suppossedly) to server 2.
Server 2 receives 4 of them, in that same second, and start to process them.
Server 2 finish processing 2 of those 4 requests in 30ms and replies to server 1.
More or less at this same second, there's nothing registered in the application logs.
After three seconds, logs are registered again, and the server finish to process the remaining 2 requests. So, although the process itself is about only some milliseconds, the response_time - request_time is 3 seconds and a few ms.
At this same time, the remaining request (the last one from the 5 request which were sent), is registered in the network monitor and is processed by the application in just a few milliseconds. However, the global processing time is anything more than 3s, as it has reached the server 3 seconds after being sent.
So there's like a hang in the middle of the process. 2 requests were successfully processed before this hang and replied in just a fraction of a second. 2 other request lasted a little bit more, the hang happened, and ended with a processing time of 3 seconds. The last one, reached the server just when the hang happened, so it didn't get into the application after the hang.
It sounds like a gc stop the world... but we have analyzed gc.logs and there's nothing wrong with that... could there be any other reason?
Thanks!
EDIT 3:
Looking at the TCP flags like the one I pasted last week, we've noticed that there are lots of packets with the CE flag, that is a notification of TCP congestion. We're not network experts, but have found that this could deal us to a 3 seconds of delay before the packet retransmission...
could anyone give us some help about that?
Thanks. Kind regards.
Finally, it was everything caused by the network congestion we discovered looking at the TCP flags. Our network admins has been looking at the problem, trying to reduce the congestion, reducing the timeout to retransmit.
The thing is that it seems that the server's Apache receives the
request after the 3 seconds.
How do you figure this out ? If you're looking at Apache logs, you can be misleaded by wrong time stamps.
I first thought that your Tomcat 6 takes 3 seconds to answer instead of 0 to 500ms, but from the question and the comments, it is not the case.
Hypothesis 1 : Garbage Collector
The GC is known for introducing latency.
Highlight the GC activity in your logs by using the GC verbosity parameters. If it is too difficult to correlate, you can use the jstat command with the gcutil option and you can compare easily with the Tomcat's log.
Hypothesis 2 : network timeout
Although 3s is a very short time (in comparison with the 21s TCP default timeout on Windows for example), it could be a timeout.
To track the timeouts, you can use the netstat command. With netstat -an , look for the SYN_SENT connections, and with netstat -s look for the error counters. Please check if there is any network resource that must be resolved or accessed in this guilty webservice caller.
I am making an application where the client sends a message to the server and then waits for 5 seconds (lets assume) for the server to respond and if there is no return message, it retries again. If the server responds with the message then the client process it. This goes on in loop and again happens after sometime.
For this purpose I was thinking to use setSoTimeout(time) on the Client Socket but after reading the javadoc and a lot of explanations on the internet I am confused as to whether this approach is right.
What I read on the internet
(1) If I use setSoTimeout on the socket then it gives the timeout for the duration in which the connection needs to be established and if it is not established then it retries to establish the connection for the given time.
(2) If I use setSoTimeout on the socket then it waits for incoming messages for the specified time interval and if no message is received then it stops waiting.
My questions are -
(1) Which of the above are true ?
(2) If the second statement is true, then can I use it for my implementation ?
(3) If the second statement is true, when does the timeout timer kickoff exactly ? Is it when I declare the socket and set the timeout period on it or is it when I send the message ?
If either of the explanation don't apply to my case then what is it that I should do to wait for a fixed interval of time on the client side for the server to reply ? If the reply does come I should process it and move on and redo the same process. If the reply doesn't come I should move ahead and redo the whole process again.
(1) If I use setSoTimeout() on the socket then it gives the timeout for the duration in which the connection needs to be established and if it is not established then it retries to establish the connection for the given time.
This is incorrect. setSoTimeout() does not cause re-establishment of the connection at all, let alone 'for the given time'.
(2) If I use setSoTimeout() on the socket then it waits for incoming messages for the specified time interval and if no message is received then it stops waiting.
This is slightly more accurate, but there is no such thing as a message in TCP.
The correct explanation is that it blocks for up to the specified timeout for at least one byte to arrive. If nothing arrives within the timeout, a SocketTimeoutException is thrown.
(1) Which of the above are true?
Neither.
(2) If the second statement is true, then can I use it for my implementation?
It isn't, so the second part doesn't apply, but if any statement is true you can use it as part of your implementation. You don't have to ask.
(3) If the second statement is true, when does the timeout timer kickoff exactly?
When you call read().
Is it when I declare the socket and set the timeout period on it or is it when I send the message?
Neither.
If either of the explanation don't apply to my case then what is it that I should do to wait for a fixed interval of time on the client side for the server to reply?
Set a read timeout.
This question already has answers here:
What is the difference between connection and read timeout for sockets?
(2 answers)
Closed 8 years ago.
Just curiosity. Is there a good reason why the class URLConnection needs to have two different timeouts?
The connectTimeout is the maximum time in milliseconds to wait while connecting. Connecting to a server will fail with a SocketTimeoutException if the timeout elapses before a connection is established.
The readTimeout is the maximum time to wait for an input stream read to complete before giving up. Reading will fail with a SocketTimeoutException if the timeout elapses before data becomes available.
Can you give me a good reason why these two values should be different? Why a call would need more time for performing the connection rather than receiving some data (or viceversa)?
I am asking this because I have to configure these values and my idea is to set the same value for both.
Let's say server is busy and is configured to accept 'N' connection and all the connections are long runner and all of sudden you send in request, What should happen? Should you wait indefinitely or should you time out? That's connectTimeout.
While let's say your server turns brain dead service just accepting connection and doing nothing with it (or say server synchronously goes to db and does some time taking activity and server ends up with deadlock for e.g.) and on the other hand client keeps on waiting for the response, in this case what should client do? Should it wait indefinitely for response or should it timeout? That's read timeout.
The connection timeout is how long you're prepared to wait to get some sort of response from the server. It's not particularly related to what it is that you're trying to achieve.
But suppose you had a service that would allow you to give it a large number, and have it return its prime factors. The server might take quite a while to generate the answer and send it to you.
You might well have clear expectations that the server would quickly respond to the connection: maybe even a delay of 5 seconds here tells you that the server is likely to be down. But the read timeout might need to be much higher: it might be a few minutes before you get to be able to read the server's answer to your query.
The connect time-out is the time-out in which you want a (in normal situations TCP) connection to be established. The default time-outs as specified in the internet RFCs and implemented by the various OSes are normally in the minute(s) range. But we know that if a server is available and reachable, it will respond in a matter of milli-seconds and otherwise not at all. A normal value would be a couple of seconds at a maximum.
The read timeout is the time in which the server is expected to respond after it received the incoming request. Read time-outs therefore depend on time within you expect the server to deliver the result. These are depending on the type of the request you are making and should be larger if the processing requires some time or the server may be very busy in some situations. Especially if you do a retry after a read time-out, it is best to put the read time-outs not too low, normally a factor 3-4 times the expected time.
From the Netty API Documentation
connectTimeoutMillis = "the connect timeout in milliseconds. 0 if disabled."
And
ReadTimeoutHandler = Raises a ReadTimeoutException when no data was read within a certain period of time.
From a client perspective, am I correct in interpreting the aforementioned as follows?
The client will attempt to connect to the host for up to "connectTimeoutMillis". If a connection is established, and a ReadTimeoutHandler is NOT added to the Pipeline, a Channel can wait on a response indefinitely. If a ReadTimeoutHandler was added to the Pipeline, a ReadTimeoutException will be raised once timeoutSeconds has elapsed.
Generally speaking, I'd like to only attempt to connect to a host for up to 'x' seconds, but if a request was sent across the wire, I'd like to wait up to 'y' seconds for the response. If it shapes/influences the answer, the client is Netty, but the server is not.
Follow-up: Is timeoutSeconds on the ReadTimeoutHandler the timeout between successive bytes read, or for the entire request/response? Example: If timeoutSeconds was 60, and a single byte (out of a total of 1024) was read every 59 seconds, would the entire response be read successfully in 60416 seconds, or would it fail because the total elapsed time exceeded 60 seconds?
ReadTimeoutHandler doesn't understand the concept of a response. It only understands either a messageReceived event in Netty 3, or an inboundBufferUpdated event in Netty 4. Speaking from an NIO perspective the exact impact of this behaviour depends on where ReadTimeoutHandler is in your pipeline. (I've never used OIO so can't say if the behaviour is exactly the same).
If ReadTimeoutHandler is below any frame decoder in your pipeline (ie closer to the network) then the behaviour you describe is correct - a single byte read will reset the timer and, as you've identified, could result in the response taking a long time to be read. If you were writing a server this could be exploited to form a denial of service attack at the cost of very little effort on behalf of the attacker.
If ReadTimeoutHandler is above your frame decoder then it applies to your entire response. I think this is the behaviour you're looking for.
Note that the ReadTimeoutHandler is also unaware of whether you have sent a request - it only cares whether data has been read from the socket. If your connection is persistent, and you only want read timeouts to fire when a request has been sent, you'll need to build a request / response aware timeout handler.
Yes, you have correctly identified the difference between connect timeout and read timeout. Note that whatever any documentation may say to the contrary, the default or zero connect timeout means about 60-70 seconds, not infinity, and you can only use the connect timeout parameter to reduce that default, not increase it.
Read timeout starts when you call read() and ends when it expires or data arrives. It is the maximum time that read() may block waiting for the first byte to arrive. It doesn't block a second time in a single invocation.