I have a very basic spring boot 2.2.4 application that queries a downstream system using webclient with a blocking call. I am not doing any configuration of the webclient (setting timeouts, etc.), just using it "out of the box".
What I find is that the response time of the webclient call is either below 3 seconds or precisely 45 seconds, which I find very strange. Why if the response is slow, it is always 45 seconds?
The only reference to the 45 seconds I could find comes from the Reactor Netty documentation:
4.6. Connection Pool
By default, the TCP client uses a “fixed” connection pool with 500 as the maximum number of the channels and 45s as the acquisition timeout. This means that the implementation creates a new channel if someone tries to acquire a channel but none is in the pool. When the maximum number of the channels in the pool is reached, new tries to acquire a channel are delayed until a channel is returned to the pool again. The implementation uses FIFO order for channels in the pool. By default, there is no idle time specified for the channels in the pool.
Does anybody have any suggestion as to why my slow webclient calls always take 45 seconds to complete?
This is happening because of the fixed pool size which is 500. If all these connections (channels) are currently in use by existing requests, a new request for a connection would be queued until one of the 500 connections becomes free. If WebClient isn't able to acquire a connection within 45 seconds for a new request (since all of the 500 channels are still blocked by existing requests), it fails with an AcquireTimeout Exception. I'm assuming the ones that finished before 3 seconds are success ones while the 45 second ones are failures. Depending on the Throughput of your application, you can adjust the pool size accordingly.
In the past, when I've seen delays like this on successful connections, it was caused like this:
You try to connect with a domain name, so the client calls DNS first to get an address;
DNS returns address records for both IPv4 and IPv6
The client tries IPv6 first, but IPv6 is not properly configured on the network, and this fails after the IPv6 connection timeout, which can be in the 45s range.
The client then tries IPv4, which succeeds.
Whether or not this happens to you depends on your OS+version, your java version, and the configuration of your network.
Try connecting with in IPv4 IP address, like http://192.168.10.1/the/rest. If it's always fast then you probably had a problem like above. Unfortunately you can't usually connect this way with HTTPS, but depending on your OS you could try stuffing the correct v4 address in /etc/hosts.
Look here for more: https://serverfault.com/questions/548777/how-to-prevent-delays-associated-with-ipv6-aaaa-records
I think this is related to this netty issue:
https://github.com/reactor/reactor-netty/issues/1012
I will have a look at that first now.
Thanks for all the replies!
Related
I want to create a Jersey Client for a multithreaded project. Do i need to create a connection pool for the Client ?
There is 500 TPS traffic on the server .How to calculate the below parameters for best performance.
ConnectionTimout,SocketTimeout,ReadTimeout,MaxConnectionPerHost,MaxConnections.
What is concept of reset connection pool and when to use it?
Jersey Client is thread-safe for multiple request executions. Documentation
Methods to create instances of WebResource are thread-safe. Methods
that modify configuration and or filters are not guaranteed to be
thread-safe.
The creation of a Client instance is an expensive operation and the
instance may make use of and retain many resources. It is therefore
recommended that a Client instance is reused for the creation of
WebResource instances that require the same configuration settings.
It is recommended that the same instance of Client class is reused for multiple request executions. Changing Client configuration is not thread save and must be handled appropriately.
Do I need to create a connection pool for the Client ?
Short answer Yes.
By default Jersey Client use BasicHttpClientConnectionManager. Documentation
It is a simple connection manager that maintains only one
connection at a time. Even though this class is thread-safe it ought
to be used by one execution thread only.
BasicHttpClientConnectionManager will make an effort to reuse the
connection for subsequent requests with the same route. It will,
however, close the existing connection and re-open it for the given
route, if the route of the persistent connection does not match that
of the connection request. If the connection has been already been
allocated, then java.lang.IllegalStateException is thrown.
For multithread application, you need to override the default value with PoolingHttpClientConnectionManager.
It is a more complex implementation that manages a pool of client
connections and is able to service connection requests from multiple
execution threads. Connections are pooled on a per route basis. A
request for a route for which the manager already has a persistent
connection available in the pool will be serviced by leasing a
connection from the pool rather than creating a brand new connection.
PoolingHttpClientConnectionManager maintains a maximum limit of
connections on a per route basis and in total. Per default this
implementation will create no more than 2 concurrent connections per
given route and no more 20 connections in total. For many real-world
applications these limits may prove too constraining, especially if
they use HTTP as a transport protocol for their services.
For details Apache Connection Management
Configuration
The general recommendation is to avoid the infinity timeouts, which Jersey set by default. It can produce threads stuck in case of problems. See Best practices for web service timeouts choose the correct value. There are no specific values, they should be set based on services and environment performance. The correct timeouts and connection size will come thru the time after performance testing or real-time usage.
Just implement it flexible, add the ability to adjust settings on the fly.
Read Timeout
Read timeout interval property, in milliseconds. The value MUST be an
instance of Integer. If the property is absent then the default value
is an interval of infinity. A value of zero 0 is equivalent to an
interval of infinity
You can set 1 minute like the initial value. Also, you can override timeout per request in case of exceptional cases.
Connection Timeout
Connect timeout interval property, in milliseconds. The value MUST be
an instance of Integer. If the property is absent then the default
value is an interval of infinity. A value of 0 is equivalent to an
interval of infinity
Set 500 - 1000 milliseconds like the initial value.
MaxConnectionPerHost
Set 20 connections like initial value.
MaxConnections
Set 200 connections like initial value.
I am working on socket programming on Java recently and something is confusing me. I have three questions about it.
First one is;
There is a ServerSocket method in Java. And this method can take up to 3 parameters such as port, backlog and ip address. Backlog means # of clients that can connect as a form of queue into a server. Now lets think about this situation.
What happens if 10 clients try to connect this server at the same
time?
Does Server drop last 5 clients which tried to connect? Lets increase the number of clients up to 1 million per hour. How can I handle all of them?
Second question is;
Can a client send messages concurrently without waiting server's response? What happens if a client sends 5 messages into server that has 5 backlog size?
The last one is not a question actually. I have a plan to manage load balancing in my mind. Lets assume we have 3 servers running on a machine.
Let the servers names are A, B and C and both of them are running smoothly. According to my plan, if I gave them a priority according to incoming messages then smallest priority means the most available server. For example;
Initial priorities -> A(0), B(0), C(0) and respond time is at the end of 5. time unit.
1.Message -> A (1), B(0), C(0)
2.Message -> A (1), B(1), C(0)
3.Message -> A (1), B(1), C(1)
4.Message -> A (2), B(1), C(1)
5.Message -> A (2), B(2), C(1)
6.Message -> A (1), B(2), C(2)
.
.
.
Is this logic good? I bet there is a far better logic. What do I do to handle more or less a few million requests in a day?
PS: All this logic is going to be implemented into Java Spring-Boot project.
Thanks
What happens if 10 clients try to connect this server at the same time?
The javadoc explains it:
The backlog argument is the requested maximum number of pending connections on the socket. Its exact semantics are implementation specific. In particular, an implementation may impose a maximum length or may choose to ignore the parameter altogther.
.
Lets increase the number of clients up to 1 million per hour. How can I handle all of them?
By accepting them fast enough to handle them all in one hour. Either the conversations are so quick that you can just handle them one after another. Or, more realistically, you will handle the various messages in several threads, or use non-blocking IO.
Can a client send messages concurrently without waiting server's response?
Yes.
What happens if a client sends 5 messages into server that has 5 backlog size?
Sending messages has nothing to do with the backlog size. The backlog is for pending connections. Messages can only be sent once you're connected.
All this logic is going to be implemented into Java Spring-Boot project.
Spring Boot is, most of the time, not used for low-level socket communication, but to expose web services. You should probably do that, and let standard solutions (a reverse proxy, software or hardware) do the load-balancing for you. Especially given that you don't seem to understand how sockets, non-blocking IO, threads, etc. work yet.
So for your first question, the backlog queue is something where the clients will be held in wait if you are busy with handling other stuff (IO with already connected client e.g.). If the list grows beyond backlog, the those news clients will get a connection refused. You should be ok with 10 clients connect at the same time. It's long discussion, but keep a thread pool, as soon you get a connected socket from accept, hand it to your thread pool and go back to wait in accept. You can't support millions of client "practically" on one single server period! You'll need to load balance.
Your second question is not clear, clients can't send messages, as long as they are on the queue, they will be taken off the queue, once you accept them & then it's not relevant how long the queue is.
And lastly your question about load balancing, I'd suggest if you are going to have to serve millions of clients, invest in some good dedicated load-balancer :), that can do round robin as well as you mentioned.
With all that said, don't reinvent the wheel :), there are some open source java servers, my favorite: https://netty.io/
In the docs (Tomcat 7 Config), it is written:
The number of milliseconds this Connector will wait, after accepting a connection, for the request URI line to be presented. Use a value of -1 to indicate no (i.e. infinite) timeout. The default value is 60000 (i.e. 60 seconds) but note that the standard server.xml that ships with Tomcat sets this to 20000 (i.e. 20 seconds). Unless disableUploadTimeout is set to false, this timeout will also be used when reading the request body (if any).
When a client sends request to a server, it will take N milliseconds to establish a connection. If this N exceeds the connection timeout that is set on the client's end, the request will fail in the client as expected.
I'm not able to understand what Tomcat's connectionTimeout does differently. Specifically, what does "after accepting a connection, for the request URI line to be presented" means ?
The connectionTimeout is the limit of time after which the server will automatically close the connection with the client not the other way around. It is a way to limit the impact of a Denial Of Service Attack. Indeed a typical way to do a DOS Attack is to launch several requests on a given server, and each request will last forever making the server waits for nothing and filling up its pool of threads such that the Server won't be able to accept any new requests. Thanks to this timeout, after x milliseconds it will ignore the request considering it as a potential attack.
Here is an interesting discussion on globally the same subject that goes a little bit deeper.
This question already has answers here:
What is the difference between connection and read timeout for sockets?
(2 answers)
Closed 8 years ago.
Just curiosity. Is there a good reason why the class URLConnection needs to have two different timeouts?
The connectTimeout is the maximum time in milliseconds to wait while connecting. Connecting to a server will fail with a SocketTimeoutException if the timeout elapses before a connection is established.
The readTimeout is the maximum time to wait for an input stream read to complete before giving up. Reading will fail with a SocketTimeoutException if the timeout elapses before data becomes available.
Can you give me a good reason why these two values should be different? Why a call would need more time for performing the connection rather than receiving some data (or viceversa)?
I am asking this because I have to configure these values and my idea is to set the same value for both.
Let's say server is busy and is configured to accept 'N' connection and all the connections are long runner and all of sudden you send in request, What should happen? Should you wait indefinitely or should you time out? That's connectTimeout.
While let's say your server turns brain dead service just accepting connection and doing nothing with it (or say server synchronously goes to db and does some time taking activity and server ends up with deadlock for e.g.) and on the other hand client keeps on waiting for the response, in this case what should client do? Should it wait indefinitely for response or should it timeout? That's read timeout.
The connection timeout is how long you're prepared to wait to get some sort of response from the server. It's not particularly related to what it is that you're trying to achieve.
But suppose you had a service that would allow you to give it a large number, and have it return its prime factors. The server might take quite a while to generate the answer and send it to you.
You might well have clear expectations that the server would quickly respond to the connection: maybe even a delay of 5 seconds here tells you that the server is likely to be down. But the read timeout might need to be much higher: it might be a few minutes before you get to be able to read the server's answer to your query.
The connect time-out is the time-out in which you want a (in normal situations TCP) connection to be established. The default time-outs as specified in the internet RFCs and implemented by the various OSes are normally in the minute(s) range. But we know that if a server is available and reachable, it will respond in a matter of milli-seconds and otherwise not at all. A normal value would be a couple of seconds at a maximum.
The read timeout is the time in which the server is expected to respond after it received the incoming request. Read time-outs therefore depend on time within you expect the server to deliver the result. These are depending on the type of the request you are making and should be larger if the processing requires some time or the server may be very busy in some situations. Especially if you do a retry after a read time-out, it is best to put the read time-outs not too low, normally a factor 3-4 times the expected time.
From the Netty API Documentation
connectTimeoutMillis = "the connect timeout in milliseconds. 0 if disabled."
And
ReadTimeoutHandler = Raises a ReadTimeoutException when no data was read within a certain period of time.
From a client perspective, am I correct in interpreting the aforementioned as follows?
The client will attempt to connect to the host for up to "connectTimeoutMillis". If a connection is established, and a ReadTimeoutHandler is NOT added to the Pipeline, a Channel can wait on a response indefinitely. If a ReadTimeoutHandler was added to the Pipeline, a ReadTimeoutException will be raised once timeoutSeconds has elapsed.
Generally speaking, I'd like to only attempt to connect to a host for up to 'x' seconds, but if a request was sent across the wire, I'd like to wait up to 'y' seconds for the response. If it shapes/influences the answer, the client is Netty, but the server is not.
Follow-up: Is timeoutSeconds on the ReadTimeoutHandler the timeout between successive bytes read, or for the entire request/response? Example: If timeoutSeconds was 60, and a single byte (out of a total of 1024) was read every 59 seconds, would the entire response be read successfully in 60416 seconds, or would it fail because the total elapsed time exceeded 60 seconds?
ReadTimeoutHandler doesn't understand the concept of a response. It only understands either a messageReceived event in Netty 3, or an inboundBufferUpdated event in Netty 4. Speaking from an NIO perspective the exact impact of this behaviour depends on where ReadTimeoutHandler is in your pipeline. (I've never used OIO so can't say if the behaviour is exactly the same).
If ReadTimeoutHandler is below any frame decoder in your pipeline (ie closer to the network) then the behaviour you describe is correct - a single byte read will reset the timer and, as you've identified, could result in the response taking a long time to be read. If you were writing a server this could be exploited to form a denial of service attack at the cost of very little effort on behalf of the attacker.
If ReadTimeoutHandler is above your frame decoder then it applies to your entire response. I think this is the behaviour you're looking for.
Note that the ReadTimeoutHandler is also unaware of whether you have sent a request - it only cares whether data has been read from the socket. If your connection is persistent, and you only want read timeouts to fire when a request has been sent, you'll need to build a request / response aware timeout handler.
Yes, you have correctly identified the difference between connect timeout and read timeout. Note that whatever any documentation may say to the contrary, the default or zero connect timeout means about 60-70 seconds, not infinity, and you can only use the connect timeout parameter to reduce that default, not increase it.
Read timeout starts when you call read() and ends when it expires or data arrives. It is the maximum time that read() may block waiting for the first byte to arrive. It doesn't block a second time in a single invocation.