We Keep Coding

Unable to execute HTTP request: Acquire operation took longer than the configured maximum time

I am running a batch job in AWS which consumes messages from a SQS queue and writes them to a Kafka topic using akka. I've created a Sqs Async Client with the following parameters:
private static SqsAsyncClient getSqsAsyncClient(final Config configuration, final String awsRegion) {
var asyncHttpClientBuilder = NettyNioAsyncHttpClient.builder()
.maxConcurrency(100)
.maxPendingConnectionAcquires(10_000)
.connectionMaxIdleTime(Duration.ofSeconds(60))
.connectionTimeout(Duration.ofSeconds(30))
.connectionAcquisitionTimeout(Duration.ofSeconds(30))
.readTimeout(Duration.ofSeconds(30));
return SqsAsyncClient.builder()
.region(Region.of(awsRegion))
.httpClientBuilder(asyncHttpClientBuilder)
.endpointOverride(URI.create("https://sqs.us-east-1.amazonaws.com/000000000000")).build();
}
private static SqsSourceSettings getSqsSourceSettings(final Config configuration) {
final SqsSourceSettings sqsSourceSettings = SqsSourceSettings.create().withCloseOnEmptyReceive(false);
if (configuration.hasPath(ConfigPaths.SqsSource.MAX_BATCH_SIZE)) {
sqsSourceSettings.withMaxBatchSize(10);
}
if (configuration.hasPath(ConfigPaths.SqsSource.MAX_BUFFER_SIZE)) {
sqsSourceSettings.withMaxBufferSize(1000);
}
if (configuration.hasPath(ConfigPaths.SqsSource.WAIT_TIME_SECS)) {
sqsSourceSettings.withWaitTime(Duration.of(20, SECONDS));
}
return sqsSourceSettings;
}
But, whilst running my batch job I get the following AWS SDK exception:
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Acquire operation took longer than the configured maximum time. This indicates that a request cannot get a connection from the pool within the specified maximum time. This can be due to high request rate.
The exception still seems to occur even after I try tweaking the parameters mentioned here:
Consider taking any of the following actions to mitigate the issue: increase max connections, increase acquire timeout, or slowing the request rate. Increasing the max connections can increase client throughput (unless the network interface is already fully utilized), but can eventually start to hit operation system limitations on the number of file descriptors used by the process. If you already are fully utilizing your network interface or cannot further increase your connection count, increasing the acquire timeout gives extra time for requests to acquire a connection before timing out. If the connections doesn't free up, the subsequent requests will still timeout. If the above mechanisms are not able to fix the issue, try smoothing out your requests so that large traffic bursts cannot overload the client, being more efficient with the number of times you need to call AWS, or by increasing the number of hosts sending requests
Has anyone run into this issue before?

I encountered the same issue, and I ended up firing 100 async batch requests then wait for those 100 to get cleared before firing another 100 and so on.

About SpringBoot Tomcat acceptCount pool

I'm dealing with the Tomcat configuration on springboot.
Let's supposse i have the following configuration:
server:
tomcat:
min-spare-threads: ${min-tomcat-threads:20}
max-threads: ${max-tomcat-threads:20}
accept-count: ${accept-concurrent-queue:1}
max-connections: ${max-tomcat-connections:100}
I have a simple RestController with this code:
public String request(#Valid #RequestBody Info info) {
log.info("Thread sleeping");
Thread.sleep(8000);
return "OK";
}
Then i make the following test:
I send 200 HTTP request per second.
I check the log and as I expected I see 100 simultaneous executions and after 8 seconds I see the last one (queued).
Other executions are rejected.
The main problem that i have with this is that if i have a timeout control on client call (for example, 5 seconds), the queued operation will be processed on server anyways even if it was rejected on client.
I want to avoid this situation, so I tried:
server:
tomcat:
min-spare-threads: ${min-tomcat-threads:20}
max-threads: ${max-tomcat-threads:20}
accept-count: ${accept-concurrent-queue:0}
max-connections: ${max-tomcat-connections:100}
But this "0" is totally ignored (i think in this case it means "infinite").
So, my question is:
¿Is it possible to configure Tomcat to don't queue operations if the max-connections limit is reached?
Or maybe
¿Is it possible to configure Tomcat to reject any operation queued?
Thank you very much in advance.
Best regards.

The value of the acceptCount parameter is passed directly to the operating system: e.g. for UNIX-es it is passed to listen. Since an incoming connection is always put in the OS queue before the JVM accepts it, values lower than 1 make no sense. Tomcat explicitly ignores such values and keeps its default 100.
However, the real queue in Tomcat are the connections that where accepted from the OS queue, but are not being processed due to a lack of processing threads (maxThreads). You might have at most maxConnections - maxThreads + 1 such connections. In your case it's 81 connections waiting to be processed.

Connection timed out latency

I have a microservice for spring boot 2 and it interacts with other services. I got a problem that when I call another service using resttemlate, and it is not available, the main thread stops and waits for a response from this service and as a result, after 15 seconds, it receives a Connection time out response. The problem is that I cannot wait this time. I need the service to work without delay and not wait a few seconds until another service responds or does not respond. How can this problem be solved? I think you can start another thread with a request to that service, and let the main thread continue to work without delay, is this the right solution? Is there a ready-made solution in Spring Boot for my problem?

You can set timeout for resttemplate. Give your desired value.
int timeout = 1; // time here is in milliseconds.
HttpComponentsClientHttpRequestFactory rf =
(HttpComponentsClientHttpRequestFactory) restTemplate.getRequestFactory();
rf.setConnectTimeout(timeout);

Handle client-side request timeout in java

A client sends a request and catches a timeout exception. However the server is still processing the request and saving it to the database. Before that happening, the client already sent a second request which doubles the record on the database. How do I prevent that from happening? Im using java servlets and javascript.

A few suggestions:-
1) Increase the client timeout.
2) Make the server more efficient so it can respond faster.
3) Get the server to respond with an intermediate "I'm working on it" response before returning with the main response.
4) Does the server need to do all the work before it responds to the client, or can some be offloaded to a seperate process for running later?

A client sends a request and catches a timeout exception. However the server is still processing the request
Make the servlet generate some output (can be just blank spaces) and flush the stream every so often (every 15 seconds for example).
If the connection has been closed on the client side, the write will fail with a socket exception.
Before that happening, the client already sent a second request which doubles the record on the database
Use the atomicity of the database, for example, a unique key. Start the process by creating a unique record (maybe in some "unfinished" status), it will fail if the record already exists.

Read timeout on a web service, but the operation still completes?

I have a Java web service client running on Linux (using Axis 1.4) that invokes a series of web services operations performed against a Windows server. There are times that some transactional operations fail with this Exception:
java.net.SocketTimeoutException: Read timed out
However, the operation on the server is completed (even having no useful response on the client). Is this a bug of either the web service server/client? Or is expected to happen on a TCP socket?

This is the expected behavior, rather than a bug. The operation behind the web service doesn't know anything about your read timing out so continues processing the operation.
You could increase the timeout of the connection - if you are manually manipulating the socket itself, the socket.connect() method can take a timeout (in milliseconds). A zero should avoid your side timing out - see the API docs.
If the operation is going to take a long time in each case, you may want to look at making this asynchronous - a first request submits the operations, then a second request to get back the results, possibly with some polling to see when the results are ready.
If you think the operation should be completing in this time, have you access to the server to see why it is taking so long?

I had similar issue. We have JAX-WS soap webservice running on Jboss EAP6 (or JBOSS 7). The default http socket timeout is set to 60 seconds unless otherwise overridden in server or by the client. To fix this issue I changed our java client to something like this. I had to use 3 different combinations of propeties here
This combination seems to work as standalone java client or webservice client running as part of other application on other web server.
//Set timeout on the client
String edxWsUrl ="http://www.example.com/service?wsdl";
URL WsURL = new URL(edxWsUrl);
EdxWebServiceImplService edxService = new EdxWebServiceImplService(WsURL);
EdxWebServiceImpl edxServicePort = edxService.getEdxWebServiceImplPort();
//Set timeout on the client
BindingProvider edxWebserviceBindingProvider = (BindingProvider)edxServicePort;
BindingProvider edxWebserviceBindingProvider = (BindingProvider)edxServicePort;
edxWebserviceBindingProvider.getRequestContext().put("com.sun.xml.internal.ws.request.timeout", connectionTimeoutInMilliSeconds);
edxWebserviceBindingProvider.getRequestContext().put("com.sun.xml.internal.ws.connect.timeout", connectionTimeoutInMilliSeconds);
edxWebserviceBindingProvider.getRequestContext().put("com.sun.xml.ws.request.timeout", connectionTimeoutInMilliSeconds);
edxWebserviceBindingProvider.getRequestContext().put("com.sun.xml.ws.connect.timeout", connectionTimeoutInMilliSeconds);
edxWebserviceBindingProvider.getRequestContext().put("javax.xml.ws.client.receiveTimeout", connectionTimeoutInMilliSeconds);
edxWebserviceBindingProvider.getRequestContext().put("javax.xml.ws.client.connectionTimeout", connectionTimeoutInMilliSeconds);