We Keep Coding

Specify server request timeout in Spring Boot & Spring WebFlux (Netty)

We are using Spring Boot in 2.4.2 with Spring WebFlux.
I want the Spring Boot application to terminate all requests to the application that take longer than say 3 seconds to process.
There is server.netty.connection-timeout, but that doesn't seem to do the trick.
Is there a way to specify such a server request timeout?

I was also facing the same issue i.e. even after configuring server.netty.connection-timeout request would get canceled. So, after some debugging found that timeout was getting set to '30000' by AsyncContext.
So, I configured the following property spring.mvc.async.request-timeout which change the timeout being set in AsyncContext and the request stopped getting canceled.

TL;DR:
Netty has no request timeout*. Add this WebFilter to set a request-timeout of 3 seconds using the reactor timeout on every request (here in kotlin, but in Java it works accordingly):
#Component
class RequestTimeoutWebFilter : WebFilter {
override fun filter(exchange: ServerWebExchange, chain: WebFilterChain): Mono<Void> {
return chain
.filter(exchange)
.timeout(Duration.ofSeconds(3))
}
}
* at least I could not find any information in the netty docs.
Detailed answer
The connection-timeout does not refer to the duration that a request is allowed to take for processing, but it refers to the time it takes for establishing the connection.
First, I could not find any spring configuration option that allows setting the request timeout for netty. Then I went through the netty documentation to find out that there is no concept of request timeouts on the http server (only on the http client).
Wondering about why such important feature would not be supported, I remembered that we often cannot apply the same concepts as in blocking servers for good reasons. Next, I remembered, that in the reactive world we do not directly implement the handling of the request, but how the handling is assembled - i.e. we hold a Mono<Void> that will handle the request. Hence, we can just look at reactor and how to timeout a Mono, which is very easy:
Mono.create(...)
.timeout(Duration.ofSeconds(3))
Next, we just need to figure out, how to apply this to all requests. This is easy as well, because we can just use a WebFilter to intercept all requests to apply our timeout (here in kotlin, but in Java it works accoringly):
#Component
class RequestTimeoutWebFilter : WebFilter {
override fun filter(exchange: ServerWebExchange, chain: WebFilterChain): Mono<Void> {
return chain
.filter(exchange)
.timeout(Duration.ofSeconds(3))
}
}
This effectively cancels a request within the set timeout with the following error:
2022-10-21 00:08:00.981 ERROR 6289 --- [ parallel-4] a.w.r.e.AbstractErrorWebExceptionHandler : [59dfa990-7] 500 Server Error for HTTP GET "/some/route"
java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 3000ms in 'source(MonoDefer)' (and no fallback has been configured)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:295) ~[reactor-core-3.4.22.jar:3.4.22]
More tips and hints
To make the timeout configurable, we can use a custom config variable instead of the hard-coded duration.
To custimize the 500 status code we can either change the exception by providing a fallback to the timeout as 2nd argument and handle that exception in a controller advice - or we can just use reactors onErrorReturn.
The documentation for WebFilter actually states that they should be used to implement timeouts:
Contract for interception-style, chained processing of Web requests that may be used to implement cross-cutting, application-agnostic requirements such as security, timeouts, and others.
Still I think it is expected that spring provides such implementation out-of-the box that can be easily configured. Maybe we oversaw that it is there, but then I would argue it is too hard to find. ^^
Alternative solution path
As an alternative, we could use circuit breakers. However, those are implemented on the readers side and conceptually are used to protect the reading side against failure of the downstream - rather than protecting the internal processing within the downstream from running too long. They can only be applied to mimic a request timeout when applying them in a dedicated server (e.g. a spring cloud gateway server) that sits between the actual client and the actual service. When using Resilience4j as implementation, you can use TimeLimiter to achieve it.

Spring boot thread hanging?

I have a simple (thats what I think) Spring boot application. There are 4 layers:
Rest Controller
Application Service (called by the Rest Controller)
Domain Service (called by Application Service. It connects to the database - repository layer)
Adapter Service (called by Application Service for outbound calls via Hystrix)
Now the problem is that it can only handle a max of 15 parallel calls. If any additional REST API request arrives while these calls are being processed, it makes it to the Application Service layer and then waits. Once one of those 15 parallel call returns, then the new request proceeds to make call to the Domain Service Layer and return.
I tried multiple things:
Increasing spare threads for the server in application.properties file
server.tomcat.min-spare-threads=1000
server.tomcat.max-connections=1000
server.tomcat.max-threads=1000
Once I do this, I see the # of http-nio-* threads increase to 1000 but the hanging issue is not fixed.
I found this snippet online to customize the tomcat container but it didn't help either:
#Bean
public WebServerFactoryCustomizer<TomcatServletWebServerFactory> containerCustomizer() {
return new WebServerFactoryCustomizer<TomcatServletWebServerFactory>() {
#Override
public void customize(TomcatServletWebServerFactory factory) {
factory.addConnectorCustomizers(new TomcatConnectorCustomizer() {
#Override
public void customize(Connector connector) {
Arrays.stream(connector.getProtocolHandler().findUpgradeProtocols())
.filter(upgradeProtocol -> upgradeProtocol instanceof Http2Protocol)
.map(upgradeProtocol -> (Http2Protocol) upgradeProtocol)
.forEach(http2Protocol -> {
http2Protocol.setMaxConcurrentStreamExecution(1000);
});
}
});
}
};
}
I tried configuring the thread pool via code
#Bean(name = "taskExecutor")
public TaskExecutor threadPoolTaskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(200);
executor.setMaxPoolSize(300);
executor.setQueueCapacity(300);
executor.setThreadNamePrefix("anniversary");
executor.initialize();
System.out.println("******* name " + executor.getThreadNamePrefix());
System.out.println("********** core pool size " + executor.getCorePoolSize());
return executor;
}
But none of this helps and I believe the issue is not with the number of threads but elsewhere since the request is not able to go from one service to another. There are hundreds of http-nio-* threads in waiting state and when a new request comes in, its assigned its own thread and I can see that in the Debug mode.
Any pointers, help, tips are much appreciated. What resource is required for service to service invocation by Spring boot?

I believe your observation is right - it's most likely not tomcat who's the bottleneck here. From what you write, would rather look at the domain service. Is the domain service doing some communication with the database or talking to something else over the network (for example over HTTP)?
If you happen to do database in there, check for spring's datasource configuration. There is going to be a database connection pool with a limited number of maximum concurrent connections to the database. Once these connections are all in use, threads that want to talk to the DB will be blocked until one of the connection becomes free again.
Similar connection pools are in place with many other things that talk over network (e.g. Apache HTTP Client also has a connection pool that can be configured).
That's where i would look next.
Cheers,
Matthias

Perform route Shutdown Logic with apache camel

I've recently started playing with Apache Camel, and one of the things I've been having issues with is properly performing shutdown logic on selective routes. Since the shutdown logic would vary between routes, Camel's RoutePolicy made the most sense. Here's an example of why I'm trying to do.
public class ProcessingRouteBuilder extends RouteBuilder {
private ProducerTemplate prodTemplate;
public class ProcessingRouteBuilder(ProducerTemplate aProdTemplate) {
prodTemplate = aProdTemplate;
}
#Override
public void configure() {
from("direct://processing")
.routePolicy(new RoutePolicySupport() {
#Override
public void onStop(Route route) {
super.onStop(route);
prodTemplate.sendBody("direct://shutdownRoute", "msg");
}
})
.process(ex -> // Do stuff)
from("direct://shutdownRoute")
.log("Running shutdown A route body - ${body}");
}
}
The shutdown is done like (http://camel.apache.org/how-can-i-stop-a-route-from-a-route.html). The ProducerTemplate comes from the primary CamelContext (read that it is good practice to create one ProducerTemplate per context).
Running this gives me a DirectConsumerNotAvailableException, I've used seda and vm (i don't plan to interact with multiple contexts, but I gave this a shot anyways), both don't exception, but the shutdown routes are never hit. Some questions I have
I might be using the Producer Template wrong? It doesn't look like it's creating an exchange.
Can I even use the ProducerTemplate once the Shutdown hook has been initiated? I'm not sure how Camel performs the shutdown, but it makes sense that it wouldn't allow new messages to be sent, and if the shutdown route is even available at the time of sending.
One thing to note, that I'm not handling here, is ensuring that the shutdown route is performed after the processing route finishes processing all messages in its queue. I'm not entirely sure if the onStop() method is called after there are no more inflight messages and if not, how to enforce it?
I figure another approach is to use when/choice at the beginning of each route and send some sort of shutdown notifier or message, but this seems a little more clunkier.
Thanks guys!

To programmatic shut down a route you can also use the Control Bus EIP.
However the "stop" logic is not clear as you'd want to send a message to the shutdownroute when the processing route stops, but if the stop happen because you are shutting down the camel context it may be possible that the shutdownRoute has already been stopped.

Jersey #ManagedAsync and copying data between HTTP thread and Worker thread

I am working on a project that works in two flavors with and without multi tenancy.
The project exposes a REST service which I would like to be asynchronous.
So my basic service looks like
#Component
#Path("/resouce")
#Consumes(MediaType.APPLICATION_JSON)
public class ResouceEndpoint {
#POST
#ManagedAsync
public void add(final Event event, #Suspended final AsyncResponse asyncResponse) {
resouce.insert (event);
asyncResponse.resume( Response.status(Response.Status.NO_CONTENT).build());
}
}
That works fine without multi tenancy and I get the benefits of the internal Jersey executor service for free. See #ManagedAsync
When I switch to multi tenancy I add a filter on the request that resolve the tenant id and place it on the thread local (in our case the HTTP thread).
When the processing chain hits the "add()" method above the current thread is the one provided by the Jersey executor service, so it does not include my tenant id.
I could think only on the following options to work around this issue.
Extend the ResouceEndpoint to MutliTenantResouceEndpoint and drop the #ManagedAsync
Using my own thread executor
public class MutliTenantResouceEndpoint extends ResouceEndpoint {
#POST
public void add(final Event event, #Suspended final AsyncResponse asyncResponse) {
final String tenantId = getTeantIdFromThreadLocal();
taskExecutor.submit(new Callable<Void>() {
#Override
public Void call() throws Exception {
setTeantIdToThreadLocal(tenantId);
browserEventsAnalyzer.insertEvent(event);
Response response = Response.status(Response.Status.NO_CONTENT).build();
asyncResponse.resume(response);
return null;
}
});
}
}
But this way I need to manage my own thread executor and it feel's like I am missing something here.
Any suggestion on a different approach?

Here are a handful of recommendations, in order.
For context, I've been using Jersey for 2 years, and faced this exact problem 18 months ago.
1. Stop using #ManagedAsync
If you have control over the http server that Jersey is running on, I would recommend you stop using #ManagedAsync.
Instead of setting up Jersey to return it's http handling thread immediately and offload real request work to a managed executor service thread, use something like Grizzly for your http server, and configure it to have a larger worker thread pool. This accomplishes the same thing, but pushes the async responsibility down a layer, below Jersey.
You'll run into many pain points over the course of a year if you use #ManagedAsync for any medium-to-large project. Here are some of them off the top of my head:
If any ContainerRequestFilter's hits an external service (e.g. an auth filter hits your security module, which hits the database) you will lose the benefits you thought you were gaining
If your DB chokes and that auth filter call takes 5 seconds, Jersey hasn't offloaded work to the async thread yet, so your main thread needed to receive a new conn is blocked
If you set up logback's MDC in a filter, and you want that context throughout your request, you'll need to set up the MDC again on the managed async thread
Resource methods are cryptic to new comers and ugly to read because:
they need an extra parameter
they return void, hiding their real response type
they can "return" anywhere, without any actual return statements
Swagger or other API doc tools cannot automatically document async resource endpoints
Guice or other DI frameworks may have trouble dealing with certain scope bindings and/or providers in async resource endpoints
2. Use #Context and ContainerRequest properties
This would involve involved calling requestContext.setProperty("tenant_id", tenantId) in your filter, then calling calling requestContext.getProperty("tenant_id") in your resource with a #Context injected request.
3. Use HK2 AOP instead of Jersey filters
This would involve setting up an HK2 binding of InterceptionService which has a MethodInterceptor that checks for managed async resource methods and manually executes all RequestScoped bound ContainerRequestFilters. Instead of your filters being registered with Jersey, you'd register them with HK2, to be run by the method interceptor.
I can add more detail and code samples to options 2/3 if you'd like, or give additional suggestions, but it would first be helpful to see more of your filter code, and I again suggest option 1 if possible.

How do I init a spring integration:gateway at startup?

I'm using spring integration to communicate via active mq. I've got the following config.
<integration:channel id="riskApprovalRequestChannel"/>
<integration:channel id="riskApprovalResponseChannel"/>
<jms:outbound-gateway id="riskApprovalServiceGateway"
request-destination-name="${risk.approval.queue.request}"
reply-destination-name="${risk.approval.queue.response}"
request-channel="riskApprovalRequestChannel"
reply-channel="riskApprovalResponseChannel"
connection-factory="jmsConnectionFactory"
receive-timeout="1000"/>
<integration:gateway id="riskApprovalService" service-interface="com.my.ServiceInterface"
default-request-channel="riskApprovalRequestChannel"
default-reply-channel="riskApprovalResponseChannel"/>
This works well except for the 1st request, which is slow. I always hit the 1 second timeout on the 1st request. There's obviously some lazy loading going on somewhere.
My question is, how do I do a full initialisation at start up to avoid the 1st request always timing out?
Cheers,
Peter

It is probably connection establishment that's the issue.
One thing you can do is wrap the vendor ConnectionFactory in a Spring CachingConnectionFactory (good idea to do this anyway, for performance reasons), and call createConnection() during initialization.
There are a number of ways to do that - including injecting the connection factory into some other bean, with an init-method, #PostConstruct method, or an InitializingBean with afterPropertiesSet(); it will be called during context initialization.
This will cause the cache's connection to be eagerly connected.
You might want to do it in a try block, to avoid the application context failing to initialize because the JMS broker is not available (of course messaging will still fail, but the app will be up and ready to connect when the broker is available).