When using Spring webflux with Mono or Flux return type, the http connecting thread is parked/released while the connection waits for the response. Thus, the connection is not taking max-connections.
Question: how can I test/proof that the connection is really released while waiting for the response, and not blocking max-connections?
I already enabled DEBUG logging, but that did not show anything regarding this question.
#RestController
public class MyServlet {
#GetMapping("/")
public Mono<String>/Flux<String> test() {
return Mono.just(service.blocking());
}
}
#Service
public class SlowService {
public String blocking() {
TimeUnit.SECONDS.sleep(10);
return "OK";
}
}
Or is that incorrect at all, and I'd have to use:
Mono.fromCallable(() -> service.blocking()).subscribeOn(Schedulers.elastic());
But still, how can I see from the logs that the connection gets parked correctly?
To test, I'm using server.tomcat.max-threads=5. I'm trying to rewrite my blocking service so that those threads are not blocked during the sleep, and thus more than 5 connections can reach my service concurrently.
There are 2 thread pools, the forkjoin pool that will handle all the regular work. Then you have the Schedular pool that will handle scheduled tasks.
return Mono.just(service.blocking());
This will block one of the threads in the ForkJoinPool so less events can be handled by webflux hence slowing down your service.
Mono.fromCallable(() -> service.blocking()).subscribeOn(Schedulers.elastic());
This will assign the task and "offload" it to the scheduler pool of threads so another thread pool will handle this and not hog one of the ForkJoinPool threads.
How to test this? You need to load test your service, or as most people do, trust the framework to do what it is set out to do and trust that the spring team has tested their side of things.
Related
I have such a controller and such a service class. Why am I getting this warning in IDEA - "Possibly blocking call in non-blocking context could lead to thread starvation" ?
#PostMapping(value = {"/create"})
public Mono<ResponseEntity<ResponseDto>> create(
#RequestBody RequestDto request) {
ResponseDto result = service.create(request);
return Mono.just(ResponseEntity.ok(result));
}
#Transactional
public ResponseDto create(RequestDto request) {
taskRepository.save(request);
return new ResponseDto("Ок");
}
This is apparently caused by the #Transactional annotation. When I remove it, the warning disappears. What is this problem and how can it be fixed?
p.s. this example is schematic. the real code is bigger.
The reactive process is contrary to the norm. You cannot use blocking elements here! With Tomcat, it creates a separate thread for each request so that the topic can be blocked. Reactive Netty will NOT create a new thread, just uses a fixed pool.
With the loose approach, you can think that if a process is waiting for a response, it gives the resource of that thread to another. If you block it, it won't be able to do that. Therefore, even with a single-threaded Netty, it can handle to serve multiple parallel requests.
Therefore, thread-based data storage also does not work properly, because another process can interfere or modify it. Therefore, reactive context is available instead.
There is a article to reactive transaction. I don't know it will be solution for you:
https://itnext.io/integrating-hibernate-reactive-with-spring-5427440607fe
I am using Spring Webflux with Spring data jpa using PostgreSql as backend db.
I don't want to block the main thread while making db calls like find and save.
To achieve the same, I have a main scheduler in Controller class and a jdbcScheduler service classes.
The way I have defined them is:
#Configuration
#EnableJpaAuditing
public class CommonConfig {
#Value("${spring.datasource.hikari.maximum-pool-size}")
int connectionPoolSize;
#Bean
public Scheduler scheduler() {
return Schedulers.parallel();
}
#Bean
public Scheduler jdbcScheduler() {
return Schedulers.fromExecutor(Executors.newFixedThreadPool(connectionPoolSize));
}
#Bean
public TransactionTemplate transactionTemplate(PlatformTransactionManager transactionManager) {
return new TransactionTemplate(transactionManager);
}
}
Now, while doing a get/save call in my service layer I do:
#Override
public Mono<Config> getConfigByKey(String key) {
return Mono.defer(
() -> Mono.justOrEmpty(configRepository.findByKey(key)))
.subscribeOn(jdbcScheduler)
.publishOn(scheduler);
}
#Override
public Flux<Config> getAllConfigsAfterAppVersion(int appVersion) {
return Flux
.fromIterable(configRepository.findAllByMinAppVersionIsGreaterThanEqual(appVersion))
.subscribeOn(jdbcScheduler)
.publishOn(scheduler);
}
#Override
public Flux<Config> addConfigs(List<Config> configList) {
return Flux.fromIterable(configRepository.saveAll(configList))
.subscribeOn(jdbcScheduler)
.publishOn(scheduler);
}
And in controller, I do:
#PostMapping
#ResponseStatus(HttpStatus.CREATED)
Mono<ResponseDto<List<Config>>> addConfigs(#Valid #RequestBody List<Config> configs) {
return configService.addConfigs(configs).collectList()
.map(configList -> new ResponseDto<>(HttpStatus.CREATED.value(), configList, null))
.subscribeOn(scheduler);
}
Is this correct? and/or there is a way better way to do it?
What I understand by:
.subscribeOn(jdbcScheduler)
.publishOn(scheduler);
is that task will run on jdbcScheduler threads and later result will be published on my main parallel scheduler. Is this understanding correct?
Your understanding is correct with regards to publishOn and subscribeOn (see reference documentation in the reactor project about those operators).
If you call blocking libraries without scheduling that work on a specific scheduler, those calls will block one of the few threads available (by default, the Netty event loop) and your application will only be able to serve a few requests concurrently.
Now I'm not sure what you're trying to achieve by doing that.
First, the parallel scheduler is designed for CPU bound tasks, meaning you'll have few of them, as many (or a bit more) as CPU cores. In this case, it's like setting your threadpool size to the number of cores on a regular Servlet container. Your app won't be able to process a large number of concurrent requests.
Even if you choose a better alternative (like the elastic Scheduler), it will be still not as good as the Netty event loop, which is where request processing is scheduled natively in Spring WebFlux.
If your ultimate goal is performance and scalability, wrapping blocking calls in a reactive app is likely to perform worse than your regular Servlet container.
You could instead use Spring MVC and:
use usual blocking return types when you're dealing with a blocking library, like JPA
use Mono and Flux return types when you're not tied to such libraries
This won't be non-blocking, but this will be asynchronous still and you'll be able to do more work in parallel without dealing with the complexity.
IMHO, there a way to execute this operation doing a better use of resources from machine. Following documentation you can wrap the call in other Thread and with this you can continue your execution.
I have a simple (thats what I think) Spring boot application. There are 4 layers:
Rest Controller
Application Service (called by the Rest Controller)
Domain Service (called by Application Service. It connects to the database - repository layer)
Adapter Service (called by Application Service for outbound calls via Hystrix)
Now the problem is that it can only handle a max of 15 parallel calls. If any additional REST API request arrives while these calls are being processed, it makes it to the Application Service layer and then waits. Once one of those 15 parallel call returns, then the new request proceeds to make call to the Domain Service Layer and return.
I tried multiple things:
Increasing spare threads for the server in application.properties file
server.tomcat.min-spare-threads=1000
server.tomcat.max-connections=1000
server.tomcat.max-threads=1000
Once I do this, I see the # of http-nio-* threads increase to 1000 but the hanging issue is not fixed.
I found this snippet online to customize the tomcat container but it didn't help either:
#Bean
public WebServerFactoryCustomizer<TomcatServletWebServerFactory> containerCustomizer() {
return new WebServerFactoryCustomizer<TomcatServletWebServerFactory>() {
#Override
public void customize(TomcatServletWebServerFactory factory) {
factory.addConnectorCustomizers(new TomcatConnectorCustomizer() {
#Override
public void customize(Connector connector) {
Arrays.stream(connector.getProtocolHandler().findUpgradeProtocols())
.filter(upgradeProtocol -> upgradeProtocol instanceof Http2Protocol)
.map(upgradeProtocol -> (Http2Protocol) upgradeProtocol)
.forEach(http2Protocol -> {
http2Protocol.setMaxConcurrentStreamExecution(1000);
});
}
});
}
};
}
I tried configuring the thread pool via code
#Bean(name = "taskExecutor")
public TaskExecutor threadPoolTaskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(200);
executor.setMaxPoolSize(300);
executor.setQueueCapacity(300);
executor.setThreadNamePrefix("anniversary");
executor.initialize();
System.out.println("******* name " + executor.getThreadNamePrefix());
System.out.println("********** core pool size " + executor.getCorePoolSize());
return executor;
}
But none of this helps and I believe the issue is not with the number of threads but elsewhere since the request is not able to go from one service to another. There are hundreds of http-nio-* threads in waiting state and when a new request comes in, its assigned its own thread and I can see that in the Debug mode.
Any pointers, help, tips are much appreciated. What resource is required for service to service invocation by Spring boot?
I believe your observation is right - it's most likely not tomcat who's the bottleneck here. From what you write, would rather look at the domain service. Is the domain service doing some communication with the database or talking to something else over the network (for example over HTTP)?
If you happen to do database in there, check for spring's datasource configuration. There is going to be a database connection pool with a limited number of maximum concurrent connections to the database. Once these connections are all in use, threads that want to talk to the DB will be blocked until one of the connection becomes free again.
Similar connection pools are in place with many other things that talk over network (e.g. Apache HTTP Client also has a connection pool that can be configured).
That's where i would look next.
Cheers,
Matthias
we experience the following problem: We have got some service bean that does some handling and will iterate over a set and start an async thread for each entry, which will look like following:
#Service
public class ServiceBean {
#Autowired
private AsyncHandler asyncHandler;
public void doService(Set<?> theSet) {
for (Object obj : theSet) {
Future<?> future = asyncHandler.doHandle(obj);
// ...
}
// wait for all futures to complete
}
}
#Service
public class AsyncHandler {
#Async
public Future<?> doHandle(Object object) {
// dosth
return future;
}
}
Now we see that each thread will hold its own transaction and thereby database connection, which will cause the connection pool to run empty very fast.
What is the desired way to share the transaction accross all threads?
Thanks in advance!
What is the desired way to share the transaction accross all threads?
Don't do that. That makes it incredibly hard to understand what is going on. There's a reason why a transaction is attached to the thread that has created it. I guess your plan is to start all those asynchronous threads and then wait for them to complete to send the commit signal.
If you don't, who's going to complete the transaction? If a thread rollbacks the transaction, what happens in other threads that are still using it? You could pass the Connection around obviously but I'd expect things to be serialized anyway as most connection implementation are using synchronized in various places.
I have used Spring Framework's Scheduled to schedule my job to run at every 5 mins using cron. But sometime my job waits infinitely for an external resource and I can't put timeout there. I can't use fixedDelay as previous process sometime goes in wait infinitely mode and I have to refresh data at every 5 mins.
So I was looking any option in Spring Framework's Scheduled to stop that process/thread after a fixed-time either it run successfully or not.
I have found below setting which initialized ThreadPoolExecutor with 120 seconds for keepAliveTime which I put in #Configuration class. Can anybody tell me will this work as I expected.
#Bean(destroyMethod="shutdown")
public Executor taskExecutor() {
int coreThreads = 8;
int maxThreads = 20;
final ThreadPoolExecutor threadPoolExecutor = new ThreadPoolExecutor(
coreThreads, maxThreads, 120L,
TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>()
);
threadPoolExecutor.allowCoreThreadTimeOut(true);
return threadPoolExecutor;
}
I'm not sure this will work as expected. Indeed the keepAlive is for IDLE thread and I don't know if your thread waiting for resources is in IDLE. Furthermore it's only when the number of threads is greater than the core so you can't really know when it happen unless you monitor the threadpool.
keepAliveTime - when the number of threads is greater than the core, this is the maximum time that excess idle threads will wait for new tasks before terminating.
What you can do is the following:
public class MyTask {
private final long timeout;
public MyTask(long timeout) {
this.timeout = timeout;
}
#Scheduled(cron = "")
public void cronTask() {
Future<Object> result = doSomething();
result.get(timeout, TimeUnit.MILLISECONDS);
}
#Async
Future<Object> doSomething() {
//what i should do
//get ressources etc...
}
}
Don't forget to add #EnableAsync
It's also possible to do the same without #Async by implementing a Callable.
Edit: Keep in mind that it will wait until timeout but the thread running the task won't be interrupted. You will need to call Future.cancel when TimeoutException occurs. And in the task check for isInterrupted() to stop the processing. If you are calling an api be sure that isInterrupted() is checked.
allowCoreThreadTimeOut and timeout setting doesn't help cause it just allow work thread to be ended after some time without work (See javadocs)
You say your job waits infinitely for an external resource. I'am sure it's because you (or some third-party library you using) use sockets with time out infinite-by-default.
Also keep in mind what jvm ignores Thread.interrupt() when it blocked on socket.connect/read.
So find out witch socket library used in your task (and how exactly it used) and change it's default timeout settings.
As example: there is RestTemplate widely used inside Spring (in rest client, in spring social, in spring security OAuth and so on). And there is ClientHttpRequestFactory implementation to create RestTemplate instances. By default, spring use SimpleClientHttpRequestFactory which use JDK sockets. And by default all it's timeouts are infinite.
So find out where exactly you freeze, read it's docs and configure it properly.
P.S. If you don't have enough time and "feeling lucky" try to run your app with setting jvm properties sun.net.client.defaultConnectTimeout and
sun.net.client.defaultReadTimeout to some reasonable values (See docs for more details)
The keepAliveTime is just for cleaning out worker threads that hasn't been needed for a while - it doesn't have any impact on the execution time of the tasks submitted to the executor.
If whatever is taking time respects interrupts you can start a new thread and join it with a timeout, interrupting it if it doesn't complete in time.
public class SomeService {
#Scheduled(fixedRate = 5 * 60 * 1000)
public void doSomething() throws InterruptedException {
Thread taskThread = new TaskThread();
taskThread.start();
taskThread.join(120 * 000);
if(taskThread.isAlive()) {
// We timed out
taskThread.interrupt();
}
}
private class TaskThread extends Thread {
public void run() {
// Do the actual work here
}
}
}