I have a simple (thats what I think) Spring boot application. There are 4 layers:
Rest Controller
Application Service (called by the Rest Controller)
Domain Service (called by Application Service. It connects to the database - repository layer)
Adapter Service (called by Application Service for outbound calls via Hystrix)
Now the problem is that it can only handle a max of 15 parallel calls. If any additional REST API request arrives while these calls are being processed, it makes it to the Application Service layer and then waits. Once one of those 15 parallel call returns, then the new request proceeds to make call to the Domain Service Layer and return.
I tried multiple things:
Increasing spare threads for the server in application.properties file
server.tomcat.min-spare-threads=1000
server.tomcat.max-connections=1000
server.tomcat.max-threads=1000
Once I do this, I see the # of http-nio-* threads increase to 1000 but the hanging issue is not fixed.
I found this snippet online to customize the tomcat container but it didn't help either:
#Bean
public WebServerFactoryCustomizer<TomcatServletWebServerFactory> containerCustomizer() {
return new WebServerFactoryCustomizer<TomcatServletWebServerFactory>() {
#Override
public void customize(TomcatServletWebServerFactory factory) {
factory.addConnectorCustomizers(new TomcatConnectorCustomizer() {
#Override
public void customize(Connector connector) {
Arrays.stream(connector.getProtocolHandler().findUpgradeProtocols())
.filter(upgradeProtocol -> upgradeProtocol instanceof Http2Protocol)
.map(upgradeProtocol -> (Http2Protocol) upgradeProtocol)
.forEach(http2Protocol -> {
http2Protocol.setMaxConcurrentStreamExecution(1000);
});
}
});
}
};
}
I tried configuring the thread pool via code
#Bean(name = "taskExecutor")
public TaskExecutor threadPoolTaskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(200);
executor.setMaxPoolSize(300);
executor.setQueueCapacity(300);
executor.setThreadNamePrefix("anniversary");
executor.initialize();
System.out.println("******* name " + executor.getThreadNamePrefix());
System.out.println("********** core pool size " + executor.getCorePoolSize());
return executor;
}
But none of this helps and I believe the issue is not with the number of threads but elsewhere since the request is not able to go from one service to another. There are hundreds of http-nio-* threads in waiting state and when a new request comes in, its assigned its own thread and I can see that in the Debug mode.
Any pointers, help, tips are much appreciated. What resource is required for service to service invocation by Spring boot?
I believe your observation is right - it's most likely not tomcat who's the bottleneck here. From what you write, would rather look at the domain service. Is the domain service doing some communication with the database or talking to something else over the network (for example over HTTP)?
If you happen to do database in there, check for spring's datasource configuration. There is going to be a database connection pool with a limited number of maximum concurrent connections to the database. Once these connections are all in use, threads that want to talk to the DB will be blocked until one of the connection becomes free again.
Similar connection pools are in place with many other things that talk over network (e.g. Apache HTTP Client also has a connection pool that can be configured).
That's where i would look next.
Cheers,
Matthias
Related
I have a Spring Boot service that needs to listen to messages across multiple RMQ vhosts. So far I only need to consume messages, though in the short future I might need to publish messages to a third vhost. For this reason I've moved towards explicit configuration of the RMQ connection factory - one connection factory per vhost.
Looking at the documentation the PooledChannelConnectionFactory fits my needs. I do not need strict ordering of the messages, correlated publisher confirms, or caching connections to a single vhost. Everything I do with rabbit is take a message and update an entry in the database.
#Bean
PooledChannelConnectionFactory pcf() throws Exception {
ConnectionFactory rabbitConnectionFactory = new ConnectionFactory();
//Set the credentials
PooledChannelConnectionFactory pcf = new PooledChannelConnectionFactory(rabbitConnectionFactory);
pcf.setPoolConfigurer((pool, tx) -> {
if (tx) {
// configure the transactional pool
}
else {
// configure the non-transactional pool
}
});
return pcf;
}
What I require assistance with is understanding what the difference between the transactional and non-transactional pool is. My understanding of RMQ and AMQP is that everything is async unless you build RPC semantics ontop of it (reply queues and exchanges). Because of that how can this channel pool have transactional properties?
My current approach is to disable one of the configurations by setting min/max to 0, and set the other to a min/max of 1. I do not expect to have extreme volume through the service, and I expect to be horizontally scaling the application which will scale the capacity to consume messages. Is there anything else I should be considering?
The pools are independent; you won't get any transactional channels as long as you don't use a RabbitTemplate with channelTransacted set to true, so there is no need to worry about configuring the pool like that.
Transactions can be used, for example, to atomically send a series of messages (all sent or none sent if the transaction is rolled back). Useful if you are synchronizing with some other transaction, such as JDBC.
I have a spring boot project, deploying in two servers and using nginx. One method in the project will do:
set some key-values in redis
insert something in db
After 1, I want to do 2 in async way.
One solution is to let doDB() be a springboot #async method:
Class A {
public void ***() {
doRedis() // 1.set some key-values in redis
doDB() // 2.insert something in db
}
}
Class B {
#async
doDB()
}
Another solution is to send message to MQ:
Class A {
public void ***() {
doRedis() // 1.set some key-values in redis
sendMessage()
}
}
Class B {
onMessage(){
doDB()
}
}
If Class A and B are both in the spring boot project, just deploying this project in two servers. I think using #async is enough, there is no need to use MQ to achieve the async way because there is no difference between server one to do Class B doDB() and server two to do Class B doDB(). If class B is in another project, then using MQ is good because it's decoupling for project one doing redis work and project two doing db work.
Is it right? Thanks!
Basically, you are right, if it is going to be in the same application within the same server, no need for MQ because async is already has a queue. But there are some key points you should be decided on even if in the same application
if you care about ordering message queue is more meaningful, you can use async in this case too but you have to configure the thread pool to use only one thread to process async
if you care about losing messages and if some bad things happen before processing messages, you should use an MQ that saves messages to the disk or somewhere to process the rest of the messages later on
if your application gets a lot of requests and you did not carefully set threads in the async thread pool, you could get overflow errors or other problems with using machine resources
Choose within capabilities within your application, do not over-engineer from day one, you spin up from what you have and what you already tested
I have a Spring RESTful service using a Tomcat web servlet that processes 2 different types of data and therefore has 2 rest controllers for each type of data. Controller #1 has the potential to perform an intensive task using lots of memory so I would like to allow up to, for instance, 10 connections on this controller. But if all 10 connections are processing on controller #1 I would also like controller #2 to have its own thread pool so it can continue processing while controller #1 is full.
The proper way to configure Tomcat is set its properties in the application.yml as described here in the spring docs.
To set the total number of max connection one would use:
server.tomcat.max-connections: 10
server.tomcat.max-threads: 10
But, this configures the maximum number of connections/threads for the entire application, both controllers combined. I would need each controller to have its own thread pool and its own number of maximum connections. Is this possible?
You can't *. Spring Boot sets up an embedded Tomcat servlet container and registers a DispatcherServlet. The entire Tomcat pool of threads is used to handle all requests going through the DispatcherServlet (or any other servlets/filters registered).
* You should create a ThreadPoolTaskExecutor or ExecutorService bean for each type of data, then inject them into your #Controller beans appropriately and dispatch all the work to them.
#Controlller
class FirstController {
private final ExecutorService threadPool;
public FirstController(#Qualifier("first-type-data") ExecutorService threadPool) {
this.threadPool = threadPool;
}
#RequestMapping("/endpoint1")
public CompletableFuture<Foo> handleEndpoint() {
CompletableFuture<Foo> foo = new CompletableFuture<>();
threadPool.submit(() -> {
// handle all your business logic
foo.complete(...);
});
return foo;
}
}
The Spring MVC "user space stack" doesn't really know about connections. You can pass around the HttpServletRequest and maintain your own count. Once you hit a threshold, you could send back an appropriate response directly without starting any of the business logic.
When using Spring webflux with Mono or Flux return type, the http connecting thread is parked/released while the connection waits for the response. Thus, the connection is not taking max-connections.
Question: how can I test/proof that the connection is really released while waiting for the response, and not blocking max-connections?
I already enabled DEBUG logging, but that did not show anything regarding this question.
#RestController
public class MyServlet {
#GetMapping("/")
public Mono<String>/Flux<String> test() {
return Mono.just(service.blocking());
}
}
#Service
public class SlowService {
public String blocking() {
TimeUnit.SECONDS.sleep(10);
return "OK";
}
}
Or is that incorrect at all, and I'd have to use:
Mono.fromCallable(() -> service.blocking()).subscribeOn(Schedulers.elastic());
But still, how can I see from the logs that the connection gets parked correctly?
To test, I'm using server.tomcat.max-threads=5. I'm trying to rewrite my blocking service so that those threads are not blocked during the sleep, and thus more than 5 connections can reach my service concurrently.
There are 2 thread pools, the forkjoin pool that will handle all the regular work. Then you have the Schedular pool that will handle scheduled tasks.
return Mono.just(service.blocking());
This will block one of the threads in the ForkJoinPool so less events can be handled by webflux hence slowing down your service.
Mono.fromCallable(() -> service.blocking()).subscribeOn(Schedulers.elastic());
This will assign the task and "offload" it to the scheduler pool of threads so another thread pool will handle this and not hog one of the ForkJoinPool threads.
How to test this? You need to load test your service, or as most people do, trust the framework to do what it is set out to do and trust that the spring team has tested their side of things.
I am using Spring Webflux with Spring data jpa using PostgreSql as backend db.
I don't want to block the main thread while making db calls like find and save.
To achieve the same, I have a main scheduler in Controller class and a jdbcScheduler service classes.
The way I have defined them is:
#Configuration
#EnableJpaAuditing
public class CommonConfig {
#Value("${spring.datasource.hikari.maximum-pool-size}")
int connectionPoolSize;
#Bean
public Scheduler scheduler() {
return Schedulers.parallel();
}
#Bean
public Scheduler jdbcScheduler() {
return Schedulers.fromExecutor(Executors.newFixedThreadPool(connectionPoolSize));
}
#Bean
public TransactionTemplate transactionTemplate(PlatformTransactionManager transactionManager) {
return new TransactionTemplate(transactionManager);
}
}
Now, while doing a get/save call in my service layer I do:
#Override
public Mono<Config> getConfigByKey(String key) {
return Mono.defer(
() -> Mono.justOrEmpty(configRepository.findByKey(key)))
.subscribeOn(jdbcScheduler)
.publishOn(scheduler);
}
#Override
public Flux<Config> getAllConfigsAfterAppVersion(int appVersion) {
return Flux
.fromIterable(configRepository.findAllByMinAppVersionIsGreaterThanEqual(appVersion))
.subscribeOn(jdbcScheduler)
.publishOn(scheduler);
}
#Override
public Flux<Config> addConfigs(List<Config> configList) {
return Flux.fromIterable(configRepository.saveAll(configList))
.subscribeOn(jdbcScheduler)
.publishOn(scheduler);
}
And in controller, I do:
#PostMapping
#ResponseStatus(HttpStatus.CREATED)
Mono<ResponseDto<List<Config>>> addConfigs(#Valid #RequestBody List<Config> configs) {
return configService.addConfigs(configs).collectList()
.map(configList -> new ResponseDto<>(HttpStatus.CREATED.value(), configList, null))
.subscribeOn(scheduler);
}
Is this correct? and/or there is a way better way to do it?
What I understand by:
.subscribeOn(jdbcScheduler)
.publishOn(scheduler);
is that task will run on jdbcScheduler threads and later result will be published on my main parallel scheduler. Is this understanding correct?
Your understanding is correct with regards to publishOn and subscribeOn (see reference documentation in the reactor project about those operators).
If you call blocking libraries without scheduling that work on a specific scheduler, those calls will block one of the few threads available (by default, the Netty event loop) and your application will only be able to serve a few requests concurrently.
Now I'm not sure what you're trying to achieve by doing that.
First, the parallel scheduler is designed for CPU bound tasks, meaning you'll have few of them, as many (or a bit more) as CPU cores. In this case, it's like setting your threadpool size to the number of cores on a regular Servlet container. Your app won't be able to process a large number of concurrent requests.
Even if you choose a better alternative (like the elastic Scheduler), it will be still not as good as the Netty event loop, which is where request processing is scheduled natively in Spring WebFlux.
If your ultimate goal is performance and scalability, wrapping blocking calls in a reactive app is likely to perform worse than your regular Servlet container.
You could instead use Spring MVC and:
use usual blocking return types when you're dealing with a blocking library, like JPA
use Mono and Flux return types when you're not tied to such libraries
This won't be non-blocking, but this will be asynchronous still and you'll be able to do more work in parallel without dealing with the complexity.
IMHO, there a way to execute this operation doing a better use of resources from machine. Following documentation you can wrap the call in other Thread and with this you can continue your execution.