Currently experimenting reactive programming with Spring 5.0.0.RC2, Reactor 3.1.0.M2 and Spring Boot 2.0.0.M2.
Wondering about the concurrency and threading model used by WebFlux and Reactor to properly code the application and handle the mutable state.
The Reactor doc states that the library is considered concurrency agnostic and mentions the Scheduler abstraction. The WebFlux doc does not give information.
Yet when using WebFlux through Spring Boot, a threading model is defined.
From my experimentations here is what I got:
The model is neither 1 event thread, nor 1 event thread + workers
Several thread pools are used
"reactor-http-nio-3" threads: probably one per core, handle the incoming HTTP requests
"Thread-7" threads: used by async requests to MongoDB or HTTP resources
"parallel-1" threads: one per core, created by Schedulers.parallel() from Reactor, used by delay operators and such
Shared mutable state must be synchronized by the application
ThreadLocal (for application state, MDC logging, etc) are not request scoped, so are not very interesting
Is this correct ? What is the concurrency and threading model of WebFlux: for example what are the default thread pools?
Thank you for the information
After the question, the present documentation does provide some clues about the concurrency model and the threads one could expect (but I still think that clearer/better descriptions of what happens under-the-scene from a multi-threading perspective would be highly appreciated by Spring newcomers).
It discusses the difference between Spring MVC and Spring WebFlux (1-thread-per-request model vs. event-loop):
In Spring MVC, and servlet applications in general, it is assumed that applications may block the current thread, e.g. for remote calls, and for this reason servlet containers use a large thread pool, to absorb potential blocking during request handling.
In Spring WebFlux, and non-blocking servers in general, it is assumed that applications will not block, and therefore non-blocking servers use a small, fixed-size thread pool (event loop workers) to handle requests.
Invoking a Blocking API
But notice that Spring MVC apps can also introduce some asynchronicity (cf., Servlet 3 Async). And I suggest this presentation for a discussion wrt Servlet 3.1 NIO and WebFlux.
Back to the docs: it also suggests that, when working with reactive streams, you have some control:
What if you do need to use a blocking library?
Both Reactor and RxJava provide the publishOn operator to continue
processing on a different thread.
(For more details on this, refer to scheduling in Reactor)
It also discusses the threads you may expect in WebFlux applications (bold is mine):
Threading Model
What threads should you expect to see on a server running with Spring WebFlux?
On a "vanilla" Spring WebFlux server (e.g. no data access, nor other optional dependencies), you can expect one thread for the server, and several others for request processing (typically as many as the number of CPU cores). Servlet containers, however, may start with more threads (e.g. 10 on Tomcat), in support of both servlet, blocking I/O and servlet 3.1, non-blocking I/O usage.
The reactive WebClient operates in event loop style. So you’ll see a small, fixed number of processing threads related to that, e.g. "reactor-http-nio-" with the Reactor Netty connector. However if Reactor Netty is used for both client and server, the two will share event loop resources by default.
Reactor and RxJava provide thread pool abstractions, called Schedulers, to use with the publishOn operator that is used to switch processing to a different thread pool. The schedulers have names that suggest a specific concurrency strategy, e.g. "parallel" for CPU-bound work with a limited number of threads, or "elastic" for I/O-bound work with a large number of threads. If you see such threads it means some code is using a specific thread pool Scheduler strategy.
Data access libraries and other 3rd party dependencies may also create and use threads of their own.
In part, you can configure the details of the threading model via configuration
To configure the threading model for a server, you’ll need to use server-specific config APIs, or if using Spring Boot, check the Spring Boot configuration options for each server. The WebClient can be configured directly. For all other libraries, refer to their respective documentation.
Moreover, as e.g. the discussion Default number of threads in Spring boot 2.0 reactive webflux configuration
highlights,
The default number of threads for request handling is determined by the underlying web server; by default, Spring Boot 2.0 is using Reactor Netty, which is using Netty's defaults
it is a matter of default components and their defaults (and overall configuration, including that injected transparently through annotations) -- which may also change across versions of Spring/Boot and corresponding dependencies.
Said that, your guesses seem correct.
Related
I have a service which is deployed as microservice and a mongodb with some documents with "few" states, for example: READY, RUNNING, COMPLETED. I need to pick the documents with state "READY" and then process them. But with multiple instances running there is high possibility of processing the "duplicates". I have seen the below thread, but it is only concerned about one instance only picking up tasks.
Spring boot Webservice / Microservices and scheduling
Above talks about solution using Hazlecast and mongodb. But what I am looking at is that all instances wait for the lock, get their own "documents (non-duplicates) and process them. I have checked the various documents and unfortunately I am not able to find any solution.
One of the option I thought is to introduce Kafka, where we can "assign" specific tasks to specific consumers. But before opting would like to see if we any solutions which can be implemented using simple methods such as database locks etc. Any pointers towards this are highly appreciated.
We have a flow which we would like to implement with Reactive programming using Spring Boot 2 WebFlux. Currently we have no experience with Reactive programming.
As part of this flow we are going to create on or more HTTP requests (I guess using WebClient) and also read some data from DB.
We are considering to use AWS DynamoDB but as far as I understand the Java SDK does not support reactive API. This read will be a blocking I/O operation, my question is whether there is a benefit for implementing part of this flow with WebFlux? More generally, does a single blocking I/O operation in the flow eliminates all the benefit that we get from implementing with reactive programming?
Based on your question reactive is the idle way to deal with blocking operation especially IO (network, file and etc...)
you can use a library that implements this api in a reactive way or wrap a blocking request with a reactive api, this usually done by placing the blocking op on anther thread pool
in spring webflux you can achieve something similar like
#GetMapping
public Mono<Response> getResponse() {
return Mono.fromCallable(() -> blockingOp())
.publishOn(Schedulers.elastic());
}
publishOn in that case will cause all this flow to happen on another thread, you can choose dedicated thread pool as your choice
from the docs, elastic is a
Scheduler that dynamically creates ExecutorService-based Workers and caches the thread pools, reusing them once the Workers have been shut down.
The following may not answer your question fully, but might be a little helpful. There is a question mentioned in the FAQ for the Spring Framework 5, which is,
What if there is no reactive library for my database?
The answer to this is:
One suggestion for handling a mix of blocking and non-blocking code
would be to use the power of a microservice boundary to separate the
blocking backend datastore code from the non blocking front-end API.
Alternatively, you may also go with a worker thread pool for blocking
operations, keeping the main event loop non-blocking that way.
I think someone from Pivotal might be the right person to give more insights on this.
I'm trying to understand Spring WebFlux. The things I've found so far are reactive at the core, no Servlet API, no thread per request, HTTP 2, server pushes, application/stream+json.
But what is the difference between asynchronous calls in Spring MVC? I mean in Spring MVC when you return Future, DefferedResult and etc you get logic in the request handler (controller method) executed in a separate thread, so you can benefit from saving thread pool resources for dispatching requests as well.
So could you please highlight differences related to that? Why WebFlux is better here?
Thank you for your time very much!
The Servlet async model introduces an async boundary between the container threads (1 Servlet request/thread model) and the processing of the request in your application. Processing can happen on a different thread or wait. In the end, you have to dispatch back to a container thread and read/write in a blocking way (InputStream and OutputStream are inherently blocking APIs).
With that model, you need many threads to achieve concurrency (because many of those can be blocked waiting for I/O). This costs resources and it can be a tradeoff, depending on your use case.
With non-blocking code, you only need a few threads to process a lot of requests concurrently. This is a different concurrency model; like any model, there are benefits and tradeoffs coming with it.
For more information about that comparison, this Servlet vs. Reactive stacks talk should be of interest.
Servlet API is blocking I/O which requires 1 thread per HTTP request. Spring MVC async relies on Servlet APIs which only provides async behavior between container threads and request processing threads but not end to end.
Spring WebFlux on the other hand achieves concurrency by a fixed number of threads by using HTTP sockets and pushing chunks of data at a time through the sockets. This mechanism is called event loop, an idea made popular by Node.js. Such an approach is scalable and resilient. Spring 5's spring-webflux uses the event loop approach to provide async behavior.
More can be read from
Servlet vs. Reactive
Spring Boot performance battle
Comparing WebFlux with Spring Web MVC
I'd like for incoming Java servlet web requests to invoke RabbitMQ using the RPC approach as described here.
However, I'm not sure how to properly reuse callback queues between requests, as per the RabbitMQ tutorial linked above creating a new callback queue per every request is inefficient (RabbitMQ may not cope even if using the Queue TTL feature).
There would generally be only 1-2 RPC calls per every servlet request, but obviously a lot of servlet requests per second.
I don't think I can share the callback queues between threads, so I'd want at least one per each web worker thread.
My first idea was to store the callback queue in a ThreadLocal, but that can lead to memory leaks.
My second idea was to store them in a session, but I am not sure they will serialize properly and my sessions are currently not replicated/shared between web servers, so it is IMHO not a good solution.
My infrastructure is Tomcat / Guice / Stripes Framework.
Any ideas what the most robust/simple solution is?
Am I missing something in this whole approach, and thus over-complicating things?
Note 1- This question relates to the overall business case described here - see option 1.
Note 2 - There is a seemingly related question How to setup RabbitMQ RPC in a web context, but it is mostly concerned with proper shutdown of threads created by the RabbitMQ client.
The scope/context of this question:
I am to develop a Java/Java EE based distributed server-side application that is scalable (scale-up, rather than scale-out).
My application comprises of servlets utilizing multiple instances of distributed back-end services for processing client requests. If I need to achieve more throughput, I want to be able to just add more instances of these distributed services (JVMs on the same or another machine) and (expect to) see an increase in throughput.
To achieve this, I was thinking of a loosely-coupled asynchronous system.
I thought I would use Async Servlets (servlet 3.0) and an application-managed thread-pool that places client requests on JMS queues, which would be picked by one of the distributed service instances and processed. The responses can be relayed back to the client using JMS, from the service instances to a response-thread in the servlet container.
However, an asynchronous system seems to be (obviously) more complex than a synchronous one (ex: error-handling and error-relaying to the client, request tracking etc). I am also worried about the future maintainability of the design/code.
So, a question arises Does it make sense to do this synchronously, while still remaining distributed, scalable and loosely-coupled ?
If the answer is yes, then pls also share possible ways of achieving this (while remaining 'constructive').
If I can do this well in a synchronous way, then it will simplify the entire system.
I dont want to add complexity to the system unnecessarily.
(Assuming it makes sense) One possible implementation I could think of is using RMI.
For ex: A service registry for the distributed service instances to register and have a load-balancer distribute the RMI calls across all the available instances. But it feels to be a old-generation solution. Are there any better options available ?
Edit:
Other details about the scope of this question:
The client-side is browser-based does not demand an asynchronous
server-side.
I dont need server-push.
At any time, I wont have more outstanding requests than max-worker-threads of the popular web servers (even Apache).
For the above reasons, the use-cases mentioned in a related question dont seem to apply to my scenario.
Loose coupling and distribution are independent of whether processing is synchronous or asynchronous.
With scalability, the matter is more complex. In a synchronous model, you will need one thread per pending request. If you need to scale to really high load (say, thousands of concurrent requests per server), an asynchronous model may scale better. To reap the benefit of that however, the entire processing, starting from the handling of incoming connections, needs to be done in an asynchronous way. There is little point to have a synchronous request processing thread delegate to a asynchronous thread pool, and blocking until that thread pool has computed the result - after all, the request thread could just as well have done the work himself.
If you need to return a response, I'd therefore go for synchronous request processing whenever scalabity permits (which it usually does).
Edit:
There are numerous ways to talk to the distributed backend servers. You might simply use EJB (which, if I recall correctly, uses RMI under the hood). Or, you might use webservices behind a load balancer.