I've been reading "Spring in Action" and got confused by the following description of Servlet-based frameworks (emphasis mine):
Typical Servlet-based web frameworks, such as Spring MVC, are blocking and multi-threaded in nature, using a single thread per connection. As requests are handled, a worker thread is pulled from a thread pool to process the request. Meanwhile, the request thread is blocked until it’s notified by the worker thread that it’s finished.
I was under the impression that servlet containers like Tomcat do something like this under the hood (very simplified, of course):
while (true) {
Socket socket = serverSocket.accept();
// the actual handling of request is done by Servlet-based framework and application code.
workerThreadPool.submit(() -> handleRequest(socket));
}
I mean, we do not create a thread for each connection but reuse existing worker threads.
Also I'm not sure what a request thread is. I thought it could be the thread in which we accept() connections, but this thread doesn't block until request has been processed. My understanding is that with Spring Boot and embedded Tomcat worker threads are the ones with names http-nio-8889-exec-* (see VisualVM screenshot). What is the request thread here?
So, my questions are:
What important things are missing in the pseudocode above?
Does Spring MVC (or servlet container) actually create a thread per each connection?
What is the role of request thread (or threads) and who creates them? How can I identify them in a typical Spring Boot application?
No sensible web server uses a thread per connection. Such a design won't scale with the number of connections.
Typical Servlet-based web frameworks, such as Spring MVC, are blocking
and multi-threaded in nature, using a single thread per connection. As
requests are handled, a worker thread is pulled from a thread pool to
process the request. Meanwhile, the request thread is blocked until
it’s notified by the worker thread that it’s finished.
Even though they say 'thread per connection' the description talks about thread pools.
The thread that 'accepts', blocks on the selector. As soon as new TCP/IP connection is established, the accepting thread unblocks. I'm not sure about the implementation; but one way of dealing with is to detach the connection from the accepting selector and use one more selectors that wait for read/write events. Of course each of these selectors will have their own thread.
My guess is that the http-nio-8889-exec are the request threads. It has been a very long time I used Spring, but in the past, Spring just piggybacked on the threads provided by the servlet container.
Related
For frameworks like Node.js and ASP.NET Core, they are capable of processing requests asynchronously for I/O tasks without creating additional threads. Are java servlet containers also capable of doing this? If not, do java servlet containers wait I/O tasks in the thread until the request is fully processed?
Okay, I found the answer myself.
According to the Jakarta EE 9 documentation,:
There are two common scenarios in which a thread associated with a request can be sitting idle.
The thread needs to wait for a resource to become available or process data before building the response. For example, an application may need to query a database or access data from a remote web service before generating the response.
The thread needs to wait for an event before generating the response. For example, an application may have to wait for a Jakarta Messaging message, new information from another client, or new data available in a queue before generating the response.
These scenarios represent blocking operations that limit the scalability of web applications. Asynchronous processing refers to assigning these blocking operations to a new thread and retuning the thread associated with the request immediately to the container.
So, java servlet containers are capable of asynchronous processing. However they will create new thread for both I/O and CPU bond tasks, which is not the same model as Node.js and ASP.NET Core.
I'm currently working on an app which is heavily connected to maps. To display a map, we are generating a bunch of tiles in many threads, store them and get them if a user wants to see a certain part of the map.
The problem is, I'm naming threads that generate tiles a certain way, but then, when I want to get tiles to show a map, my servlet container's taking random threads from the pool, so the thread named for generating a tile ends up getting it from the storage. Of course, I could just rename the thread after generating a tile back, but I wonder if there is an alternative.
I wonder if I somehow can configure my servlet container for it to maybe kill threads after some time being idle or to create a new thread where I want to or to allocate several threads to work with this part of the code?
All I could find in terms of configuring servlet container is setting its min and max thread pool size, which I think won't help me.
The container is 100% in control over it's threading.
If you are attempting to manipulate the threading of the container then you are fighting a losing battle.
It is not possible to safely kill or stop threads on a running container, as this is incredibly unsafe, and will lead to many memory issues (leaks) and unclosed resources. The Thread.stop() method has been deprecated since Java 1.2.
Now that we have the negatives out of the way ...
Jetty is a 100% Async Java Web Server.
The classic assumption that 1 request uses 1 thread is wrong. (if you want this kind of behavior, then you should use Jetty 6 or older. Jetty versions older than 9.2 are now all EOL / End of Life)
When you use a Servlet call that is traditionally a blocking call, the Jetty server has to fake that blocking call to satisfy the API contract.
Even if using old school / traditional blocking Servlet APIs you'll still experience many situations where that 1 request has been handled by multiple threads over the lifetime of that 1 request.
If you want to work with the Servlet API and it's container then the first thing you should do is start to use both the Servlet Async Processing APIs and Servlet Async I/O APIs combined. Make sure you read about the gotchas on both APIs!
Async Processing will allow you to handle more processing of requests on the server side, not use the container threads that heavily, allow more control over how the threading behaves, will grant you better control over request timeouts, and even get notified of request/response error cases that you will always deal with on a web server.
Async I/O will allow you to only use a thread if there is content from the request/connection to read or if the connection allows a write. That connection will not consume a thread unless I/O is possible. This means more connections/requests per server, and ill behaving clients (slow, dead, problematic, etc) will not impact the behavior of your other clients by consuming threads that are not doing anything productive for you.
If you don't want to work with the Servlet API and do things your own way, then you'll have to manage your own Executor / ThreadGroup / ThreadPool that the server is unaware of. But that still means you'll need to use the Servlet Async Processing APIs to allow the 2 to coexist in harmony (you'll need to use the AsyncContext to inform the container that you are now taking control over the processing of the request, and then later inform it via the AsyncContext that you are done and the request is complete).
The biggest gotcha with this approach is that you cannot safely write to the HttpServletResponse from a thread that the container wasn't in control over.
Meaning the container dispatched on a thread to your application, that thread is the only one that can safely use the HttpServletResponse to write the response. You can have a different thread do the processing, a different thread provide the data to the HttpServletResponse, even a different thread that pumps the dispatch thread with content. But that thread you were dispatched to needs to be used to write.
This is the mixed threading behavior gotcha in the servlet spec. (you are in servlet async mode, on a different thread to process, but not using async mode to read/write.) It's a terribly complex, and ill defined, behavior in the servlet spec that leads to many issues, and I advise you to not chase this path.
This gotcha goes away if you also use the Servlet Async I/O APIs, but at that point the difference in the two above choices is negligible.
I have an application which consumes a message from RabbitMQ using Spring AMQP.
I have to implement Threads in consumer to handle the request. If my threads pool are available , i am consuming the message using threads i will process the message.
I have a question where what happens when all the threads are busy. I
dont have a threads to process it. Will the message be consumed from
RabbitMQ ? Will it wait till my thread pool becomes available. How do
handle this using spring amqp?
Is there any thread logic to be implemented from Spring AMQP side as well?
Please suggest.
You should not add your own threading in your listener, it will cause messages to be ack'd early and potentially lost. Instead, use the container's concurrentConsumers property to determine how many threads to use.
I suggest you read about all the configuration options before asking questions like this here.
How is asynchronous request processing in web frameworks like play! ,jersey ,spring different from the typical multi-threaded servers with pooled threads.
https://jersey.java.net/documentation/latest/async.html
https://www.playframework.com/documentation/2.3.x/JavaAsync
One thread is listening and heavy processing happens in new thread
Also non blocking requests in web can not be compared to non blocking IO in java using selectors where one thread can read multiple channels.
From a server side perspective what does non-blocking mean.A multi threaded server is already non-blocking.Is this wrong?
What are the real benefits/use-cases of asynchronous web frameworks over multi threaded servers(with thread pooling)?
One might be ability to keep the HTTP request(s) alive in some way(How?)
Traditionally, each connection requires a thread. Since HTTP connections can be long-lived, this approach cannot sustain too many concurrent users. At least that's what they say. But in practice, this does not seems to be a huge problem for java servers; you can always throw in more machines:)
With non-blocking, one thread can be used to server many connections; 10,000 shouldn't be a problem. So this approach might be more resilient for some applications. The catch is, programming in non-blocking is a little more difficult.
I was reading about the new springframework 3.2 feature, Servlet 3 Async support here:
http://blog.springsource.org/2012/05/07/spring-mvc-3-2-preview-introducing-servlet-3-async-support
and I have a question.
The blog article indicates that spring now supports:
Controller returning a Callable
The callable gets invoked in a background thread (TaskExecutor)
The response remains open (request.startAsync()).
The stated motivation is to avoid eating up all of the web container threads. However, my understanding is this:
The web container thread pool is tunable.
There is a hard limit on that thread pool because it's preferable to degrade serving rather than bring down the process by exhausting all resources (as would happen with, say, an unbounded thread pool and large enough number of requests).
The point of servlet-3/comet is to limit the number of open threads while not limiting the number of open connections to a web server (because threads are expensive and sockets are cheap)
So given this, the feature Spring is proposing makes very little sense to me. Are they not just hacking an unbounded thread pool on top of the container's bounded pool? Does that not miss the point of limiting the number of threads?