How is asynchronous request processing in web frameworks like play! ,jersey ,spring different from the typical multi-threaded servers with pooled threads.
https://jersey.java.net/documentation/latest/async.html
https://www.playframework.com/documentation/2.3.x/JavaAsync
One thread is listening and heavy processing happens in new thread
Also non blocking requests in web can not be compared to non blocking IO in java using selectors where one thread can read multiple channels.
From a server side perspective what does non-blocking mean.A multi threaded server is already non-blocking.Is this wrong?
What are the real benefits/use-cases of asynchronous web frameworks over multi threaded servers(with thread pooling)?
One might be ability to keep the HTTP request(s) alive in some way(How?)
Traditionally, each connection requires a thread. Since HTTP connections can be long-lived, this approach cannot sustain too many concurrent users. At least that's what they say. But in practice, this does not seems to be a huge problem for java servers; you can always throw in more machines:)
With non-blocking, one thread can be used to server many connections; 10,000 shouldn't be a problem. So this approach might be more resilient for some applications. The catch is, programming in non-blocking is a little more difficult.
Related
I've been reading "Spring in Action" and got confused by the following description of Servlet-based frameworks (emphasis mine):
Typical Servlet-based web frameworks, such as Spring MVC, are blocking and multi-threaded in nature, using a single thread per connection. As requests are handled, a worker thread is pulled from a thread pool to process the request. Meanwhile, the request thread is blocked until it’s notified by the worker thread that it’s finished.
I was under the impression that servlet containers like Tomcat do something like this under the hood (very simplified, of course):
while (true) {
Socket socket = serverSocket.accept();
// the actual handling of request is done by Servlet-based framework and application code.
workerThreadPool.submit(() -> handleRequest(socket));
}
I mean, we do not create a thread for each connection but reuse existing worker threads.
Also I'm not sure what a request thread is. I thought it could be the thread in which we accept() connections, but this thread doesn't block until request has been processed. My understanding is that with Spring Boot and embedded Tomcat worker threads are the ones with names http-nio-8889-exec-* (see VisualVM screenshot). What is the request thread here?
So, my questions are:
What important things are missing in the pseudocode above?
Does Spring MVC (or servlet container) actually create a thread per each connection?
What is the role of request thread (or threads) and who creates them? How can I identify them in a typical Spring Boot application?
No sensible web server uses a thread per connection. Such a design won't scale with the number of connections.
Typical Servlet-based web frameworks, such as Spring MVC, are blocking
and multi-threaded in nature, using a single thread per connection. As
requests are handled, a worker thread is pulled from a thread pool to
process the request. Meanwhile, the request thread is blocked until
it’s notified by the worker thread that it’s finished.
Even though they say 'thread per connection' the description talks about thread pools.
The thread that 'accepts', blocks on the selector. As soon as new TCP/IP connection is established, the accepting thread unblocks. I'm not sure about the implementation; but one way of dealing with is to detach the connection from the accepting selector and use one more selectors that wait for read/write events. Of course each of these selectors will have their own thread.
My guess is that the http-nio-8889-exec are the request threads. It has been a very long time I used Spring, but in the past, Spring just piggybacked on the threads provided by the servlet container.
For frameworks like Node.js and ASP.NET Core, they are capable of processing requests asynchronously for I/O tasks without creating additional threads. Are java servlet containers also capable of doing this? If not, do java servlet containers wait I/O tasks in the thread until the request is fully processed?
Okay, I found the answer myself.
According to the Jakarta EE 9 documentation,:
There are two common scenarios in which a thread associated with a request can be sitting idle.
The thread needs to wait for a resource to become available or process data before building the response. For example, an application may need to query a database or access data from a remote web service before generating the response.
The thread needs to wait for an event before generating the response. For example, an application may have to wait for a Jakarta Messaging message, new information from another client, or new data available in a queue before generating the response.
These scenarios represent blocking operations that limit the scalability of web applications. Asynchronous processing refers to assigning these blocking operations to a new thread and retuning the thread associated with the request immediately to the container.
So, java servlet containers are capable of asynchronous processing. However they will create new thread for both I/O and CPU bond tasks, which is not the same model as Node.js and ASP.NET Core.
I'm currently working on an app which is heavily connected to maps. To display a map, we are generating a bunch of tiles in many threads, store them and get them if a user wants to see a certain part of the map.
The problem is, I'm naming threads that generate tiles a certain way, but then, when I want to get tiles to show a map, my servlet container's taking random threads from the pool, so the thread named for generating a tile ends up getting it from the storage. Of course, I could just rename the thread after generating a tile back, but I wonder if there is an alternative.
I wonder if I somehow can configure my servlet container for it to maybe kill threads after some time being idle or to create a new thread where I want to or to allocate several threads to work with this part of the code?
All I could find in terms of configuring servlet container is setting its min and max thread pool size, which I think won't help me.
The container is 100% in control over it's threading.
If you are attempting to manipulate the threading of the container then you are fighting a losing battle.
It is not possible to safely kill or stop threads on a running container, as this is incredibly unsafe, and will lead to many memory issues (leaks) and unclosed resources. The Thread.stop() method has been deprecated since Java 1.2.
Now that we have the negatives out of the way ...
Jetty is a 100% Async Java Web Server.
The classic assumption that 1 request uses 1 thread is wrong. (if you want this kind of behavior, then you should use Jetty 6 or older. Jetty versions older than 9.2 are now all EOL / End of Life)
When you use a Servlet call that is traditionally a blocking call, the Jetty server has to fake that blocking call to satisfy the API contract.
Even if using old school / traditional blocking Servlet APIs you'll still experience many situations where that 1 request has been handled by multiple threads over the lifetime of that 1 request.
If you want to work with the Servlet API and it's container then the first thing you should do is start to use both the Servlet Async Processing APIs and Servlet Async I/O APIs combined. Make sure you read about the gotchas on both APIs!
Async Processing will allow you to handle more processing of requests on the server side, not use the container threads that heavily, allow more control over how the threading behaves, will grant you better control over request timeouts, and even get notified of request/response error cases that you will always deal with on a web server.
Async I/O will allow you to only use a thread if there is content from the request/connection to read or if the connection allows a write. That connection will not consume a thread unless I/O is possible. This means more connections/requests per server, and ill behaving clients (slow, dead, problematic, etc) will not impact the behavior of your other clients by consuming threads that are not doing anything productive for you.
If you don't want to work with the Servlet API and do things your own way, then you'll have to manage your own Executor / ThreadGroup / ThreadPool that the server is unaware of. But that still means you'll need to use the Servlet Async Processing APIs to allow the 2 to coexist in harmony (you'll need to use the AsyncContext to inform the container that you are now taking control over the processing of the request, and then later inform it via the AsyncContext that you are done and the request is complete).
The biggest gotcha with this approach is that you cannot safely write to the HttpServletResponse from a thread that the container wasn't in control over.
Meaning the container dispatched on a thread to your application, that thread is the only one that can safely use the HttpServletResponse to write the response. You can have a different thread do the processing, a different thread provide the data to the HttpServletResponse, even a different thread that pumps the dispatch thread with content. But that thread you were dispatched to needs to be used to write.
This is the mixed threading behavior gotcha in the servlet spec. (you are in servlet async mode, on a different thread to process, but not using async mode to read/write.) It's a terribly complex, and ill defined, behavior in the servlet spec that leads to many issues, and I advise you to not chase this path.
This gotcha goes away if you also use the Servlet Async I/O APIs, but at that point the difference in the two above choices is negligible.
The scope/context of this question:
I am to develop a Java/Java EE based distributed server-side application that is scalable (scale-up, rather than scale-out).
My application comprises of servlets utilizing multiple instances of distributed back-end services for processing client requests. If I need to achieve more throughput, I want to be able to just add more instances of these distributed services (JVMs on the same or another machine) and (expect to) see an increase in throughput.
To achieve this, I was thinking of a loosely-coupled asynchronous system.
I thought I would use Async Servlets (servlet 3.0) and an application-managed thread-pool that places client requests on JMS queues, which would be picked by one of the distributed service instances and processed. The responses can be relayed back to the client using JMS, from the service instances to a response-thread in the servlet container.
However, an asynchronous system seems to be (obviously) more complex than a synchronous one (ex: error-handling and error-relaying to the client, request tracking etc). I am also worried about the future maintainability of the design/code.
So, a question arises Does it make sense to do this synchronously, while still remaining distributed, scalable and loosely-coupled ?
If the answer is yes, then pls also share possible ways of achieving this (while remaining 'constructive').
If I can do this well in a synchronous way, then it will simplify the entire system.
I dont want to add complexity to the system unnecessarily.
(Assuming it makes sense) One possible implementation I could think of is using RMI.
For ex: A service registry for the distributed service instances to register and have a load-balancer distribute the RMI calls across all the available instances. But it feels to be a old-generation solution. Are there any better options available ?
Edit:
Other details about the scope of this question:
The client-side is browser-based does not demand an asynchronous
server-side.
I dont need server-push.
At any time, I wont have more outstanding requests than max-worker-threads of the popular web servers (even Apache).
For the above reasons, the use-cases mentioned in a related question dont seem to apply to my scenario.
Loose coupling and distribution are independent of whether processing is synchronous or asynchronous.
With scalability, the matter is more complex. In a synchronous model, you will need one thread per pending request. If you need to scale to really high load (say, thousands of concurrent requests per server), an asynchronous model may scale better. To reap the benefit of that however, the entire processing, starting from the handling of incoming connections, needs to be done in an asynchronous way. There is little point to have a synchronous request processing thread delegate to a asynchronous thread pool, and blocking until that thread pool has computed the result - after all, the request thread could just as well have done the work himself.
If you need to return a response, I'd therefore go for synchronous request processing whenever scalabity permits (which it usually does).
Edit:
There are numerous ways to talk to the distributed backend servers. You might simply use EJB (which, if I recall correctly, uses RMI under the hood). Or, you might use webservices behind a load balancer.
I have written an application using embedded Jetty that makes network calls to other services.
I presume that the serving threads are idle whilst waiting for the network calls to complete.
Is there any way to have a worker thread that switches between requests to perform work that can be done at the current time and then when the network calls return also handle that? A request would be returned when all work has been completed for it.
I know this is a common paradigm, and I have used it for non-blocking TCP networking, but I'm unsure as to how to achieve this on a Java HTTP server whilst also waiting on external results.
Any links or explanations are appreciated.
Thanks
Update:
I'm using Membase and ElasticSearch (the only network calls). Membase returns "Future" objects and ElasticSearch returns "ListenableActionFuture". I'd like to be able to continue processing on a thread in response to these objects being returned.
You may take a look at Deft, which is single threaded, asynchronous, event driven web server.
Netty is a java library that allows you to do asynchronous networking.
http://www.jboss.org/netty
Netty supports http, but it is a fairly low level library.
A higher level library is finangle by twitter,
http://twitter.github.com/finagle/
Finangle is built on top of netty, but supports connection pooling, load balancing, and has a lot of other features. Finangle supports http.
If you want to do work at the same time as IO, I suggest you add a thread pool to perform the work. It is possible to re-use the existing threads but its a lot of extra work for possibly too little benefit.