We Keep Coding

Process multiple request at same time

I have a service class
#Path("/")
public class ABC {
#Path(/process/{param})
public String processRequest(#PathParam("param") String param){
Thread t = new Thread(()->{
// Do some processing with the param parmeter
System.out.println("Processing started for -> "+ param);
//Do many more things
//DB transactions,etc.
});
t.start();
return "Your request will be processed";
}
}
I accept some parameter and start processing it in a new thread and at the same time so that it should complete the processing within 30 secs, I break my connection with the client by acknowledging him that his request will be processed.
It works fine and till now without any issues, Currently, it can process more than 5k requests. The problem starts when there is a lot of requests come at the same time maybe more than 50k so my application creates a new thread for every new request which causes the application to allocate a lot of memory and also sometimes makes JVM memory to exhaust.
Is there any another way by which I can immediately start the processing without bothering for the number of requests and process all the requests within 30 secs and also limit the no of threads active working threads.
One way I found was the Producer-Consumer implementation in which I can accept all the requests and put simultaneously into the producers and my consumers pick up the request and start processing it, For this implementation i need to specify the maximum no of request which can be accepted by producer(Ex : 100 000) and no of consumers which can process the request(Ex : 1000) so that only 1000 threads are active and process one after another but issue with this approach is that if any of the consumer (working) thread if locks due to some reason and if not released then there are only the remaining unlocked threads left to process the request and the incoming request is continuously increasing in the producers. Only increasing the no of consumers creates more working thread but at the same time there can be a lot of locked threads processing the task.
Please let me know any another approach by which I can do it.
Note : All the request should be processed within 30 secs and if unable to do then it fails the success criteria.

You probably want a queueing mechanism, like RabbitMq.
Your application will run like this:
request -> push to queue -> return ACK to client
queue -> worker threads.
The queue consumer speed is determined by your worker threads speed, so you will never exhausted your system.
Under load, there're lots of message will be queued, mean while your workers reliably takes messages from queue and process them.

Your need is to serve a large no of (may be concurrent) requests and also want to control no of threads spawned (max cap on number of threads). Your best option is to use ExecuterService which is kind of managed thread pool where you can specify thread pool size and submit multiple Runnable or Callable objects for processing.
ExecutorService executorService = Executors.newFixedThreadPool(10);
It’s explained very well here. Thread Concurrency using ExecutorService in Java 8

You can use queuing system to put requests in queue and acknowledge client about processing and later you can process queue

Java: Does making async calls increase number of threads?

So let's say I have a web app and for every request we spawn a new thread. Hundreds of requests come in, somewhere in the web server code we make synchronous calls to several services, we block and wait. This approach bloats the number of threads that we have as the sync calls create a bottleneck.
Supposedly, if we switch these calls to async requests we get rid of the bottleneck as the threads can continue and the callbacks will handle whatever needs to happen.
As far as I understand, in Java, in order to make an async call we spawn a new thread that makes the network call and contains the callback (I won't be implementing this, I'm assuming thats how some of the Java http libraries work).
So my question, how is this solving the problem of many threads? Async requests end up creating more threads (one for each request) and then go to sleep until something is returned, doesn't this create many sleeping threads?
The problem I am trying to solve is that at some point, when there's too many threads, the JVM explodes.

Specifically in web service / servlet environments:
In the simplest configuration, common web servers (Jetty, Tomcat) are configured with a fixed number of threads, or range of number of threads. If more requests arrive than there are threads, then those requests will pile up in the kernel connection queue. A thread accepts a connection and does all the work. When the response is sent, the thread is available for another connection. Adding your own thread pool or executor service won't help that.
In more complex configurations, the web container accepts connections on one pool of threads, and then dispatches the work on another, with a queue in between. Then, instead of blocking clients on connect, or having them fail to connect, they just wait.
In async Servlet processing, such as the JAX-RS #suspended AsyncResponse object, you get to control the details of this yourself. The servlet calls you with a data structure that includes the connection. Your code can put that object into some queue (possibly just the queue built into an Executor Service), and return. That frees the web server thread to accept another container. Your threads, probably from an Executor Service, work through the queue, processing requests and sending responses.
What you never do is create an unbounded number of threads.

Asynchronous means the request is processed by another thread. It doesn't have to be a dedicated thread, let alone a new thread.
For instance, consider JAX-RS asynchronous client callbacks:
target().path("http://example.com/resource/")
.request().async().get(new InvocationCallback<String>() {
#Override
public void completed(String dataFromBackendServer) {
respondWith(dataFromBackendServer);
}
#Override
public void failed(Throwable throwable) {
respondWithError(throwable);
}
});
Here, the InvocationCallback is executed in a thread provided by the JAX-RS implementation, that waits for a response to any pending backend request, then processes that response using the appropriate InvocationCallback. Because a single thread can wait for any number of pending backend requests, fewer threads are needed.
That said, synchronous processing is often easier to implement, and while it does not scale quite as well as asynchronous processing, it scales sufficiently for many applications. That is, unless you have thousands of concurrent requests, the plain old synchronous processing model will do.

There is no problem like too many threads, servers always have a thread pool. They assign a thread from the thread pool to each request, if a thread is not available server just makes the request socket wait in the queue of the ServerSocket.
The problem that async request processing in Servlet 3 is trying to solve is the less utilization of resources due to blocking of request processing threads.
So if there are long running requests which just wait for I/O, they are put on hold till the response is received from I/O channel, and that thread is assigned to another request waiting in the socket queue.
This provides us with better resource (CPU mainly) utilization, and more resource through put as more requests (ones with short duration) are served per second.

How sync and async request processing differs in Tomcat?

I cannot figure out what difference is between sync and async calls in Tomcat.
Everywhere I use NIO. I have thousand connections managed by few Tomcat threads. When long sync request incomes a thread borrows from Tomcat thread pool and processes the request. This thread waits for long process to be completed and then writes result to HTTPResponse. So resources are wasted just for awaiting. When long async request incomes then Tomcat thread creates separate thread and long process starts within this new thread and Tomcat thread returns to pool alomost immedately.
Am i understood right? If so I don't see any difference between sync and async modes because in both modes same amount of threads is used

The difference is "pull" versus "push". Yes, you are correct, either way a thread must be allocated for doing the work.
But with sync request you would have to create the worker thread manually and poll the task result from client, whereas with async the server will push the result to the client when the task completes.
The latter is slightly more efficient because your server doesn't have to process many poll requests per result.

Thanks, figured out. Sync request is a case when one thread borrows for one request and awaits and pulls neccessary data. Async request is a case when there is just one thread separated from requests which waits for data and pushes it to requests async contexts, i.e. client's output streams. When client produces aync request it doesn't creates any additional threads but its async context stands to subscribers list. When data appears then one thread walks through this list and writes data to every async context. Result is - sync request means one thread per request, async request means one (or little more) thread for many simultaneous requests

Netty client uses only one thread

I'm implementing a binary protocol above TCP/IP and using Netty to achieve this. My problem is that the performance is rather poor (600 msg/s). I'm connecting as a client to a server with one connection only. When I investigated running instance with JTop I saw that Netty was using 1 worker thread very heavily and the other 5 worker threads are doing nothing (0% ussage). I was digging on the web and all I found is mention of ExecutionHandler. But why should I use this if those 6 worker threads should be enough. Or am I misunderstanding how Netty uses these threads?
My Netty init code:
this.channelFactory = new NioClientSocketChannelFactory(this.executors, DaemonExecutors.newCachedDaemonThreadPool(), 1, 6);
this.clientBootstrap = new ClientBootstrap(channelFactory);
this.channelGroupHandler = new ClientChannelGroupHandler(this.channels);
this.clientBootstrap.getPipeline().addLast("ChannelGroupHandler", this.channelGroupHandler);
Thanks for any hints
Matous

NIO, or rather the non-blocking version of NIO ("New" I/O) allows you to use a single thread for multiple connections, since the thread doesn't block (hence the name) on the read/write operations. Blocking I/O requires a thread for each connection, as the blocking would prevent you from handling traffic between different connections.
This allows you to perform more efficient communication, since you no longer have thread overhead for one.
A decent tutorial is available here (the original Oracle tutorial seems to have vanished from the face of the Google).

The reason you only see one worker thread being used is because you are making only a single connection to the server. Had you made multiple connections, more worker threads would have been used.
If each connection's work is suited for parallelization, then you can implement a handler that uses threads internally, but Netty won't to do that for you.
As for the NIO/OIO distinction, it's true that the idea of NIO is to have one thread handling the events for multiple connections. However, this doesn't mean one thread will handle the all the work. The "single thread" only dispatches work to other (i.e. worker) threads.
Here is an excerpt from the Netty doc:
How threads work There are two types of threads in a
NioServerSocketChannelFactory; one is boss thread and the other is
worker thread.
Boss threads Each bound ServerSocketChannel has its own boss thread.
For example, if you opened two server ports such as 80 and 443, you
will have two boss threads. A boss thread accepts incoming connections
until the port is unbound. Once a connection is accepted successfully,
the boss thread passes the accepted Channel to one of the worker
threads that the NioServerSocketChannelFactory manages.
Worker threads One NioServerSocketChannelFactory can have one or more
worker threads. A worker thread performs non-blocking read and write
for one or more Channels in a non-blocking mode.

Blocking vs Non blocking main thread in tomcat

Normally in tomcat, a thread will be running and when a request comes in,it will assign the responsibility of servicing the request to a thread from thread pool.
Does it matter if that main thread is blocking or non blocking in terms of scalability?

Non Blocking IO has the following advantages:
Highly Scalable : Because no-more you require one thread per client. It can effectively support more number of clients.
High Keep Alive : Blocking IO requires to block until the keepalive time for the next request. Non-Blocking being notification model, it can support high keepalive times.
Better Performance on High Load : Because in blocking IO has one thread per connection, it requires n threads for n connections. As the value n increases, the performance degrades because more thread context switching.

When an incoming request is processed in tomcat it will assign the connection to a thread from its thread pool.
What matters here is to run the thread as fast as possible. You typically run blocking io calls in this thread, for file io, db and so on.
You need to adjust the size of this thread pool apropriatley to handle your expected traffic.
Esentially when using the Java EE servlet spec you are forced into handling your requests in a one thread per incoming connection manner.
There are a few non blocking frameworks out there. Check out http://www.playframework.org/ and Jetty ( Jetty nonblocking by default? )