CODE-1
new NioServerSocketChannelFactory(Executors.newCachedThreadPool(), Executors.newCachedThreadPool(),WORKER_SIZE)
CODE-2
OrderedMemoryAwareThreadPoolExecutor executor = new OrderedMemoryAwareThreadPoolExecutor(48, 0, 0, 1, TimeUnit.SECONDS);
pipeline.addLast("executor", new ExecutionHandler(executor));
If IO worker thread pool size (default is 2*count of cpu) can be set from CODE-1, what is the purpose of adding executer (a thread pool) to pipeline in CODE-2 ?
IO operations are done from worker threads. Does that mean, a client with slow connection or bad network keeps IO worker thread busy until data is completely sent ? If so, increasing WORKER_SIZE would help me prevent latencies ?
Slow Connections does not affect Netty threads in NIO normally (check the update note).
Some points about Netty server internal threads
by default there will be only one Boss Thread per server port, and it
will accept connection and handover the connection to worker
thread.
to be precise: WORKER_SIZE is the maximum number of NioWorker
runnables a server can have. for example If the server has
only one connection, then there will be 1 worker thread. If number of connections are increasing and it can not be assigned to next worker (active connections > WORKER_SIZE), then connections will be assigned to a worker in a round robin fashion.
If IO worker thread pool size (default is 2*count of cpu) can be set from CODE-1, what is the purpose of adding executer (a thread pool) to pipeline in CODE-2 ?
If your upstream tasks are blocking then you should execute them in a separate thread pool using a execution handler. Otherwise Nio read/write will not work on time (latency?). I think having a execution handler will help to reduce the latency than setting big value to WORKER_SIZE.
IO operations are done from worker threads. Does that mean, a client with slow connection or bad network keeps IO worker thread busy until data is completely sent ? If so, increasing WORKER_SIZE would help me prevent latencies ?
Generally speaking, increasing the WORKER_SIZE >= number of cpu * 2 does not help because,
NIO is non blocking and If I am not mistaken, its CPU intensive.For CPU intensive task CPU * 2 number of threads are chosen mostly.
Update:
NioWorker runs a loop with selector.select(500ms) to receive OP_READ, selector.select with timeout a blocking call and if most of the connections are slow, performance may reduce?. You can reduce the timeout in org.jboss.netty.channel.socket.nio.SelectorUtil.java and test.
The thread pool[s] you are adding in CODE-1 are for the boss threads, and worker threads. The boss threads accept connections and pass it on to worker thread to handle.
The executor you add in CODE-2 is for handling the messages read by the worker threads.
Slow connections will not affect performance since you are using a non-blocking architecture (NIO) - which is set in Netty to not block (it could if it wanted to)
Related
I have a service class
#Path("/")
public class ABC {
#Path(/process/{param})
public String processRequest(#PathParam("param") String param){
Thread t = new Thread(()->{
// Do some processing with the param parmeter
System.out.println("Processing started for -> "+ param);
//Do many more things
//DB transactions,etc.
});
t.start();
return "Your request will be processed";
}
}
I accept some parameter and start processing it in a new thread and at the same time so that it should complete the processing within 30 secs, I break my connection with the client by acknowledging him that his request will be processed.
It works fine and till now without any issues, Currently, it can process more than 5k requests. The problem starts when there is a lot of requests come at the same time maybe more than 50k so my application creates a new thread for every new request which causes the application to allocate a lot of memory and also sometimes makes JVM memory to exhaust.
Is there any another way by which I can immediately start the processing without bothering for the number of requests and process all the requests within 30 secs and also limit the no of threads active working threads.
One way I found was the Producer-Consumer implementation in which I can accept all the requests and put simultaneously into the producers and my consumers pick up the request and start processing it, For this implementation i need to specify the maximum no of request which can be accepted by producer(Ex : 100 000) and no of consumers which can process the request(Ex : 1000) so that only 1000 threads are active and process one after another but issue with this approach is that if any of the consumer (working) thread if locks due to some reason and if not released then there are only the remaining unlocked threads left to process the request and the incoming request is continuously increasing in the producers. Only increasing the no of consumers creates more working thread but at the same time there can be a lot of locked threads processing the task.
Please let me know any another approach by which I can do it.
Note : All the request should be processed within 30 secs and if unable to do then it fails the success criteria.
You probably want a queueing mechanism, like RabbitMq.
Your application will run like this:
request -> push to queue -> return ACK to client
queue -> worker threads.
The queue consumer speed is determined by your worker threads speed, so you will never exhausted your system.
Under load, there're lots of message will be queued, mean while your workers reliably takes messages from queue and process them.
Your need is to serve a large no of (may be concurrent) requests and also want to control no of threads spawned (max cap on number of threads). Your best option is to use ExecuterService which is kind of managed thread pool where you can specify thread pool size and submit multiple Runnable or Callable objects for processing.
ExecutorService executorService = Executors.newFixedThreadPool(10);
It’s explained very well here. Thread Concurrency using ExecutorService in Java 8
You can use queuing system to put requests in queue and acknowledge client about processing and later you can process queue
Got question about WebSphere and cannot find anything in the documentation...
What happens when WebContainer thread pool is fully used and new request is received? I am talking about the situation when all are used and we reached max thread pool size meaning that new one may not be created to process the request.
Will the request:
- fail immediately and the response will contain some kind of error?
- WAS will somehow "queue" the request for given period and process it when one of the threads is back in the pool? Some kind of error/timeout may still happen if the wait time is too long?
- WAS will "queue" the request indefinitely and the timeout might only happen on the User side (web browser/app)?
The exact behavior is probably undocumented so that the details can be changed between releases in order to improve behavior. You can likely infer the behavior by looking at javacores or by gleaning information from various documents, for example the BoundedBuffer section of this IBM WebSphere Application Server Performance Cookbook document:
The thread pool request buffer is essentially a backlog in front of
the thread pool. If the thread pool is at its maximum size and all of
the threads are dispatched, then work will queue in the requestBuffer.
The maximum size of the requestBuffer is equal to the thread pool
maximum size; however, if the unit of work is executed on the thread
pool with a blocking mode of EXPAND_WHEN_QUEUE_IS_FULL_ERROR_AT_LIMIT
or EXPAND_WHEN_QUEUE_IS_FULL_WAIT_AT_LIMIT, then the maximum size is
ThreadPoolMaxSize * 10. When the requestBuffer fills up, then
WSVR0629I is issued (although only the first time this happens per JVM
run per thread pool). When the requestBuffer is full, work will either
wait or throw a ThreadPoolQueueIsFullException, depending on how the
unit of work is executed.
In practice, that means after maxThreads threads are busy executing work, an additional maxThreads requests will be queued in a bounded buffer, and when that buffer is full, the socket thread will block until it can queue the work, which means that further incoming requests will be blocked until a thread becomes available and makes space in the bounded buffer.
I'm implementing a binary protocol above TCP/IP and using Netty to achieve this. My problem is that the performance is rather poor (600 msg/s). I'm connecting as a client to a server with one connection only. When I investigated running instance with JTop I saw that Netty was using 1 worker thread very heavily and the other 5 worker threads are doing nothing (0% ussage). I was digging on the web and all I found is mention of ExecutionHandler. But why should I use this if those 6 worker threads should be enough. Or am I misunderstanding how Netty uses these threads?
My Netty init code:
this.channelFactory = new NioClientSocketChannelFactory(this.executors, DaemonExecutors.newCachedDaemonThreadPool(), 1, 6);
this.clientBootstrap = new ClientBootstrap(channelFactory);
this.channelGroupHandler = new ClientChannelGroupHandler(this.channels);
this.clientBootstrap.getPipeline().addLast("ChannelGroupHandler", this.channelGroupHandler);
Thanks for any hints
Matous
NIO, or rather the non-blocking version of NIO ("New" I/O) allows you to use a single thread for multiple connections, since the thread doesn't block (hence the name) on the read/write operations. Blocking I/O requires a thread for each connection, as the blocking would prevent you from handling traffic between different connections.
This allows you to perform more efficient communication, since you no longer have thread overhead for one.
A decent tutorial is available here (the original Oracle tutorial seems to have vanished from the face of the Google).
The reason you only see one worker thread being used is because you are making only a single connection to the server. Had you made multiple connections, more worker threads would have been used.
If each connection's work is suited for parallelization, then you can implement a handler that uses threads internally, but Netty won't to do that for you.
As for the NIO/OIO distinction, it's true that the idea of NIO is to have one thread handling the events for multiple connections. However, this doesn't mean one thread will handle the all the work. The "single thread" only dispatches work to other (i.e. worker) threads.
Here is an excerpt from the Netty doc:
How threads work There are two types of threads in a
NioServerSocketChannelFactory; one is boss thread and the other is
worker thread.
Boss threads Each bound ServerSocketChannel has its own boss thread.
For example, if you opened two server ports such as 80 and 443, you
will have two boss threads. A boss thread accepts incoming connections
until the port is unbound. Once a connection is accepted successfully,
the boss thread passes the accepted Channel to one of the worker
threads that the NioServerSocketChannelFactory manages.
Worker threads One NioServerSocketChannelFactory can have one or more
worker threads. A worker thread performs non-blocking read and write
for one or more Channels in a non-blocking mode.
Normally in tomcat, a thread will be running and when a request comes in,it will assign the responsibility of servicing the request to a thread from thread pool.
Does it matter if that main thread is blocking or non blocking in terms of scalability?
Non Blocking IO has the following advantages:
Highly Scalable : Because no-more you require one thread per client. It can effectively support more number of clients.
High Keep Alive : Blocking IO requires to block until the keepalive time for the next request. Non-Blocking being notification model, it can support high keepalive times.
Better Performance on High Load : Because in blocking IO has one thread per connection, it requires n threads for n connections. As the value n increases, the performance degrades because more thread context switching.
When an incoming request is processed in tomcat it will assign the connection to a thread from its thread pool.
What matters here is to run the thread as fast as possible. You typically run blocking io calls in this thread, for file io, db and so on.
You need to adjust the size of this thread pool apropriatley to handle your expected traffic.
Esentially when using the Java EE servlet spec you are forced into handling your requests in a one thread per incoming connection manner.
There are a few non blocking frameworks out there. Check out http://www.playframework.org/ and Jetty ( Jetty nonblocking by default? )
I have 3 ThreadPoolExecutors in my system.
One for Netty's Master process, another for netty's worker process and last one for processing ad-hoc processing (sending request to mail server).
ExecutorService bossExecutors = Executors.newFixedThreadPool(1,
new ServerThreadFactory("netty-boss"));
ExecutorService workerExecutors = Executors.newFixedThreadPool(10,
new ServerThreadFactory("netty-worker"));
ChannelFactory factory = new NioServerSocketChannelFactory(
bossExecutors,
workerExecutors,
Runtime.getRuntime().availableProcessors());
ExecutorService mailExecutor = Executors.newFixedThreadPool(40);
This works perfectly fine until mailExecutor starts making request to mail server. Until, that batch requests using mailExecutor, generally making 5000+ requests to mail server is completed, netty threads get blocked.
I don't understand why netty threads seem to be getting blocked that time since, I have allocated definite thread pools. During that time, Netty can't even process single request.
Any idea why it's happening or what I'm doing wrong?
Can you provide a thread-dump ?
jstack <pid>
Also you should never use a fixed threadpool for the worker / poss threadpool. Use a cached one, this way you can be sure you never get into any starvation. You should specify the worker count with the 3 argument in the constructor.
It sounds like a scheduling issue. You have 40 threads under heavy load, vs the availableProcessors number of threads for handling Netty work (what is your availableProcessors() count at the time you create your factory?).
So it could just be that the Netty threads are too few and are being starved since they never happen to be picked for execution compared to the 40 threads handling the mail work.
It may also be that for some reason, your worker threads are blocked on the mail threads finishing, perhaps due to some shared object that is being synchronized on (is there some queue or list of mail to be sent that the netty threads need to write to, and which the mail threads have locked while they send?).