I am working on an application in which thousands of tasks associated
with hundreds of devices, each task requiring, < 5ms to begin execution, and
taking on average 100ms to complete.
The conditions are as such:
Each device can only process a single
task at a time, e.g., one task must finish running on its assigned
device prior to subsequent task's being processeed.
The scheduler should be efficient. Currently, processing a given device's
work queue takes longer than the sum of it's tasks.
Here is basic description of the current implementation:
Each device contains a work queue which is filled with tasks associated with
that device.
When a task is enqueued, that device's work queue is placed into a
global run queue (a queue of queue's). The global run queue is consumed by a worker thread
which dequeue's the device's task objects, processes one, then places
the device queue at the back of the global run queue. When that given device
has been dequeued again, the worker thread checks to see if the task has completed,
if so, the next task is executed. This process continues, until all device queues
have been depleted of tasks in the global runqueue.
Any suggestions for improvements? Have I stated this clearly? If not, please let me know, and I'll do my best to clarify.
Thanks for taking the time to look this over. Regards.
How about something like the ExecutorCompletionService together with a ThreadPoolExecutor. This gives you callback on completion, which you can use to submit subsequent jobs, and a managed threadpool of executors, which you can tweak to improve throughput.
I suggest you run your code through a profiler to see which threads are currently blocking (sounds like it will be your producer device threads). I can recommend YourKit Java Profiler, however its not free.
The problem with your design is that only one task is run at a time, which means that any device is idle while any other device is in use.
Since there are hundreds of devices, it would probably not be a good idea to assign a thread to each of these devices, but a thread pool could very well be used.
Related
I want to implement a single-producer - multi-consumer logic where each consumer processing time depends on a hardware response.
**EDIT
I have a Set of objects (devices). Each object (device) corresponds to a hardware real unit I want to simulate in software.
My main class distributes a list of tasks to each device. Each task takes a certain time to complete - which I want to have control, in order to simulate the hardware operation. Each device object has its own SingleThreadExecutorService service executor to manage its own queued tasks. A Sleep on a task of a specific device object should not interfere on main, or other devices object's performance.
So far things are working but I am not sure how to get a future from the tasks without blocking the main thread with a while(!future.isDone()). When I do it, two problems occur:
task 1 is submitted to device[ 1 ].executor. Tasks 1 sleeps to simulate hardware operation time.
task 2 should be submitted to device[ 2 ].executor as soon as task 1 is submitted, but it won't, because main thread is hold while waiting for task 1 to return a Future. This issue accumulates delay on the simulation since every task added causes the next device to have to wait for the previous to complete, instead of running simultaneously.
Orange line indicates a command to force device to wait for 1000 milliseconds.
When Future returns, it then submits a new task to device 2, but it is already 1 second late, seen in blue line. And so on, green line shows the delay increment.
If I don't use Future to get when tasks were finished, the simulation seems to run correctly. I couldn't find a way to use future.isDone() without having to create a new thread just to check it. Also, I would really be glad if someone could advice me how to proceed in this scenario.
If your goal is to implement something where each consumer task is talking to a hardware device during the processing of its task, then the run method of the task should simply talk to the device and block until it receives the response from the device. (How you do that will depend on the device and its API ...)
If your goal is to do the above with a simulated device (i.e. for testing purposes) then have the task call Thread.sleep(...) to simulate the time that the device would take to respond.
Based on your problem description (as I understand it), the PausableSchedulerThreadPoolExecutor class that you have found won't help. What that class does is to pause the threads themselves. All of them.
UPDATE
task 2 should be submitted to device[ 2 ].executor as soon as task 1 is submitted, but it won't, because main thread is hold while waiting for task 1 to return a Future.
That is not correct. The Future object is returned immediately ... when the task is submitted.
You mistake (probably) is that the main thread is calling get on the Future. That will block. But the point is that is your main thread actually needs to call get on the Future before submitting the next task then it is essentially single-threaded.
Real solution: figure out how to break that dependency that makes your application single threaded. (But beware: if you pass the Future as a parameter to a task, then the corresponding worker thread may block. Unless you have enough threads in the thread pool you could end up with starvation and reduced concurrency.)
I'm new to multithreading... multithreading is used to improve performance, however how can this be so if the processor is already working as fast as it can for a single thread?
To explain:
In a single threaded environment User A starts a task which takes 1 second - the task is completed in 1 second.
User B starts the same task a fraction of a second later and has to wait for user A, therefore user Bs task completes in almost 2 seconds. Now, if this was a multithreaded environment would both tasks not be run similtaeously causing both to take 2 seconds? ...e.g. part of task A done, then part of task B, then part of task A again, then part of task B, ...till eventually both tasks complete in around two seconds?
Is it only faster if there is more than one processor? Or should multithreading be reserved for when there is a big task being dealt with and smaller tasks need to pop in and out of existence during this time?
If the task is 100% CPU bound, and you have only one CPU core, then multithreading will make things slower. If you have more than one CPU core, clearly you can allocate as many threads as you have cores to improve performance. If the tasks interact with anything external (I/O) then multithreading can increase performance to a point. While one thread is waiting for I/O to complete, other threads can be performing CPU-based processing.
The classic example is a program that does computation and also displays a GUI. You update the GUI on one thread (the event thread) and do all processing on other "background" threads. The event thread does nothing but handle user interactions. If you don't do this then when a user-requested operation takes appreciable time the GUI stops responding. In this situation, you would run multithreaded even on a single-core system.
How to tune an application's use of threads depends on a lot of factors and could take up an entire textbook.
Ok now consider that Task A of your's needs a particular resource (ex a network file or a user input) to complete its work. Now say the resource needed by the Task A is not currently available so what happens next in a single threaded environment is that Task A has control of the CPU so it'll wait for the resource to be available and at this time the CPU will be idle which means we're wasting an important resource i.e, the CPU time while waiting for some other resource. But if the Task A has been implemented in multithreaded environment when Task A waits for the resource, it(Task A thread) gets suspended till the resource become available and the CPU time can be used efficiently to execute other tasks. Hope this helps :)
I'm trying to set up a job that will run every x minutes/seconds/milliseconds/whatever and poll an Amazon SQS queue for messages to process. My question is what the best approach would be for this. Should I create a ScheduledThreadPoolExecutor with x number of threads and schedule a single task with scheduleAtFixedRate method and just run it very often (like 10 ms) so that multiple threads will be used when needed, or, as I am proposing to colleagues, create a ScheduledThreadPoolExecutor with x number of threads and then create multiple scheduled tasks at slightly offset intervals but running less often. This to me sounds like how the STPE was meant to be used.
Typically I use Spring/Quartz for this type of thing but that's out of at this point.
So what are your thoughts?
I recommend that you use long polling on SQS, which makes your ReceiveMessage calls behave more like calls to take on a BlockingQueue (which means that you won't need to use a scheduled task to poll from the queue - you just need a single thread that polls in an infinite loop, retrying if the connection times out)
Well it depends on the frequency of tasks. If you just have to poll on timely interval and the interval is not very small, then ScheduledThreadPoolExecutor with scheduleAtFixedRate is a good alternative.
Else I will recommend using netty's HashedWheelTimer. Under heavy tasks it gives the best performance. Akka and play uses this for scheduling. This is because STPE for every task adding takes O(log(n)) where as HWT takes O(1).
If you have to use STPE, I will recommend one task at a rate else it results in excess resource.
Long Polling is like a blocking queue only for a max of 20 seconds after which the call returns. Long polling is sufficient if that is the max delay required between poll cycles. Beyond that you will need a scheduledExector.
The number of threads really depends on how fast you can process the received messages. If you can process the message really fast you need only a single thread. I have a setup as follows
SingleThreadScheduledExecutor with scheduleWithFixedDelay executes 5 mins after the previous completion
In each execution messages are retrieved in batch from SQS till there are no more messages to process (remember each batch receive a max of 10 messages).
The messages are processed and then deleted from queue.
For my scenario single thread is sufficient. If the backlog is increasing (for example, a network operation is required for each message which may involve waits), you might want to use multiple threads. If one processing node become resource constrained you could always start another instance (EC2 perhaps) to add more capacity.
There is one fixed thread pool (let it be with size=100), that I want to use for all tasks across my app.
It is used to limit server load.
Task = web crawler, that submits first job to thread pool.
That job can generate more jobs, and so on.
One job = one HTTP I/O request.
Problem
Suppose that there is only one executing task, that generated 10000 jobs.
Those jobs are now queued in thread pool queue, and all 100 threads are used for their execution.
Suppose that I now submit a second task.
The first job of the second task is 10001th in the queue.
It will be executed only after the 10000 jobs that the first task queued up.
So, this is a problem - I don't want the second task to wait so long to start its first job.
Idea
The first idea on my mind is to create a custom BlockingQueue and pass it to the thread pool constructor.
That queue will hold several blocking queues, one for each task.
Its take method will then choose a random queue and take an item from it.
My problem with this is that I don't see how to remove an empty queue from this list when its task is finished. This would mean some or all workers could get blocked on the take method, waiting for jobs from tasks that are finished.
Is this the best way to solve this problem?
I was unable to find any patterns for it in books or on the Internet :(
Thank you!
I would use multiple queues and draw from a random of the queues that contains items. Alternatively you could prioritize which queue should get the highest priority.
I would suggest using a single PriorityBlockingQueue and using the 'depth' of the recursive tasks to compute the priority. With a single queue, workers get blocked when the queue is empty and there is no need for randomization logic around the multiple queues.
I have a sort of complex problem like below.
- we have a real time system with large number threads requirement. In order to optimize the performance, we are thinking of following design.
create a thread pool executor with max number of threads
each thread is used to create scheduled executor service.
now the tasks are being assigned to these executor services evenly based on load
BUT the biggest problem is, if one of the task in the queue contains a sleep (for few secs), it blocks the corresponding Schedule executor service thread for that duration and subsequently all the following tasks in that queue.
In this regard, please suggest me how to suspend the execution of the task with sleep OR overriding the sleep somehow and rejoin/schedule the task again to the queue.
Thanks in advance
Seshu
Assuming I understand your question, your Schedule Executor service threads have a deadline requirement, but the actual workers can sleep for an unknown length of time, possibly throwing off the timing of the Schedule Executors. From your description I'm guessing what you want is for a task that needs to sleep to actually stop, save progress information and then requeue itself for the remainder of the work to be rescheduled at some future time. You'd have to build this into your application architecture.
Alternatively, you could have the scheduler threads launch the worker tasks in their own separate threads, letting them sleep as necessary, with one scheduler thread collecting all the worker terminations.
To get a better answer you're going to have to provide more information about what you're trying to accomplish.
Tasks which sleep are inherently unfriendly for running in any kind of bounded thread pool. The sleep is explicitly telling the thread that it must do nothing for a period of time.
If possible, split the task into 2 (or more parts), eliminating the sleep completely. Get the first half-task to schedule the second task with an appropriate delay.
Failing that, you could consider increasing the size of your thread pool somewhat - either setting a much larger cap to its size, or possibly even eliminating the cap altogether (not recommended for a server than might end up with many clients).
Alternatively, move the tasks with sleep statements in them into their own Scheduled executor. Then, they'll delay each other, but better-behaved tasks, with no wait statements in them, will get preferential treatment.