We Keep Coding

SimpleAsyncTaskExecutor with a queue

I have a need to run some threads concurrently, but need to force each process to run in a new Thread (this is due to some ThreadLocal bleed that I don't have full control over). To do so, I have been using the SimpleAsyncTaskExecutor. However, the issue with this is that it doesn't maintain a queue that allows new tasks to be submitted once it's reached the concurrency limit. What I really need to do is have functionality like the SimpleAsyncTaskExecutor but where tasks can still be submitted even after the concurrency limit has been reached - I just want those tasks to wait in the queue until another slot frees up. This is what I have right now:
SimpleAsyncTaskExecutor taskExecutor = new SimpleAsyncTaskExecutor();
taskExecutor.setConcurrencyLimit(maxThreads);
return taskExecutor;
Is there some out-of-the-box solution for this, or do I need to write something custom?

To ensure you need to execute every task in a new Thread, You are basically against use of any ThreadPool (ThreadLocal behavior in a ThreadPool is something you need to get rid of, sooner or later).
To overcome this, you can simply produce something like this,
class ThreadPerTaskExecutor implements Executor {
public void execute(Runnable r) {
Thread t = new Thread(r);
t.start();
try {
t.join();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
which executes the Runnable always in a new Thread.
Coming to a crude implementation, We can do something like
final Executor executor = new ThreadPerTaskExecutor();
final ExecutorService service = Executors.newFixedThreadPool(3);
for (int i = 0; i < 100; i++) {
service.submit(new Runnable() {
public void run() {
try {
System.out.println("Executed inside Thread pool with concurrency level 3"
+ Thread.currentThread().toString());
executor.execute(new Runnable() {
public void run() {
try {
Thread.sleep(3000); //Some expensive operations here.
System.out.println(
"Executed inside new Thread always" + Thread.currentThread().toString());
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
} catch (Exception e) {
e.printStackTrace();
}
}
});
}
This can be improved with lambdas as well after Java 8. Hope this sheds the basic idea.

Is there some out-of-the-box solution for this, or do I need to write something custom?
I think there is no out-of-the-box solution for this, and you need to write your own code for this.
You can extend the SimpleAsyncTaskExecutor for simpler/quicker implementation. Example:
public class SimpleAsyncQueueTaskExecutor extends SimpleAsyncTaskExecutor {
private Queue<Runnable> queue = new ConcurrentLinkedQueue<Runnable>();
private AtomicInteger concurrencyValue = new AtomicInteger(0);
private void checkAndExecuteFromQueue() {
int count = concurrencyValue.get();
if (isThrottleActive() && !queue.isEmpty() &&
(count < getConcurrencyLimit())) {
Runnable task = queue.poll();
concurrencyValue.incrementAndGet();
doExecute(new ConcurrencyThrottlingRunnable(task));
}
}
private void afterExecute(Runnable task) {
queue.remove(task);
concurrencyValue.decrementAndGet();
// Check and execute other tasks
checkAndExecuteFromQueue();
}
#Override
public void execute(Runnable task, long startTimeout) {
Assert.notNull(task, "Runnable must not be null");
if (isThrottleActive() && startTimeout > TIMEOUT_IMMEDIATE) {
queue.offer(task);
checkAndExecuteFromQueue();
} else {
doExecute(task);
}
}
private class ConcurrencyThrottlingRunnable implements Runnable {
private final Runnable target;
public ConcurrencyThrottlingRunnable(Runnable target) {
this.target = target;
}
#Override
public void run() {
try {
this.target.run();
}
finally {
afterExecute(this.target);
}
}
}
This example code just add a queue, and override the execute method.
Hope this help.

Wait for multiple AsyncTask to complete

I am parallelizing my operation by splitting it in the exact number of cores available and then, by start the same number of AsyncTask, performing the same operation but on different portions of data.
I am using executeOnExecutor(AsyncTask.THREAD_POOL_EXECUTOR, ...) in order to parallelize the execution of them.
I would like to know when every thread finishes its job so that combine all results and perform further operations.
How can I do?

You could also simply decrement a counter in a shared object as part of onPostExecute. As onPostExecute runs on the same thread (the main thread), you won't have to worry about synchronization.
UPDATE 1
The shared object could look something like this:
public class WorkCounter {
private int runningTasks;
private final Context ctx;
public WorkCounter(int numberOfTasks, Context ctx) {
this.runningTasks = numberOfTasks;
this.ctx = ctx;
}
// Only call this in onPostExecute! (or add synchronized to method declaration)
public void taskFinished() {
if (--runningTasks == 0) {
LocalBroadcastManager mgr = LocalBroadcastManager.getInstance(this.ctx);
mgr.sendBroadcast(new Intent("all_tasks_have_finished"));
}
}
}
UPDATE 2
According to the comments for this answer, OP is looking for a solution in which he can avoid building a new class. This can be done by sharing an AtomicInteger among the spawned AsyncTasks:
// TODO Update type params according to your needs.
public class MyAsyncTask extends AsyncTask<Void,Void,Void> {
// This instance should be created before creating your async tasks.
// Its start count should be equal to the number of async tasks that you will spawn.
// It is important that the same AtomicInteger is supplied to all the spawned async tasks such that they share the same work counter.
private final AtomicInteger workCounter;
public MyAsyncTask(AtomicInteger workCounter) {
this.workCounter = workCounter;
}
// TODO implement doInBackground
#Override
public void onPostExecute(Void result) {
// Job is done, decrement the work counter.
int tasksLeft = this.workCounter.decrementAndGet();
// If the count has reached zero, all async tasks have finished.
if (tasksLeft == 0) {
// Make activity aware by sending a broadcast.
LocalBroadcastManager mgr = LocalBroadcastManager.getInstance(this.ctx);
mgr.sendBroadcast(new Intent("all_tasks_have_finished"));
}
}
}

You should use a CountDownLatch. Here the documentation with examples:
java.util.concurrent.CountDownLatch
Basically you give a reference of CountDownLatch to your threads, and each of them will decrement it when finished:
countDownLatch.countDown();
The main thread will wait on the termination of all threads using:
countDownLatch.await();

First, add this class to your project
public abstract class MultiTaskHandler {
private int mTasksLeft;
private boolean mIsCanceled = false;
public MultiTaskHandler(int numOfTasks) {
mTasksLeft = numOfTasks;
}
protected abstract void onAllTasksCompleted();
public void taskComplete() {
mTasksLeft--;
if (mTasksLeft==0 && !mIsCanceled) {
onAllTasksCompleted();
}
}
public void reset(int numOfTasks) {
mTasksLeft = numOfTasks;
mIsCanceled=false;
}
public void cancel() {
mIsCanceled = true;
}
}
Then:
int totalNumOfTasks = 2; //change this to the number of tasks that you are running
final MultiTaskHandler multiTaskHandler = new MultiTaskHandler(totalNumOfTasks) {
#Override
protected void onAllTasksCompleted() {
//put the code that runs when all the tasks are complete here
}
};
Then in each task - when completed, add the line: multiTaskHandler.taskComplete();
Example:
(new AsyncTask<Void,Void,Void>() {
#Override
protected Void doInBackground(Void... voids) {
// do something...
return null;
}
#Override
protected void onPostExecute(Void aVoid) {
multiTaskHandler.taskComplete();
}
}).execute();
You can use multiTaskHandler.cancel() if you want to cancel the code that runs when all the tasks have completed. For instance - if you have an error (don't forget to also cancel all the other tasks).
* This solution will not pause the main thread!

Another Option could be to store all your new threads in an Array.
Then you could iterate over the Array and wait with thread[i].join for the thread to finish.
see join()
http://developer.android.com/reference/java/lang/Thread.html#Thread(java.lang.Runnable)
When the Iteration is finished all your threads are done and you can work on

RX Merge operator is your friend.
Get rid of AsyncTark is slower than RX and you cannot handle Errors

Stop an algorithm in a thread after a given time has passed

Let's say that I have an algorithm that does something for a given parameter. If the algorithm runs longer than 100 miliseconds then I want to stop it and try again for a different parameter.
I posted below the code that would test the algorithm for a random parameter... and how I think the code might look like:
public class StopThread {
private Lock lock = new ReentrantLock();
public static void main(String... args) {
System.out.println("Starting threads...");
(new StopThread()).startThreads(100);
}
private void startThreads(int nrOfThreads) {
for (int i = 0; i < nrOfThreads; i++) {
startThread(i, (long) (Math.random() * 10000000000l));
System.out.println("Started thread number " + (i + 1));
}
}
private void startThread(final int number, final long load) {
Thread workerThread = new Thread() {
#Override
public void run() {
try {
lock.lock();
doAlgorithmWork(load);
} finally {
System.out.println("Thread " + (number + 1) + " finished...");
lock.unlock();
}
}
};
Thread timerThread = new Thread() {
#Override
public void run() {
try {
sleep(100);
} catch (InterruptedException e) {
}
}
};
workerThread.start();
timerThread.start();
do {
if (!workerThread.isAlive() || !timerThread.isAlive()) {
workerThread.stop();
timerThread.stop();
}
} while (!workerThread.isAlive() && !timerThread.isAlive());
}
protected void doAlgorithmWork(long load) {
while (load-- > 0) {
}
}
}
I feel like this question should already have an answer, but what I found until now seemed complicated and I didn't know how to use it. I'm not that knowledgeable with threads and I would appreciate if you could post some code.

A very simple solution would look like this:
private void startThreads(int nrOfThreads) {
for (int i = 0; i < nrOfThreads; i++) {
Thread worker = new Thread() {
#Override
public void run() {
doAlgorithmWork((long) (Math.random() * 10000000000l));
}
}
worker.start();
worker.join(100); //block until either the thread is done, or 100ms passed
if (worker.isAlive()) worker.stop(); //if thread is still alive, stop it
}
}
This will achieve your goal, but suffers from a number of "drawbacks"
It is single threaded (that is, all calls to doAlgorithm execute one after another, instead of in parallel, so you are only using a single core of your machine);
It uses the discouraged Thread.stop() method. A preferred approach is to instead have a "stop" flag which is set to true (in place of the stop() call), and which is also constantly checked for in doAlgorith;
It creates a new thread for each doAlgorithm call (instead of reusing a single thread), which is "wasteful", but for your case probably has little practical implications
UPDATE:
In order to avoid the deprecated stop() call, you will need to add a flag to your worker thread, creating a separate class like this:
public class Worker implements Runnable {
private volatile boolean stopped = false;
public void stop() {
stopped = true;
}
#Override
public void run() {
doAlgorithmWork((long) (Math.random() * 10000000000l));
}
private void doAlgorithmWork(long load) {
while (!stopped && load-- > 0) {
//calculation
}
}
}
Then your runner looks like this:
private void startThreads(int nrOfThreads) {
for (int i = 0; i < nrOfThreads; i++) {
Thread worker = new Thread(new Worker());
worker.start();
worker.join(100); //block until either the thread is done, or 100ms passed
if (worker.isAlive()) worker.stop(); //if thread is still alive, stop it
}
}
You could also create a constructor for Worker which accepts the load value (instead of having it generated inside the Worker itself).
Note that if the calculation inside doAlgorithm() is too time-consuming, the thread may run for more than 100ms (since it always completes each calculation within the loop). If this is an issue, then your alternative is to interrupt the thread instead (calling worker.interrupt() will cause an InterruptedException to be thrown within the run() method).

Check that Files Execute Concurrently

I need to perform 2 tasks independently.
First Task
Once per minute it should check whether there is any file in a specific folder. If there is, it should add the names of the files to a queue.
This can be done as follows:
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ScheduledThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
public class schedulerExample extends Thread{
public void checkFile()
{
System.out.println("checking whether file exist in folder");
}
public void getFiles()
{
System.out.println("getting the file names");
}
public void enqueueFiles()
{
System.out.println("add files to queue");
}
public static void main(String[] args) {
final schedulerExample obj = new schedulerExample();
ScheduledExecutorService executor = new ScheduledThreadPoolExecutor(1);
executor.scheduleAtFixedRate(new Runnable() {
#Override
public void run() {
obj.checkFile();
obj.getFiles();
obj.enqueueFiles();
}
}, 0, 1, TimeUnit.MINUTES);
}
}
Second Task
If the queue is empty , sleep for one minute or else process the file from the queue one-by-one.
public class processModel extends Thread{
public static void getQueueSize(int size)
{
System.out.println("getting queue size");
}
public void dequeue()
{
// dequeue the queue
System.out.println("dequeue");
}
public void processFile()
{
// process the file
System.out.println("process file");
}
public static void main(String[] args) {
final boolean flag = true;
final int size = 9;
final processModel obj = new processModel();
Thread t1 = new Thread(){
public void run()
{
while(flag)
{
obj.dequeue();
obj.processFile();
getQueueSize(size);
if(size == 0)
{
try
{
Thread.sleep(60000);
}
catch(InterruptedException e)
{
}
}
}
}
};
t1.start();
}
}
Now I need to do both in a single class concurrently. Is that possible?
One thread should be fetching the files once per minute. Another thread should execute files one-by-one.. if no files are there it waits for a minute and checks again. In the second method I have used an infinite loop -- instead of that, is there a way that I can perform things one-by-one?

You might want to consider using Callables with the java ExecutorService. This can be used to easily break up tasks and allow them to run concurrently. Beyond that, you can get a Future which will allow you to check the results at any time (or postpone if it's not done).
There is a great book about java and concurrency called "Concurrency in Practice."
Beyond that, Java 7 has new functionality to allow file listeners on directories. That might allow you to abstract this "check and loop" functionality.

Synchronize on the queue object when you get file from it and when you add file to it.
In the thread that reads, call wait() if the queue is empty.
In the thread that checks for new files, call notify() after you added the new file to the queue.
This is how it's usually done.
You should also prevent adding file that is being processed to the queue.

check and process once in n sec using java

I have a requirement in which I have to check whether any file exists in the folder. If yes, then I need to process it one by one. With my basic knowledge I have arrived to a code structure which I have posted below.
I'm creating an infinite loop and checking whether the file exists in that folder. If yes, then I'm creating a thread and processing it, else it waits for a min and checks again.
class sample {
synchronized int getNoOfFiles() {
// get number of files in the folder
}
synchronized void openFile() {
// open one file
}
synchronized void getFileContents() {
// get the file content
}
synchronized void processFileContent() {
//performing some operation on file contents
}
synchronized void closeFile() {
//closing the file
}
synchronized void deleteFile() {
//delete the file
}
}
class Test {
public static void main(String args[]) {
int flag=0;
Sample obj = new Sample();
while(1) {
flag = obj.getNoOfFiles();
if(flag) {
for(i=0;i<flag;i++) {
MyThread1 t1 = new MyThread1() {
public void run() {
obj.openFile();
obj.getFileContents();
obj.processFileContent();
obj.closeFile();
obj.deleteFile();
}
};
t1.start();
}
}
else {
try {
Thread.sleep(60000);
}
}
}
}
}

Instead of doing this kind of thing yourself, I suggest you take a look at the Timer class, that can be used to do recurring tasks. Because messing around with threads manually can often result in weird bugs.
Even better would be ScheduledThreadPoolExecutor, but it might be a bit complicated if you've never used executors before. See Keppil's answer for how to do this.
The differences between the two are nicely summed up here: Java Timer vs ExecutorService?.

I would suggest using a ScheduledExecutorService:
ScheduledExecutorService executor = new ScheduledThreadPoolExecutor(1);
executor.scheduleAtFixedRate(new Runnable() {
public void run()
{
obj.openFile();
obj.getFileContents();
obj.processFileContent();
obj.closeFile();
obj.deleteFile();
}
}, 0, 1, TimeUnit.MINUTES);