We Keep Coding

Unit testing asynchronous computation that has to be interrupted manually

I have got a class that records eyetracking data asynchronously. There are methods to start and stop the recording process. The data is collected in a collection and the collection can only be accessed if the recording thread has finished its work. It basically encapsulates all the threading and synchronizing so the user of my library doesn't have to do it.
The heavily shortened code (generics and error handling omitted):
public class Recorder {
private Collection accumulatorCollection;
private Thread recordingThread;
private class RecordingRunnable implements Runnable {
...
public void run() {
while(!Thread.currentThread().isInterrupted()) {
// fetch data and collect it in the accumulator
synchronized(acc) { acc.add(Eyetracker.getData()) }
}
}
}
public void start() {
accumulatorCollection = new Collection();
recordingThread = new Thread(new RecordingRunnable(accumulatorCollection));
recordingThread.start();
}
public void stop() {
recordingThread.interrupt();
}
public void getData() {
try {
recordingThread.join(2000);
if(recordingThread.isAlive()) { throw Exception(); }
}
catch(InterruptedException e) { ... }
synchronized(accumulatorCollection) { return accumulatorCollection; }
}
}
The usage is quite simple:
recorder.start();
...
recorder.stop();
Collection data = recorder.getData();
My problem with the whole thing is how to test it. Currently i am doing it like this:
recorder.start();
Thread.sleep(50);
recorder.stop();
Collection data = recorder.getData();
assert(stuff);
This works, but it is non-deterministic and slows down the test suite quite a bit (i marked these tests as integration tests, so they have to be run separately to circumvent this problem).
Is there a better way?

There is a better way using a CountDownLatch.
The non-deterministic part of the test stems from two variables in time you do not account for:
creating and starting a thread takes time and the thread may not have started executing the runnable when Thread.start() returns (the runnable will get executed, but it may be a bit later).
the stop/interrupt will break the while-loop in the Runnable but not immediately, it may be a bit later.
This is where a CountDownLatch comes in: it gives you precise information about where another thread is in execution. E.g. let the first thread wait on the latch, while the second "counts down" the latch as last statement within a runnable and now the first thread knows that the runnable finished. The CountDownLatch also acts as a synchronizer: whatever the second thread was writing to memory, can now be read by the first thread.
Instead of using an interrupt, you can also use a volatile boolean. Any thread reading the volatile variable is guaranteed to see the last value set by any other thread.
A CountDownLatch can also be given a timeout which is useful for tests that can hang: if you have to wait to long you can abort the whole test (e.g. shutdown executors, interrupt threads) and throw an AssertionError. In the code below I re-used the timeout to wait for a certain amount of data to collect instead of 'sleeping'.
As an optimization, use an Executor (ThreadPool) instead of creating and starting threads. The latter is relative expensive, using an Executor can really make a difference.
Below the updated code, I made it runnable as an application (main method). (edit 28/02/17: check maxCollect > 0 in while-loop)
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicBoolean;
public class Recorder {
private final ExecutorService executor;
private Thread recordingThread;
private volatile boolean stopRecording;
private CountDownLatch finishedRecording;
private Collection<Object> eyeData;
private int maxCollect;
private final AtomicBoolean started = new AtomicBoolean();
private final AtomicBoolean stopped = new AtomicBoolean();
public Recorder() {
this(null);
}
public Recorder(ExecutorService executor) {
this.executor = executor;
}
public Recorder maxCollect(int max) { maxCollect = max; return this; }
private class RecordingRunnable implements Runnable {
#Override public void run() {
try {
int collected = 0;
while (!stopRecording) {
eyeData.add(EyeTracker.getData());
if (maxCollect > 0 && ++collected >= maxCollect) {
stopRecording = true;
}
}
} finally {
finishedRecording.countDown();
}
}
}
public Recorder start() {
if (!started.compareAndSet(false, true)) {
throw new IllegalStateException("already started");
}
stopRecording = false;
finishedRecording = new CountDownLatch(1);
eyeData = new ArrayList<Object>();
// the RecordingRunnable created below will see the values assigned above ('happens before relationship')
if (executor == null) {
recordingThread = new Thread(new RecordingRunnable());
recordingThread.start();
} else {
executor.execute(new RecordingRunnable());
}
return this;
}
public Collection<Object> getData(long timeout, TimeUnit tunit) {
if (started.get() == false) {
throw new IllegalStateException("start first");
}
if (!stopped.compareAndSet(false, true)) {
throw new IllegalStateException("data already fetched");
}
if (maxCollect <= 0) {
stopRecording = true;
}
boolean recordingStopped = false;
try {
// this establishes a 'happens before relationship'
// all updates to eyeData are now visible in this thread.
recordingStopped = finishedRecording.await(timeout, tunit);
} catch(InterruptedException e) {
throw new RuntimeException("interrupted", e);
} finally {
stopRecording = true;
}
// if recording did not stop, do not return the eyeData (could stil be modified by recording-runnable).
if (!recordingStopped) {
throw new RuntimeException("recording");
}
// only when everything is OK this recorder instance can be re-used
started.set(false);
stopped.set(false);
return eyeData;
}
public static class EyeTracker {
public static Object getData() {
try { Thread.sleep(1); } catch (Exception ignored) {}
return new Object();
}
}
public static void main(String[] args) {
System.out.println("Starting.");
ExecutorService exe = Executors.newSingleThreadExecutor();
try {
Recorder r = new Recorder(exe).maxCollect(50).start();
int dsize = r.getData(2000, TimeUnit.MILLISECONDS).size();
System.out.println("Collected " + dsize);
r.maxCollect(100).start();
dsize = r.getData(2000, TimeUnit.MILLISECONDS).size();
System.out.println("Collected " + dsize);
r.maxCollect(0).start();
Thread.sleep(100);
dsize = r.getData(2000, TimeUnit.MILLISECONDS).size();
System.out.println("Collected " + dsize);
} catch (Exception e) {
e.printStackTrace();
} finally {
exe.shutdownNow();
System.out.println("Done.");
}
}
}
Happy coding :)

handle multiple infinite tasks in a single thread? P.S run one task at a time and control its task behavior(i.e starting/stoping task) from outside

I want to make a single thread which would contain 3 infinite tasks.
I want one task to run at a time and start/stop running task when required.
For example first I want task 1 to run, then I want task 2 to run but after stopping task 1 and again I want task 1 to run but after stopping of task 2 and so on.
Infinite task needs to check some condition and if that condition is satisfied perform some operations and if not satisfied sleep for few seconds and after wake up perform the above same operations again.
Infinite Runnable task looks some thing like this:
new Runnable(){
while(1){
if(TaskQueue.getInstance().size()<= 100){
TaskQueue.getInstance().push("add command to the end of queue");
}else{
try {
Thread.sleep(10000);
}catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
Any help would be appreciated?
Edit : I modified my question. I want a continuous single running thread(some thing like looper ) to monitor 3 infinite tasks and control this single continuous running thread tasks from outside.

Use this for start/stop thread in real-time:
class MyThread extends Thread {
private volatile boolean running = true; // Run unless told to pause
...
#Override
public void run() {
// Only keep painting while "running" is true
// This is a crude implementation of pausing the thread
while (true) {
if (Thread.currentThread().isInterrupted()) {
return;
}
if (running) {
//Your code
} else yield;
}
}
public void pauseThread() throws InterruptedException {
running = false;
}
public void resumeThread() {
running = true;
}
}
For pause thread use this:
myThread.pauseThread();
For resume thread use this:
myThread.resumeThread();
For stop thread use this (Not recommended):
myThread.stop();
For currently stop thread use this:
myThread.interrupt();

You must use a class like Thread that already implements Runnable.
new Thread(){....};
And the way it works it's:
Thread t = new Thread(){.....};
t.start();
t.stop();

You could also initialize a new thread, like:
Thread exampleThread = new thread();
After this you can start it at any point in your code by:
exampleThread.start();

you can use Semaphore,
to Manage the amount of signal.
private final static Semaphore semaphore = new Semaphore(0);
public static void main(String[] args) throws Exception {
//入口
threadTest();
}
public static void thread1() {
try{
//…… some code
}
finally{
semaphore.release();
}
}
public static void thread2() {
semaphore.acquire(1);
}
The question is my first answer,thanks.

I finally made my task scheduler. The API of which looks something like this:
TaskScheduler taskScheduler = TaskScheduler.getInstance();
taskScheduler.startTaskOne();
taskScheduler.stopTaskOne();
taskScheduler.startTaskTwo();
taskScheduler.stopTaskTwo();
Runs one task at a time (because I used Executors.newSingleThreadExecutor()).
We can control the execution of the task from outside:
public class TaskScheduler {
private static ExecutorService mTaskRunningService;
private static TaskScheduler mInstance;
private Future mFirstTaskFuture = null;
private Future mSecondTaskFuture = null;
static {
configure();
}
private static void configure() {
mTaskRunningService = Executors.newSingleThreadExecutor();
}
public static TaskScheduler getInstance() {
if (mInstance == null) {
mInstance = new TaskScheduler();
}
return mInstance;
}
private Runnable mTaskOneRunnable = new Runnable() {
#Override
public void run() {
try {
while (true) {
/** stop this single thread (i.e executing one task at time) service if this thread is interrupted
* from outside because documentation of {#link java.util.concurrent.ThreadPoolExecutor#shutdownNow()}
* says we need to do this*/
if (Thread.currentThread().isInterrupted()) {
return;
}
// task one work.......
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
};
private Runnable mTaskTwoRunnable = new Runnable() {
#Override
public void run() {
try {
while (true) {
/** stop this single thread (i.e executing one task at time) service if this thread is interrupted
* from outside because documentation of {#link java.util.concurrent.ThreadPoolExecutor#shutdownNow()}
* says we need to do this*/
if (Thread.currentThread().isInterrupted()) {
return;
}
// task two work......
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
};
public synchronized void startTaskOne() {
if (mFirstTaskFuture == null) {
// start executing runnable
mFirstTaskFuture = mTaskRunningService.submit(mTaskOneRunnable);
}
}
public synchronized boolean stopTaskOne() {
if (mFirstTaskFuture != null) {
// stop general reading thread
mFirstTaskFuture.cancel(true);
// cancel status
boolean status = mFirstTaskFuture.isDone();
// assign null because startTaskOne() again be called
mGeneralFuture = null;
return status;
}
return true;
}
public synchronized void startTaskTwo() {
if (mSecondTaskFuture == null) {
// start executing runnable
mSecondTaskFuture = mTaskRunningService.submit(mTaskTwoRunnable);
}
}
public synchronized boolean stopTaskTwo() {
if (mSecondTaskFuture != null) {
// clear task queue
mTaskQueue.clearTaskQueue();
// stop 22 probes reading thread
mSecondTaskFuture.cancel(true);
// cancel status
boolean status = mSecondTaskFuture.isDone();
// assign null because startTaskTwo() again be called
mSecondTaskFuture = null;
return status;
}
return true;
}
}

Scheduled executor: poll for result at fix rate and exit if timeout or result valid

Problem:
I have a requirement to call a dao method at fix rate say every 10 sec, then I need to check if the result is valid if yes exit, else keep on calling that method every 10 sec till I get a valid result or defined time out (say 2 min) is over.
Approaches:
I want to keep the task and scheduler logic separate, and write a task in such a way that it can be used by different classes having similar requirement.
One way I can think is to define a new poller task
public abstract class PollerTask<T> implements Runnable {
abstract public boolean isValid(T result);
abstract public T task();
private T result;
private volatile boolean complete;
public boolean isComplete() {
return complete;
}
public T getResult() {
return result;
}
#Override
final public void run() {
result = task();
if (complete = isValid(result)) {
//may be stop scheduler ??
}
}
}
User need to simply provide implementation of task and isValid;
Then we can define a separate class that takes pooling freq and timeout and creates a scheduled executor and submit this task
public class PollerTaskExecutor {
private int pollingFreq;
private int timeout;
private ScheduledExecutorService executor;
private ScheduledExecutorService terminator;
private ExecutorService condition;
private volatile boolean done;
private ScheduledFuture future;
public PollerTaskExecutor(int pollingFreq, int timeout) {
this.pollingFreq = pollingFreq;
this.timeout = timeout;
executor = Executors.newSingleThreadScheduledExecutor();
terminator = Executors.newSingleThreadScheduledExecutor();
condition = Executors.newSingleThreadExecutor();
}
public void submitTaskForPolling(final PollerTask pollerTask) {
future = executor.scheduleAtFixedRate(pollerTask, 0, pollingFreq, TimeUnit.SECONDS);
terminator.schedule(new Runnable() {
#Override
public void run() {
complete();
}
}, timeout, TimeUnit.SECONDS);
condition.execute(new Runnable() {
#Override
public void run() {
if (pollerTask.isComplete()) {
complete();
}
}
});
}
public boolean isDone() {
return done;
}
public void complete() {
future.cancel(false);
executor.shutdown();
terminator.shutdown();
condition.shutdown();
done = true;
}
now user can wait till pollerExecutor.isDone returns true and get the result.
I had to use three executors for following purposes:
executor to run task at fix interval
executor to stop all when time out is over
executor to stop all if valid result is obtained before time out.
Can someone please suggest a better approach, this seems to be complicated for such a trivial task ?

Make it a self-scheduling task. In pseudo code:
public class PollingTaskRunner {
...
CountDownLatch doneWait = new CountDownLatch(1);
volatile boolean done;
PollingTaskRunner(Runnable pollingTask, int frequency, int period) {
...
endTime = now + period;
executor.schedule(this, 0);
}
run() {
try {
pollingTask.run();
} catch (Exception e) {
...
}
if (pollingTask.isComplete() || now + frequency > endTime) {
done = true;
doneWait.countDown();
executor.shutdown();
} else {
executor.schedule(this, frequency);
}
}
await() {
doneWait.await();
}
isDone() {
return done;
}
}
It is not that complicated but add plenty of debug statements the first time you run/test this so you know what is going on. Once it is running as intended, it is easy to re-use the pattern.

A slightly simpler method, you don't need a separate executor service for the terminator, you could simply push the terminator task into the same executor.
Even simpler. Have PollerTask place it's result in a BlockingQueue. Then have the PollingTaskRunner do a timed poll on that BlockingQueue. Whenever control is returned from the poll call ScheduledFuture.cancel because the task either succeeded or timed out.

Java Multithreading doesn't seem to be correctly working

I have a class which processes something. I'm trying to run a number of instances of this class in parallel.
However, I'm not sure if in TaskManager.startAll(), when I call r.go(), whether this would cause r to start running in its own thread, or within the main thread?
The total execution time that I'm getting seems to be very high, and despite my attempts at optimizing, nothing seems to be having any effect. Also, if I run a profiler on my project in Netbeans, it shows all the threads as sleeping. So I'd like to know if I'm doing something wrong?
This is the structure of the class:
public class TaskRunner implements Runnable {
private boolean isRunning = false;
public void run() {
while(true) {
while (! running) {
try {
Thread.sleep(1);
} catch (Exception e) {
e.printStackTrace();
}
}
process();
}
}
public void go() {
isRunning = true;
}
public void stop() {
isRunning = false;
}
private void process() {
//Do some number crunching and processing here
}
}
Here's how these are being run / managed:
public class TaskManager {
private ArrayList<TaskRunner> runners = new ArrayList<>();
public TaskManager() {
for (int i = 0; i < 10; i++) {
TaskRunner r = new TaskRunner();
new Thread(r).start();
runners.add(r);
}
}
public void startAll() {
for (TaskRunner r : runners) {
r.go();
}
}
}

Indeed, you are not "doing it right." If you want to create a multi-threaded Java application, the place to start is with the java.util.concurrent package.
It appears from your code that you want to run ten tasks in parallel. I assume that after "number crunching and processing," you'll want to aggregate the results and do something with them in the main thread. For this, the invokeAll() method of ExecutorService works well.
First, implement Callable to do the work you show in your process() method.
final class YourTask implements Callable<YourResults> {
private final YourInput input;
YourTask(YourInput input) {
this.input = input;
}
#Override
public YourResults call()
throws Exception
{
/* Do some number crunching and processing here. */
return new YourResults(...);
}
}
Then create your tasks and run them. This would take the place of your main() method:
Collection<Callable<YourResults>> tasks = new List<>(inputs.size());
for (YourInput i : inputs)
tasks.add(new YourTask(i));
ExecutorService workers = Executors.newFixedThreadPool(10);
/* The next call blocks while the worker threads complete all tasks. */
List<Future<YourResult>> results = workers.invokeAll(tasks);
workers.shutdown();
for (Future<YourResult> f : results) {
YourResult r = f.get();
/* Do whatever it is you do with the results. */
...
}

However, I'm not sure if in TaskManager.startAll(), when I call r.go(), whether this would cause r to start running in its own thread, or within the main thread?
So my first comment is that you should make isRunning be volatile since it is being shared between threads. If the threads are not starting when it goes to true (or seem to be delayed in starting) then I suspect that's your problem. volatile provides memory synchronization between the threads so the thread that calls go() and makes a change to isRunning will be seen immediately by the thread waiting for the change.
Instead of spinning like this, I would use wait/notify:
// this synchronizes on the instance of `TaskRunner`
synchronized (this) {
// always do your wait in a while loop to protect against spurious wakeups
while (!isRunning && !Thread.currentThread().isInterrupted()) {
try {
// wait until the notify is called on this object
this.wait();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
e.printStackTrace();
}
}
Then in the go() method you should do the following. stop() would be similar.
public void go() {
synchronized (this) {
isRunning = true;
this.notifyAll();
}
}
Notice that you should handle thread interrupts carefully. Test for isInterrupted() in the while running loop and re-interrupt a thread when InterruptedException is thrown is always a good pattern.
The total execution time that I'm getting seems to be very high, and despite my attempts at optimizing, nothing seems to be having any effect. Also, if I run a profiler on my project in Netbeans, it shows all the threads as sleeping.
So although the threads are mostly sleeping, they are still each looping 1000 times a second because of your Thread.sleep(1). If you increased the time sleeping (after making isRunning be volatile) they would loop less but the right mechanism is to use the wait/notify to signal the thread.

Awful solution, terrible. first I highly recommend you start reading some tutorial like [this]
Second, if threads should wait for a signal to go for some job, so why just don't you wait them!!!!!, something like this
import java.util.ArrayList;
public class TaskManager
{
//////////////////////
public volatile static Signal wait=new Signal();
//////////////////////
private ArrayList<TaskRunner> runners = new ArrayList<>();
public TaskManager()
{
for (int i = 0; i < 10; i++)
{
TaskRunner r = new TaskRunner();
new Thread(r).start();
runners.add(r);
}
try {
Thread.sleep(1000);
startAll();
Thread.sleep(1000);
pauseAll();
Thread.sleep(1000);
startAll();
Thread.sleep(1000);
haltAll();System.out.println("DONE!");
}catch(Exception ex){}
}
public void startAll()
{
synchronized(wait){
wait.setRun(true);;
wait.notifyAll();
}
}
public void pauseAll(){
wait.setRun(false);
}
public void haltAll(){
for(TaskRunner tx:runners){tx.halt();}
}
public static void main(String[] args) {
new TaskManager();
}
}
class TaskRunner implements Runnable
{
private Thread thisThread;
private volatile boolean run=true;
public void run()
{
thisThread=Thread.currentThread();
while(run){
if(!TaskManager.wait.isRun()){
synchronized(TaskManager.wait)
{
if(!TaskManager.wait.isRun()){
System.out.println("Wait!...");
try
{
TaskManager.wait.wait();
}
catch (Exception e)
{
e.printStackTrace();
break;
}
}
}}
process();
}
}
private double r=Math.random();
private void process(){System.out.println(r);try {
Thread.sleep(10);
} catch (Exception e) {
// TODO: handle exception
}}
public void halt(){run=false;thisThread.interrupt();}
}
class Signal{
private boolean run=false;
public boolean isRun() {
return run;
}
public void setRun(boolean run) {
this.run = run;
}
}
in above sample, all runners works till the Signal run boolean is true, and simple TaskManager class set tit as false for every time it needs to pause the threads. and about the halt, it just set the shutdown(run) flag to false, and also interrupt the thread because of if thread is in wait state.
I hope I could prove your solution is like dream-on story, and also could explained enough about my solution.
have a good parallel application :)

Waiting for all Runnables submitted to SWT UI thread with Display::asyncExec() to finish

Is there a way to wait for all Runnables submitted to the SWT UI Thread via asyncExec(...) to finish?
Background:
I have a long-running operation, which among other things is triggering events that in turn submit Runnables to the SWT UI thread via the asyncExec(...) instance method of Display.
The progress of the long-running operation is shown in a ProgressMonitorDialog, and I would like to close the dialog only after the UI thread has finished executing the Runnables.
Changing the calls from asyncExec(...) to syncExec(...) is not an option, as the latter is not desired when the events are triggered from other contexts.

org.eclipse.swt.widgets.Display.readAndDispatch() will process an event from the event queue and return false if there are no more events to process. But you probably don't want to use this as it processes an event.
asyncExec(*) is a FIFO queue (although OS graphics events supersede the asyncExecs), so you could do most of your long-running op processing and then place a final asyncExec in the queue:
final boolean[] done = new boolean[1];
Runnable r = new Runnable() {
public void run() {
done[0] = true;
}
};
// now wait for the event somehow. The brute force method:
while (!done[0]) {
Thread.sleep(200);
}
In theory, all of the other asyncExecs spawned from your long running op will be finished by the time you get to the last one.
EDIT: potential other option
Create your own org.eclipse.core.runtime.jobs.Job and then join() it at the end:
public static class RefCountJob extends Job {
public RefCountJob() {
super("REF_COUNT");
}
int count = 0;
public void increment() {
count++;
}
public void decrement() {
count--;
}
#Override
protected IStatus run(IProgressMonitor monitor) {
monitor.beginTask("WAITING", IProgressMonitor.UNKNOWN);
while (count > 0) {
Thread.sleep(200);
monitor.worked(1);
}
monitor.done();
return Status.OK_STATUS;
}
}
To use it, increment() it every time you are going to fire off events, and have them decrement it when they're done (You have to make sure they decrement it no matter what exception is thrown :-)
RefCountJob ref = new RefCountJob();
// ... do stuff, everybody increments and decrements ref
ref.increment();
// ... do more stuff
ref.increment();
// at the end of your long-running job
ref.schedule();
ref.join();

Thanks, I ended up with the following. I think it is a pretty clean solution. By the way I would upvote your answer if I had enough reputation for that :)
public class SWTThreadingUtils
{
public static void waitForAsyncExecsToFinish(Display display)
{
Object waitObj = new Object();
display.asyncExec(new DummyRunnable(waitObj));
synchronized (waitObj)
{
try {
waitObj.wait();
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
}
private static class DummyRunnable implements Runnable
{
private Object waitObj;
public DummyRunnable(Object waitObj)
{
this.waitObj = waitObj;
}
#Override
public void run()
{
synchronized (waitObj)
{
waitObj.notify();
}
}
}
}