We Keep Coding

Mockito verify with timeout failes unpredictably on concurrent execution

I just written a Mockito code to test verify(mock, timeout()) feature when concurrently invoked a method on a mock object.
#RunWith(Parameterized.class)
public class MockitoTest {
#Parameters
public static Collection<Object[]> data() {
return Stream.generate(() -> new Object[]{}).limit(100).collect(Collectors.toList());
}
#Test
public void testVerifyTimeout() throws Exception {
List listMock = mock(List.class);
ExecutorService executorService = Executors.newFixedThreadPool(2);
Stream.iterate(0, i -> i + 1).limit(10).map(i -> new AddToListTask(listMock, i)).forEach(executorService::submit);
verify(listMock, timeout(1000)).add(2);
executorService.shutdown();
}
private static class AddToListTask implements Callable<Void> {
private final List<Integer> list;
private final int value;
public AddToListTask(List<Integer> list, int value) {
this.list = list;
this.value = value;
}
#Override
public Void call() throws Exception {
list.add(value);
return null;
}
}
}
This test fails randomly 10-20 times out of 100 runs. This is a very basic case where we run a mock method concurrently, and the result is not always verified correctly.
Each test runs about 25ms-50ms, but even if it fails, it does not wait for 1 sec.
Any ideas?

In mockito 1.9.0 this was a bug, but already has been fixed, so from version 1.9.5 and onward (including 1.10.x, 2.x) works well.

Muti thread join & get return values

I'm trying to do multiple heavy calculations using threads.
Then I need to do something with the results after making sure all threads have finished its job.
Here's the basic code:
private class Runner implements Runnable {
private String _result = "fail";
public String get_results() {
return _result;
}
public void run() {
_result = "do complex calculation";
}
}
public void test() {
List<Thread> threads = new ArrayList<Thread>();
List<Runner> threadObjects = new ArrayList<Runner>();
for (int i = 0; i < 10; i++) {
Runner runner = new Runner();
Thread t = new Thread(runner);
t.start();
threads.add(t);
threadObjects.add(runner);
}
for (Thread thread : threads) {
try {
thread.join();
} catch (InterruptedException ex) {
}
}
for (Runner threadObject : threadObjects) {
System.out.println(threadObject.get_results());
}
}
My question is, is above snippet a common or a good approach to get calculation value?
If not please suggest a better ones.
Also sometimes I got runner.get_results() reponse = "fail", it seems calculation does not processed at all.
Thanks

You can use an ExecutorService such as the ScheduledThreadPoolExecutor;
ExecutorService executor = new ScheduledThreadPoolExecutor(numOfThreads);
With numOfThreads being the number of threads you want sitting in the thread pool.
You can then use the submit(Callable<T> task) method provided by the ScheduledThreadPoolExecutor class to execute the calculation.
You would then have a Callable implementation of your calculation and pass it to the submit() method in ExecutorService to execute the calculation;
class Calculation implements Callable {
#Override
public Object call() throws Exception { // The signature can be changed to return a different type (explained at the end)
return "do complex calculation";
}
}
As we can see from the method signature of the submit(Callable<T> task) method it returns a Future.
public <T> Future<T> submit(Callable<T> task)
The get() method of the Future class will return the result upon successful completion. This would ensure that your calculation completed before reading it.
A further note on the return type of the call() method;
Although this returns Object there is nothing stopping you changing the type of object it returns (this is known as co-variant returns)
For example the following is perfectly legal:
#Override
public String call() throws Exception {
return "do complex calculation";
}

How can I test a blocking method using junit

I have a class with a method that blocks and would like to validate that it is blocking. The method is as shown below.
public static void main(String[] args) {
// the main routine is only here so I can also run the app from the command line
applicationLauncherInstance.initialize();
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run() {
if (null != application) {
applicationLauncherInstance.terminate();
}
}
});
try {
_latch.await();
} catch (InterruptedException e) {
log.warn(" main : ", e);
}
System.exit(0);
}
How can I write a unit test for such a method. I am stuck before starting.
public class ApplicationLauncherTest extends TestCase {
public void testMain() throws Exception {
ApplicationLauncher launcher = new ApplicationLauncher();
}
}

Thanks to Kulu, I found the solution.
public void testMain() throws Exception {
Thread mainRunner = new Thread(() -> {
ApplicationLauncher.main(new String[]{});
});
mainRunner.start();
Thread.sleep(5000);
assertEquals(Thread.State.WAITING, mainRunner.getState());
mainRunner.interrupt();
}

Bwire's answer is a good way there, but I highly recommend that no
one ever use Thread.sleep() in unit tests for validation of some situation. It's impossible to get the timing right:
If it's too short, you'll get a lotta false results (random failures, yay)
If it's too long, you end up creating painfully slow tests over time. Don't underestimate this.
So, what's the answer? Any time you need to "sleep" to test something, instead "wait" for that to be true (constantly checking). This way:
As soon as the condition is true, your program resumes--no wasted time.
You can set the timeout on this "wait" to a crazy large value, to avoid random failures.
Here's a modified version of Bware's self-response...
public void testMain() throws Exception {
Thread mainRunner = new Thread(() -> {
ApplicationLauncher.main(new String[]{});
});
mainRunner.start();
expectToBlock(mainRunner, 30, TimeUnit.SECONDS);
mainRunner.interrupt();
}
private static void expectToBlock(Thread thread, long waitCount, TimeUnit waitUnits) {
long start = System.currentTimeMillis();
while (System.currentTimeMillis() - start < waitUnits.toMillis(waitCount)) {
if (thread.getState() == Thread.State.WAITING) {
return;
}
Thread.sleep(50); // Don't hog the CPU
}
Assert.fail("Timed out while waiting for thread to block");
}

How to write unit test which creates new thread

I have following method for test:
public class classToTest{
#Autowired
private Alternator alternator;
public void methodToTest(){
Thread t = new Thread(new Runnable() {
public void run() {
while(true) {
if(alternator.get()) {
System.out.print("Hello");
alternator.set(false);
}
}
}
};
t.start()
}
}
I need to check that was invoked method
alternator.set(false);
How can I do it?

Instead of starting a thread directly, can you pass in an "Executor" instance?
For example...
public class ClassToTest{
#Autowired
private Alternator alternator;
#Autowired #Qualifier("myExecutor")
private java.util.concurrent.Executor executor;
public void methodToTest() {
Runnable runnable = new Runnable() {
public void run() {
while(true) {
if(alternator.get()) {
System.out.print("Hello");
alternator.set(false);
}
}
};
executor.execute(runnable);
}
}
Now you can test this easier...
public class ClassToTestTest {
...
#Before
public void setup() {
alternator = mock(Alternator.class);
executor = mock(Executor.class);
obj = new ClassToTest();
ReflectionTestUtils.setField(obj, "alternator", alternator);
ReflectionTestUtils.setField(obj, "executor", executor);
}
#Test
public void shouldStartRunnable() {
obj.methodToTest();
ArgumentCaptor<Runnable> runnableCaptor = ArgumentCaptor.forClass(Runnable.class);
verify(executor).execute(runnableCaptor.capture());
Runnable runnable = runnableCaptor.getValue();
// Now test your actual "runnable"...
when(alternator.get()).thenReturn(true);
runnable.run();
verify(alternator).set(false);
}
}
(Have not tried to compile this, so I apologise if there are any mistakes!!)

Though Bret's post of passing in an executor is very much recommended, you can use the timeout() mock verification setting to test for asynchronous conditions.
verify(alternator, timeout(500)).set(false);
Note that this will necessarily increase the flakiness of your test (i.e. the likelihood that the test fails when the code passes). With a sensible timeout value, that flakiness should be negligible, but if you're making this a part of your core test infrastructure you may consider refactoring to allow for synchronous execution in the test.

Unit test succeeds in debug mode but fails when running it normally

Why does my unit test succeed in debug mode but fail when running it normally?
public class ExecutorServiceTest extends MockitoTestCase{
private int numThreads;
private ExecutorService pool;
private volatile boolean interruptedBitSet;
#Override
public void setUp() {
numThreads = 5;
pool = Executors.newFixedThreadPool(numThreads);
}
class TaskChecksForInterruptedBit implements Callable<String> {
#Override
public String call() throws Exception {
interruptedBitSet = false;
while (!Thread.currentThread().isInterrupted()) {
}
interruptedBitSet = Thread.currentThread().isInterrupted();
return "blah";
}
}
public void testCancelSetsInterruptedBitInCallable() throws Exception {
interruptedBitSet = false;
final Future<String> future =
pool.submit(new TaskChecksForInterruptedBit());
final boolean wasJustCancelled = future.cancel(true);
assertTrue(wasJustCancelled);
// Give time for the thread to notice the interrupted bit and set the flag
Thread.sleep(5000);
// This succeeds when stepping through w/ a debugger, but fails when running
// the test straight. WHY?
assertTrue(interruptedBitSet);
assertTrue(future.isDone());
assertTrue(future.isCancelled());
}
}

The reason is almost certainly that your breakpoint in the debugger is halting the main thread but not any of the background threads - the ones in the ExecutorService. When debugging in eclipse you can change the breakpoint to halt all threads instead of just the main one.
When not debugging the submission of the task and the immediate cancellation are so quick that you are cancelling the task before it even runs once. Try adding a sleep delay between these lines:
final Future<String> future = pool.submit(new TaskChecksForInterruptedBit());
Thread.sleep(1000);
final boolean wasJustCancelled = future.cancel(true);

You have to make sure your task actually started running. It may get cancelled before it even has a chance.
public class ExecutorServiceTest {
private int numThreads;
private ExecutorService pool;
private volatile boolean interruptedBitSet;
private static final CountDownLatch latch = new CountDownLatch(1);
#Before
public void setUp() {
numThreads = 5;
pool = Executors.newFixedThreadPool(numThreads);
}
class TaskChecksForInterruptedBit implements Callable<String> {
#Override
public String call() throws Exception {
interruptedBitSet = false;
latch.countDown();
while (!Thread.currentThread().isInterrupted()) {
System.out.println(System.currentTimeMillis());
}
System.out.println("haha");
interruptedBitSet = Thread.currentThread().isInterrupted();
return "blah";
}
}
#Test
public void testCancelSetsInterruptedBitInCallable() throws Exception {
final Future<String> future =
pool.submit(new TaskChecksForInterruptedBit());
interruptedBitSet = false;
latch.await();
final boolean wasJustCancelled = future.cancel(true);
Assert.assertTrue(wasJustCancelled);
// Give time for the thread to notice the interrupted bit and set the flag
Thread.sleep(5000);
// This succeeds when stepping through w/ a debugger, but fails when running
// the test straight. WHY?
Assert.assertTrue(interruptedBitSet);
Assert.assertTrue(future.isDone());
Assert.assertTrue(future.isCancelled());
}
}

I know this is old but I just had the same problem.
My issue was that I had an IEnumerable that I was enumerating and checking the output.
When running the Unit test, the IEnumerable was returning a different ordering that when debugging. This is the nature of IEnumerable and simply adding an OrderBy clause solved my problem.
I hope this helps someone out there as it can be a frustrating problem to find.

You should check whether all the threads dies before terminating the main thread
private void shutdownExecutionService(ExecutorService executorService) {
if (executorService != null) {
try {
executorService.shutdown();
while (!executorService.awaitTermination(10, TimeUnit.HOURS)) {
logger.info("Awaiting completion of threads.");
}
} catch (final InterruptedException e) {
logger.error("Error while shutting down threadpool", e);
}
}
}

I was having a similar issue while running a homework assignment from an online course. The grader program from the course which I added to the build path used JUnit4, my version of Eclipse added JUnit5 to any new test cases. I've created a new Java Project and added JUnit5 to the build bath for my test cases without the grader and it fixed it for me. Hope this helps.