We Keep Coding

ITestListener - ExtentReport

Currently I'm using ExtentReport to generate automation reports.
The way I'm using ExtentReport is using the IReporter implementation to generate the report at the end of my tests, which is great.
However, now I'm looking at creating a way of monitoring my tests while they are being executed which is not possible with IReporter.
I want to create a separate listener for real time results using ITestListener.
Has anyone used this with ExtentReport? Or anything similar?
Any useful articals or guidelines in the right direction would be appreciated.
Thanks.
EDIT: Basically need a way to generate the ITestListener live console outputs to an actually HTML Reprot where I can view the test progress from there rather than the console

It should look something like:
import com.relevantcodes.extentreports.*;
import org.testng.ITestContext;
import org.testng.ITestResult;
import org.testng.internal.IResultListener;
import java.util.Arrays;
import java.util.Locale;
/**
* Created by andrey.smirnov on 14.06.2016.
*/
public class ExtentListener implements IResultListener {
private ExtentReports reporter = new ExtentReports("build/SimpleReport.html", true, DisplayOrder.NEWEST_FIRST, NetworkMode.OFFLINE, Locale.ENGLISH);
private ExtentTest testReporter;
#Override
public void onTestStart(ITestResult result) {
testReporter = reporter.startTest(result.getMethod().getMethodName(), "Some description");
testReporter.log(LogStatus.INFO, "Starting test " + result.getMethod().getMethodName());
}
#Override
public void onTestSuccess(ITestResult result) {
testReporter.log(LogStatus.PASS, "Test PASSED");
reporter.endTest(testReporter);
reporter.flush();
}
#Override
public void onFinish(ITestContext context) {
reporter.close();
}
// Other interface methods
}
It will provide report updating on each test finish. Please refer to documentation about tests parallel running. Also it would be better to pass ExtentReports instance as TestNG context attribute e.g. iTestContext.setAttribute("reporter", reporter) and use it in listener.

How to add static imports to IntelliJ IDEA live template

I need to port the following Eclipse template to IntelliJ IDEA
/**
* See method name.
*/
#${testType:newType(org.junit.Test)}
public void should${testdesc}() {
// given ${cursor}
${staticImport:importStatic('org.hamcrest.Matchers.*', 'org.junit.Assert.*', 'org.mockito.BDDMockito.*')}
// when
// then
}
What I've got so far is
/**
* See method name.
*/
#org.junit.Test
public void should$EXPR$() {
// given $END$
${staticImport:importStatic('org.hamcrest.Matchers.*', 'org.junit.Assert.*', 'org.mockito.BDDMockito.*')}
// when
// then
}
And then tick the Shorten FQ names flag.
What's the IDEA equivalent for the ${staticImport:importStatic()} expression?

You cannot just import the static imports in a live template. (You can for a file template, see below). But you can when using a method in the template. You just simply fully qualify the class and then select both the "Shorten FQ names" and "Use static import if possible" options. For example, the following:
org.junit.Assert.assertEquals("$END$", $EXPECTED$, $ACTUAL$);
Will result in:
import static org.junit.Assert.*;
. . .
assertEquals("my error message", myExpectedVar, myActualVar);
when invoked. (I have the $EXPECTED$ and $ACTUAL$ variables set to variableOfType("") with corresponding default values expected and actual)
If you want certain static imports to be included in all your unit tests, then I would recommend editing the "Class" File and Code Template. For example:
package ${PACKAGE_NAME};
#if ($NAME.endsWith("Test"))
import static org.junit.Assert.*;
import static org.hamcrest.Matchers.*;
import static org.mockito.BDDMockito.*;
#end
#parse("File Header.java")
public class ${NAME}
{
#if ($NAME.endsWith("Test"))
// Add any default test methods or such you want here.
#end
}
Keep in mind however, the static import will immediately be removed if you have the "Optimize imports on the fly" option (in IDE Settings > Editor > Auto import) turned on, unless you also include a method (or other code) that makes use of the static import.

Now its possible to add live templates with static imports:
You have to check static import in Options
#org.junit.Test
public void should$EXPR$when$CONDITION$() {
org.junit.Assert.assertThat(null, org.hamcrest.CoreMatchers.is(org.hamcrest.CoreMatchers.nullValue()));
}

JTest 9.5. How to execute preparation actions before each unit test

I'm using JTest 9.5. I wanted to ask whether there is possibility to perform same preparation routine before each unit test, just like I did it in JUnit with #Before and #After annotations? If yes, then how? Let's say I have following unit test class in JTest:
public class TestArrayFileManager extends PackageTestCase {
FileManager fileMngr;
public TestArrayFileManager(String name)
{
super(name);
}
public Class getTestedClass()
{
return FileManager.class;
}
public void testFileManager1() throws Throwable
{
final String fileName = "InputFile.txt";
fileMngr = new FileManager(fileName);
fileMngr.doResetFile();
fileMngr.doReplaceNthElement(0, 3);
fileMngr.doReplaceNthElement(1, 9);
assertEquals(3, fileMngr.doReadNthElement(0L));
}
public void testFileManager2() throws Throwable
{
final String fileName = "InputFile.txt";
fileMngr = new FileManager(fileName);
fileMngr.doResetFile();
fileMngr.doReplaceNthElement(0, 3);
fileMngr.doReplaceNthElement(1, 9);
assertEquals(9, fileMngr.doReadNthElement(1L));
}
}
Notice how i keep repeating same preparation code. How can I perform it before each test?

JTest complements and extends JUnit which means that it is not there to provide the features of JUnit. You will have to use JTest with Junit in order to achieve what you want.
You can use existing JUnit test cases with JTest which will provide both setUp and tearDown methods using #Before and #After annotations.
If you want to use a JUnit Test Class with Jtest, you need to:
Include the junit.jar file on your CLASSPATH.
Make sure Jtest knows how to locate the Test Class (as described in Loading Test Classes below).
After you perform these steps, Jtest will use the JUnit Test Class when you run your test in the normal manner.

I just fount the methods I have to add in order to achieve preparation and ending tasks before each unit test. These are the methods I found in generated unit tests file and they are working:
public void setUp() throws Exception {
super.setUp();
/*
* Add any necessary initialization code here (e.g., open a socket).
* Call Repository.putTemporary() to provide initialized instances of
* objects to be used when testing.
*/
// jtest.Repository.putTemporary("name", object);
}
/**
* Used to clean up after the test. This method is called by JUnit after
* each of the tests have been completed.
*
* #see junit.framework.TestCase#tearDown()
* #author Parasoft Jtest 9.5
*/
public void tearDown() throws Exception {
try {
/*
* Add any necessary cleanup code here (e.g., close a socket).
*/
} finally {
super.tearDown();
}
}

Is there a way to make Eclipse run a JUnit test multiple times until failure?

We occasionally have bugs that appear once in every X runs. Before people check in stuff (where it is automatically JUnit'd), our devs need to pass JUnit locally via Eclipse.
Is there some convenient way (built in or high-quality Plugin) to make Eclipse run the same test X times and stop if there's a failure? An alternative to just clicking Run X times?
Note that I'm looking for something in the UI (e.g., right click and say "Run X times" instead of just "Run").

If the for loop works, then I agree with nos.
If you need to repeat the entire setup-test-teardown, then you can use a TestSuite:
Right-click on the package containing the test to repeat
Go to New and choose to create a JUnit test SUITE
Make sure that only the test you want to repeat is selected and click through to finish.
Edit the file to run it multiple times.
In the file you just find the
addTestSuite(YourTestClass.class)
line, and wrap that in a for loop.
I'm pretty sure that you can use addTest instead of addTestSuite to get it to only run one test from that class if you just want to repeat a single test method.

If you really want to run a test class until failure, you need your own runner.
#RunWith(RunUntilFailure.class)
public class YourClass {
// ....
}
which could be implemented as follows...
package com.example;
import org.junit.internal.runners.*;
import org.junit.runner.notification.*;
import org.junit.runner.*;
public class RunUntilFailure extends Runner {
private TestClassRunner runner;
public RunUntilFailure(Class<?> klass) throws InitializationError {
this.runner = new TestClassRunner(klass);
}
#Override
public Description getDescription() {
Description description = Description.createSuiteDescription("Run until failure");
description.addChild(runner.getDescription());
return description;
}
#Override
public void run(RunNotifier notifier) {
class L extends RunListener {
boolean fail = false;
public void testFailure(Failure failure) throws Exception { fail = true; }
}
L listener = new L();
notifier.addListener(listener);
while (!listener.fail) runner.run(notifier);
}
}
...releasing untested code, feeling TDD guilt :)

Based on #akuhn's answer, here is what I came up with - rather than running forever, this will run 50 times or until failure, whichever comes first.
package com.foo
import org.junit.runner.Description;
import org.junit.runner.Runner;
import org.junit.runner.notification.Failure;
import org.junit.runner.notification.RunListener;
import org.junit.runner.notification.RunNotifier;
import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runners.model.InitializationError;
public class RunManyTimesUntilFailure extends Runner {
private static final int MAX_RUN_COUNT = 50;
private BlockJUnit4ClassRunner runner;
#SuppressWarnings("unchecked")
public RunManyTimesUntilFailure(final Class testClass) throws InitializationError {
runner = new BlockJUnit4ClassRunner(testClass);
}
#Override
public Description getDescription() {
final Description description = Description.createSuiteDescription("Run many times until failure");
description.addChild(runner.getDescription());
return description;
}
#Override
public void run(final RunNotifier notifier) {
class L extends RunListener {
boolean shouldContinue = true;
int runCount = 0;
#Override
public void testFailure(#SuppressWarnings("unused") final Failure failure) throws Exception {
shouldContinue = false;
}
#Override
public void testFinished(#SuppressWarnings("unused") Description description) throws Exception {
runCount++;
shouldContinue = (shouldContinue && runCount < MAX_RUN_COUNT);
}
}
final L listener = new L();
notifier.addListener(listener);
while (listener.shouldContinue) {
runner.run(notifier);
}
}
}

I know it doesn't answer the question directly but if a test isn't passing every time it is run it is a test smell known as Erratic Test. There are several possible causes for this (from xUnit Test Patterns):
Interacting Tests
Interacting Test Suites
Lonely Test
Resource Leakage
Resource Optimism
Unrepeatable Test
Test Run War
Nondeterministic Test
The details of each of these is documented in Chapter 16 of xUnit Test Patterns.

Here is a post I wrote that shows several ways of running the tests repeatedly with code examples:
http://codehowtos.blogspot.com/2011/04/run-junit-test-repeatedly.html
You can use the #Parametrized runner, or use the special runner included in the post
There is also a reference to a #Retry implementation

I don't believe there's a built in way for junit to do exactly what you're asking for.
If multiple runs produce different result, you should have a unit test testing that case. Wich might be as simple as running a for loop in the relevant test cases.

It is possible to implement such an loop with TestRules (since JUnit 4.9)
A very simple implementation that runs every Test 10 times:
import org.junit.rules.TestRule;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;
public class SimpleRepeatRule implements TestRule {
private static class SimpleRepeatStatement extends Statement {
private final Statement statement;
private SimpleRepeatStatement(Statement statement) {
this.statement = statement;
}
#Override
public void evaluate() throws Throwable {
for (int i = 0; i < 10; i++) {
statement.evaluate();
}
}
}
#Override
public Statement apply(Statement statement, Description description) {
return new SimpleRepeatStatement(statement);
}
}
usage:
public class Run10TimesTest {
#Rule
public SimpleRepeatRule repeatRule = new SimpleRepeatRule();
#Test
public void myTest(){...}
}
For a more useful implementation based on an annotation that define which test method has to been executed how often have a look at this blog:
http://www.codeaffine.com/2013/04/10/running-junit-tests-repeatedly-without-loops/

Easy way of running the same junit test over and over?

Like the title says, I'm looking for some simple way to run JUnit 4.x tests several times in a row automatically using Eclipse.
An example would be running the same test 10 times in a row and reporting back the result.
We already have a complex way of doing this but I'm looking for a simple way of doing it so that I can be sorta sure that the flaky test I've been trying to fix stays fixed.
An ideal solution would be an Eclipse plugin/setting/feature that I am unaware of.

The easiest (as in least amount of new code required) way to do this is to run the test as a parametrized test (annotate with an #RunWith(Parameterized.class) and add a method to provide 10 empty parameters). That way the framework will run the test 10 times.
This test would need to be the only test in the class, or better put all test methods should need to be run 10 times in the class.
Here is an example:
#RunWith(Parameterized.class)
public class RunTenTimes {
#Parameterized.Parameters
public static Object[][] data() {
return new Object[10][0];
}
public RunTenTimes() {
}
#Test
public void runsTenTimes() {
System.out.println("run");
}
}
With the above, it is possible to even do it with a parameter-less constructor, but I'm not sure if the framework authors intended that, or if that will break in the future.
If you are implementing your own runner, then you could have the runner run the test 10 times. If you are using a third party runner, then with 4.7, you can use the new #Rule annotation and implement the MethodRule interface so that it takes the statement and executes it 10 times in a for loop. The current disadvantage of this approach is that #Before and #After get run only once. This will likely change in the next version of JUnit (the #Before will run after the #Rule), but regardless you will be acting on the same instance of the object (something that isn't true of the Parameterized runner). This assumes that whatever runner you are running the class with correctly recognizes the #Rule annotations. That is only the case if it is delegating to the JUnit runners.
If you are running with a custom runner that does not recognize the #Rule annotation, then you are really stuck with having to write your own runner that delegates appropriately to that Runner and runs it 10 times.
Note that there are other ways to potentially solve this (such as the Theories runner) but they all require a runner. Unfortunately JUnit does not currently support layers of runners. That is a runner that chains other runners.

With IntelliJ, you can do this from the test configuration. Once you open this window, you can choose to run the test any number of times you want,.
when you run the test, intellij will execute all tests you have selected for the number of times you specified.
Example running 624 tests 10 times:

With JUnit 5 I was able to solve this using the #RepeatedTest annotation:
#RepeatedTest(10)
public void testMyCode() {
//your test code goes here
}
Note that #Test annotation shouldn't be used along with #RepeatedTest.

I've found that Spring's repeat annotation is useful for that kind of thing:
#Repeat(value = 10)
Latest (Spring Framework 4.3.11.RELEASE API) doc:
org.springframework.test.annotation
Unit Testing in Spring

Inspired by the following resources:
blog post
this solution
commented version
Example
Create and use a #Repeat annotation as follows:
public class MyTestClass {
#Rule
public RepeatRule repeatRule = new RepeatRule();
#Test
#Repeat(10)
public void testMyCode() {
//your test code goes here
}
}
Repeat.java
import static java.lang.annotation.ElementType.ANNOTATION_TYPE;
import static java.lang.annotation.ElementType.METHOD;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
#Retention( RetentionPolicy.RUNTIME )
#Target({ METHOD, ANNOTATION_TYPE })
public #interface Repeat {
int value() default 1;
}
RepeatRule.java
import org.junit.rules.TestRule;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;
public class RepeatRule implements TestRule {
private static class RepeatStatement extends Statement {
private final Statement statement;
private final int repeat;
public RepeatStatement(Statement statement, int repeat) {
this.statement = statement;
this.repeat = repeat;
}
#Override
public void evaluate() throws Throwable {
for (int i = 0; i < repeat; i++) {
statement.evaluate();
}
}
}
#Override
public Statement apply(Statement statement, Description description) {
Statement result = statement;
Repeat repeat = description.getAnnotation(Repeat.class);
if (repeat != null) {
int times = repeat.value();
result = new RepeatStatement(statement, times);
}
return result;
}
}
PowerMock
Using this solution with #RunWith(PowerMockRunner.class), requires updating to Powermock 1.6.5 (which includes a patch).

Anything wrong with:
#Test
void itWorks() {
// stuff
}
#Test
void itWorksRepeatably() {
for (int i = 0; i < 10; i++) {
itWorks();
}
}
Unlike the case where you are testing each of an array of values, you don't particularly care which run failed.
No need to do in configuration or annotation what you can do in code.

This works much easier for me.
public class RepeatTests extends TestCase {
public static Test suite() {
TestSuite suite = new TestSuite(RepeatTests.class.getName());
for (int i = 0; i < 10; i++) {
suite.addTestSuite(YourTest.class);
}
return suite;
}
}

There's an Intermittent annotation in the tempus-fugit library which works with JUnit 4.7's #Rule to repeat a test several times or with #RunWith.
For example,
#RunWith(IntermittentTestRunner.class)
public class IntermittentTestRunnerTest {
private static int testCounter = 0;
#Test
#Intermittent(repition = 99)
public void annotatedTest() {
testCounter++;
}
}
After the test is run (with the IntermittentTestRunner in the #RunWith), testCounter would be equal to 99.

This is essentially the answer that Yishai provided above, re-written in Kotlin :
#RunWith(Parameterized::class)
class MyTest {
companion object {
private const val numberOfTests = 200
#JvmStatic
#Parameterized.Parameters
fun data(): Array<Array<Any?>> = Array(numberOfTests) { arrayOfNulls<Any?>(0) }
}
#Test
fun testSomething() { }
}

I build a module that allows do this kind of tests. But it is focused not only in repeat. But in guarantee that some piece of code is Thread safe.
https://github.com/anderson-marques/concurrent-testing
Maven dependency:
<dependency>
<groupId>org.lite</groupId>
<artifactId>concurrent-testing</artifactId>
<version>1.0.0</version>
</dependency>
Example of use:
package org.lite.concurrent.testing;
import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;
import ConcurrentTest;
import ConcurrentTestsRule;
/**
* Concurrent tests examples
*/
public class ExampleTest {
/**
* Create a new TestRule that will be applied to all tests
*/
#Rule
public ConcurrentTestsRule ct = ConcurrentTestsRule.silentTests();
/**
* Tests using 10 threads and make 20 requests. This means until 10 simultaneous requests.
*/
#Test
#ConcurrentTest(requests = 20, threads = 10)
public void testConcurrentExecutionSuccess(){
Assert.assertTrue(true);
}
/**
* Tests using 10 threads and make 20 requests. This means until 10 simultaneous requests.
*/
#Test
#ConcurrentTest(requests = 200, threads = 10, timeoutMillis = 100)
public void testConcurrentExecutionSuccessWaitOnly100Millissecond(){
}
#Test(expected = RuntimeException.class)
#ConcurrentTest(requests = 3)
public void testConcurrentExecutionFail(){
throw new RuntimeException("Fail");
}
}
This is a open source project. Feel free to improve.

You could run your JUnit test from a main method and repeat it so many times you need:
package tests;
import static org.junit.Assert.*;
import org.junit.Test;
import org.junit.runner.Result;
public class RepeatedTest {
#Test
public void test() {
fail("Not yet implemented");
}
public static void main(String args[]) {
boolean runForever = true;
while (runForever) {
Result result = org.junit.runner.JUnitCore.runClasses(RepeatedTest.class);
if (result.getFailureCount() > 0) {
runForever = false;
//Do something with the result object
}
}
}
}