I ran the following test class and it reported that the first test took 2 seconds and the second took 4 seconds. I expected JUnit to output the time for each test case without putting into account the time of the #BeforeClass or #AfterClass method.
But obviously it includes the time of the #AfterClass method into only the last test case, which doesn't make sense at all if you ask me.
Is there a way make JUnit output the times only for the time taked for each test case ?
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
public class TimeTest {
#Test
public void test1() throws InterruptedException {
Thread.sleep(2000);
}
#Test
public void test2() throws InterruptedException {
Thread.sleep(2000);
}
#AfterClass
public static void afterClass() throws InterruptedException {
Thread.sleep(2000);
}
}
#BeforeClass & #AfterClass run ones for all tests
what you should do is to use #Before, and After instead, with the help of an instance variable
public class TimeTest {
long timeInMillis=0;
#Before
public void before(){
timeInMillis = System.currentTimeMillis();
}
#Test
public void test1() throws InterruptedException {
Thread.sleep(2000);
}
#Test
public void test2() throws InterruptedException {
Thread.sleep(2000);
}
#After
public static void after() throws InterruptedException {
long diff = System.currentTimeMillis() - timeInMillis;
// log the diff or cast it to seconds
}
}
You are not the first one with such a problem. There are a lot of examples of time logging on the internet.
I suggest you to use JUnit #Rules to solve your problem and avoid a code-duplication. (You don't want to write timeInMillis = System.currentTimeMillis() in each test class, do you?)
There is an example of rule implementation
Related
I have 1000+ test cases loaded from file and run within file.
Since the tests can take overly long I introduced I timeout for all of them:
#ClassRule
public static Timeout classTimeout = new Timeout(200);
This was working fine in Junit4, but now, once I migrated the test to Junit5, I can see no support for #ClassRule.
There's #Timeout annotation but it works per test method.
I couldn't find functionality like this provided by junit, so i came up with a somewhat naive implementation to fail every test when cumulative execution time exceeds a limit.
What is needed:
BeforeAllCallback.beforeAll()
Callback that is invoked once before all tests in the current container.
BeforeTestExecutionCallback.beforeTestExecution()
Callback that is invoked immediately before an individual test is executed but after any user-defined setup methods have been executed for that test.
AfterTestExecutionCallback.afterTestExecution()
Callback that is invoked immediately after an individual test has been executed but before any user-defined teardown methods have been executed for that test.
public class AllTestsTimeoutException implements BeforeAllCallback, BeforeTestExecutionCallback, AfterTestExecutionCallback {
private final long maxElapsedTimeNanos;
private long start;
public AllTestsTimeoutException(long maxElapsedTimeNanos) {
this(maxElapsedTimeNanos, TimeUnit.NANOSECONDS);
}
public AllTestsTimeoutException(long value, TimeUnit timeUnit) {
this.maxElapsedTimeNanos = timeUnit.toNanos(value);
}
#Override
public void beforeAll(ExtensionContext context) throws Exception {
start = System.nanoTime();
}
#Override
public void beforeTestExecution(ExtensionContext context) throws Exception {
checkElapsedTimes(context, "Before check");
}
private void checkElapsedTimes(ExtensionContext context, String prefix) {
long elapsedNanos = System.nanoTime() - start;
System.out.println(prefix + " - " + context.getRequiredTestMethod().getName() + " - " + elapsedNanos);
if (context.getExecutionException().isEmpty() && elapsedNanos > maxElapsedTimeNanos) {
throw new RuntimeException("all tests time out");
}
}
#Override
public void afterTestExecution(ExtensionContext context) throws Exception {
checkElapsedTimes(context, "After check");
}
}
It needs some polishing, but it can serve as starting point to improve from.
Registration is as a public static field in the class with RegisterExtension.
public class TimeoutTests {
#RegisterExtension
public static AllTestsTimeoutException timeout = new AllTestsTimeoutException(25, TimeUnit.MILLISECONDS);
#Test
public void test1() {
//test stuff
}
#Test
public void test2() {
//test stuff
}
#Test
public void test3() {
//test stuff
}
}
I've created a custom runner by extending Suite:
public class MyRunner extends Suite {
public MyRunner(Class<?> klass, RunnerBuilder builder) throws InitializationError {
super(klass, builder);
}
#Override
public void run(RunNotifier notifier) {
notifier.addListener(new MyRunListener());
notifier.fireTestRunStarted(getDescription());
super.run(notifier);
}
}
As seen in run this registeres a MyRunListener which looks like this:
class MyRunListener extends RunListener {
#Override
public void testRunStarted(Description description) {
System.err.println("1: run started");
}
#Override
public void testStarted(Description description) {
System.err.println("2: test started");
}
#Override
public void testFinished(Description description) {
System.err.println("3: test finished");
}
#Override
public void testRunFinished(Result result) {
System.err.println("4: run finished");
}
}
I've added 1:, 2:, 3: and 4: according to the order in which I expect these methods to be called.
I've then created a test suite that looks as follows:
#RunWith(MyRunner.class)
#Suite.SuiteClasses({ MyTest.class })
public class MyTestSuite {
}
And the following test:
public class MyTest {
#BeforeClass
public static void beforeClass() {
System.err.println("A: beforeClass");
}
#Before
public void before() {
System.err.println("B: before");
}
#Test
public void test() {
System.err.println("C: Running actual test...");
}
#After
public void after() {
System.err.println("D: after");
}
#AfterClass
public static void afterClass() {
System.err.println("E: afterClass");
}
}
Again, output labeled according to the order I expect.
Here's the output I get when I run MyTestSuite through IntelliJ (as a JUnit run configuration):
2: test started
B: before
C: Running actual test...
D: after
3: test finished
E: afterClass
4: run finished
1: run started
A: beforeClass
Why on earth am I getting that output order? I must be doing something very wrong in my runner, but I've just implemented it according to tutorials on the web. (Using JUnit 4.12)
As #DmitryB suggests in the comments, it seems to be some form of output buffering. The various parts of the output is then stitched together in a weird way after the test has finished.
I added Thread.sleep(1000) after each printout and this is how it looks when I run it in IntelliJ:
The workaround for now is to use the gradle task run configuration rather than the JUnit one.
Filed this ticket at Jetbrains.
In Intelij and Eclipse IDEs (and probably some others too) it's possible to run all test classes from a package (or even all test classes in a project) without the need to put each of them explicitly in a test suite class (this is something I want to avoid). Just right click -> run all tests and voilà!
I've got one problem with that approach to testing though. I want to do some cleaning up after all the tests are done, but no matter what I do, nothing seems to work.
At first, I tried using RunListener and its testRunFinished() method, but it is called after every atomic test is done, so not what I want when running many of them.
Then I thought about finalizers and runFinalizersOnExit(true), unfortunatelly, it is deprecated and worked only on one of computers that tests are executed on.
Last thing I tried was to create a "listener" thread, that - given tests execution start and end time differences - would clean up, for instance, after five seconds of test completion. I used code below to test that solution:
import org.junit.Test;
public class Main {
static {
System.out.println("In a static block!");
new Thread(new Runnable() {
public void run() {
System.out.println("Starting static thread!");
try {
while (true) {
Thread.sleep(1000);
System.out.println("Static thread working...");
}
} catch (InterruptedException e) {
System.err.println("Static thread interrupted!");
e.printStackTrace();
} catch (Exception e) {
System.err.println("Static thread catches exception!");
e.printStackTrace();
} finally {
System.err.println("Static thread in finally method.");
Thread.currentThread().interrupt();
}
}
}).start();
System.out.println("Exiting static block!");
}
#Test
public void test() throws Exception {
System.out.println("Running test!");
Thread.sleep(3000);
System.out.println("Stopping test!");
}
}
With no luck. The thread is killed after the test is done. And even the finally block is never executed...
In a static block!
Exiting static block!
Running test!
Starting static thread!
Static thread working...
Static thread working...
Stopping test!
Static thread working...
Desired behavior would be:
right click
run all tests
TestA is running...
TestA done
TestB is running...
TestB done
... more test classes...
cleanup
Not sure if I fully have your question right, but I think you want before, beforeClass, after and afterClass methods. i.e.
#BeforeClass
public void beforeClass() {
// Do stuff before test class is run
}
#Before
public void before() {
// Do stuff before each test is run
}
#After
public void after() {
// DO stuff after each test is run
}
#AfterClass
public void afterClass() {
// DO stuff after test class is run
}
You can do things on a more global level with some hacking or other frameworks. Spring's test suites for example. But I would try to keep such things within the scope of a single test class.
I've found a solution to my problem. My colleague suggested "hey, can't you just count the test classes?" - and that's what I did.
A little bit of reflection magic is used here, so the code might not be portable:
public abstract class CleaningTestRunner extends BlockJUnit4ClassRunner {
protected abstract void cleanupAfterAllTestRuns();
private static long TEST_CLASSES_AMOUNT;
private static long TEST_RUNS_FINISHED = 0;
private static boolean CLASSES_COUNTED = false;
static {
while (!CLASSES_COUNTED) {
try {
Field f = ClassLoader.class.getDeclaredField("classes");
f.setAccessible(true);
Vector<Class> classes = (Vector<Class>) f.get(CleaningTestRunner.class.getClassLoader());
TEST_CLASSES_AMOUNT = 0;
for (Class<?> klass : classes) {
if (klass.isAnnotationPresent(RunWith.class)) {
if (CleaningTestRunner.class.isAssignableFrom(klass.getAnnotation(RunWith.class).value())) {
for (Method method : klass.getMethods()) {
if (method.isAnnotationPresent(Test.class)) {
++TEST_CLASSES_AMOUNT;
break;
}
}
}
}
}
CLASSES_COUNTED = true;
} catch (Exception ignored) {
}
}
}
public CleaningTestRunner(Class<?> klass) throws InitializationError {
super(klass);
}
#Override
public void run(RunNotifier notifier) {
notifier.addListener(new TestCleanupListener());
super.run(notifier);
}
private class TestCleanupListener extends RunListener {
#Override
public void testRunFinished(Result result) throws Exception {
++TEST_RUNS_FINISHED;
if (TEST_RUNS_FINISHED == TEST_CLASSES_AMOUNT) {
cleanupAfterAllTestRuns();
}
}
}
}
I have a problem with Idea 14 and JUnit. I can't run #BeforeClass and #AfterClass methods in proper order (before all test and after all test). Every time order is different. I tried to reinstall IDEA, delete all settings but nothing works. Please help. This is example of my test code:
package com.rent.test;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import static org.junit.Assert.*;
import org.junit.Test;
public class testnewTest {
static int num;
static int num1;
#BeforeClass
public static void OnceExecutedBeforeAll() {
System.out.println("#BeforeClass: onceExecutedBeforeAll");
num = 15;
num1 = 16;
}
#AfterClass
public static void after() throws Exception {
System.out.println("End");
}
#Test
public void testLogin() throws Exception {
System.out.println("test");
assertEquals(15, num);
}
#Test
public void testGetOrdersDate() throws Exception {
System.out.println("test2");
assertEquals(16, num1);
}
}
This is output:
test2
#BeforeClass: onceExecutedBeforeAll
test
End
What you're likely observing is the fact that the output is not always going to be synchronous in the terminal. The test themselves are running in the correct sequence.
If they weren't, then you would have failures in test2 given that it would have appeared that your #BeforeClass method fired afterwards.
I'm trying to improve an existing system of automated Selenium test.
My goal is to repeat the tests that fails because of connections problem.
I've found and tried to follow this thread How to Re-run failed JUnit tests immediately? that revealed itself quite useful.
In my case the suite is composed by classes, so I've tried to substitute #Rule with #ClassRule, in order to repeat for each try also the #Before and #After parts.
I'm sorry for my ignorance, but where am I supposed to place this rule? In my Suite class? Or in the Classes representing the test?
I am the original answerer of How to Re-run failed JUnit tests immediately?
If I understand correctly, the problem that you are having is due to the #Before being executed before the code in the RetryRule, and the #After being executed afterwards.
So your current behaviour is something like:
#Before
#Retry
test code
#Retry
#After
But you can implement your #Before and #After as a rule - there is a rule ExternalResource which does exactly that. You would implement #Before and #After as a rule:
#Rule public ExternalResource beforeAfter = new ExternalResource() {
public void before() {
// code that was in #Before
}
public void after() {
// code that was in #After
}
}
Then you don't need the #Before and #After. You can then chain these rules using RuleChain. This forces an order of execution to your rules:
#Rule public RuleChain chain= RuleChain
.outerRule(new LoggingRule("outer rule")
.around(new LoggingRule("middle rule")
.around(new LoggingRule("inner rule");
so your final solution would be something like:
private ExternalResource beforeAfter = ...
private RetryRule retry = ...
#Rule public RuleChain chain = RuleChain
.outerRule(retry)
.around(beforeAfter);
Note that if you are using RuleChain, you no longer need the #Rule annotation on the ExternalResource and RetryRule, but you do on the RuleChain.
Here is my solution based on the one mentionned in the question.
It's a combinaison of a #Rule, FailedRule and a #ClassRule, RetryRule
public class RetryTest
{
public static class FailedRule implements TestRule
{
#Override
public Statement apply(final Statement base, final Description description)
{
return new Statement()
{
#Override
public void evaluate() throws Throwable
{
try
{
base.evaluate();
}
catch (Throwable t)
{
System.out.println(description.getDisplayName() + " failed");
retry.setNotGood();
if (retry.isLastTry())
{
System.out.println("No more retry !");
throw t;
}
else
{
System.out.println("Retrying.");
}
}
}
};
}
}
public static class RetryRule implements TestRule
{
private int retryCount, currentTry;
private boolean allGood = false;
public RetryRule(int retryCount)
{
this.retryCount = retryCount;
this.currentTry = 1;
}
public boolean isLastTry()
{
return currentTry == retryCount;
}
public void setNotGood()
{
allGood = false;
}
public Statement apply(final Statement base, final Description description)
{
return new Statement()
{
#Override
public void evaluate() throws Throwable
{
// implement retry logic here
for (; currentTry <= retryCount && !allGood; currentTry++)
{
allGood = true;
System.out.println("Try #" + currentTry);
base.evaluate();
}
}
};
}
}
#ClassRule
public static RetryRule retry = new RetryRule(3);
#Rule
public FailedRule onFailed = new FailedRule();
#BeforeClass
public static void before()
{
System.out.println("Before...");
}
#AfterClass
public static void after()
{
System.out.println("...After\n");
}
#Test
public void test1()
{
System.out.println("> test1 running");
}
#Test
public void test2()
{
System.out.println("> test2 running");
Object o = null;
o.equals("foo");
}
}
It gives :
Try #1
Before...
> test1 running
> test2 running
test2(RetryTest) failed
Retrying.
...After
Try #2
Before...
> test1 running
> test2 running
test2(RetryTest) failed
Retrying.
...After
Try #3
Before...
> test1 running
> test2 running
test2(RetryTest) failed
No more retry !
...After
If I am commenting the o.equals("foo"); in test2, everything runs fine in the firt try :
Try #1
Before...
> test1 running
> test2 running
...After
You decorate the test name itself with the #After or #Afterclass attributes:
#After
#Test
#Category(SmokeTests.class)
public void testProductPageOnly() throws TimeoutException {
//Some tests here.
}
#Afterclass
public static void SomeTest {
//Some test here.
}
Something to note, #Afterclass will always run; even if you are using a #Beforeclass that throws an exception.
May this can solve problem:
1) Test class should be inherited from junit.framework.TestCase
2) Run your tests with something like this
YourTestClass testClass = new YourTestClass();
TestResult result = testClass.run();
Enumeration<TestFailure> failures = result.failures();
if (result.failureCount() != 0)
{
TestFailure fail = failes.nextElement();
junit.framework.Test test = fail.failedTest();
test.run( result );
}
At the end result will contains last results of test running, so after analyzing what test was failed you can run it again.