We Keep Coding

Selenium - abort the tests gracefully if a catastrophic error occurs - but generate test report

I have reviewed existing questions/answers about the this topic, and have not seen any that relate to my question.
To start off, my test automation uses two listeners - TestListener and CustomReportListener - along with TestNG framework with Selenium Webdriver to test a website.
Question is: How do I terminate rest of the tests (i.e. the rest of the test suite) if a critical condition is detected in one of the tests - but still generate test report that is mailed to the testers?
The test that ran into the critical issue would be marked Failed but the rest (ideally) would be marked Skipped.
The easiest way to terminate the tests is to use System.exit(1). I have tried that but it kills the test process - thereby, not creating a test report.

You can skip the remaining tests using your TestListener.
In you test methods, if the critical condition has occurred, then you have to set an attribute in the ITestContext object:
#Test
public void testCase(ITestContext ctx) {
//.....rest of the code....
if(/*the critical condition*/) {
ctx.setAttribute("criticalFailure", true);
}
// or if you are expecting an exception to identify the critical issue..
try {
// code which you expect may throw the critical exception
} catch (YourExpectedException e) {
ctx.setAttribute("criticalFailure", true);
}
}
Now in your listener, check if this attribute was set.
public class TestListener implements ITestListener {
#Override
public void onTestStart(ITestResult result) {
// Note that you should not use result.getAttribute because
// in the test method, we have set the attribute in the test context.
// (ITestResult is not available in test methods)
Object isCritical = result.getTestContext().getAttribute("criticalFailure");
if (isCritical != null && (boolean) isCritical) {
throw new SkipException("Critical error occurred");
}
}
}
If you are considering all failures as critical error, then it is much more easy. You don't have to change anything in your test methods. The only change would be in your test listener.
public class TestListener implements ITestListener {
#Override
public void onTestFailure(ITestResult result) {
result.setAttribute("failed", true);
}
#Override
public void onTestStart(ITestResult result) {
Object failed = result.getAttribute("failed");
if (failed != null && (boolean) failed) {
throw new SkipException("Critical error occurred");
}
}
}

How to embed every test in a wrapper logic?

Given the requirement that every junit test have to run in the following wrapper:
#Test
public void testFooBar() {
SpecialLogic.runWith(new SpecialLogic("blah", "foo", ANYTHING), () -> {
// my test
});
}
I am trying to avoid adding SpecialLogic.runWith(...) for each test.
Is there any possibility by using #BeforeEach or any other way?
Otherwise, there is much of duplicated code:
#Test
public void testFooBar_2() {
SpecialLogic.runWith(new SpecialLogic("blah", "foo", ANYTHING), () -> {
// my test logic 2
});
}
#Test
public void testFooBar_3() {
SpecialLogic.runWith(new SpecialLogic("blah", "foo", ANYTHING), () -> {
// my test logic 3
});
}

There are two ways of doing this:
Write your custom Runner, all the tests will have to run with this runner.
This may be inappropriate if you already use another runner (say for spring or mockito)
Write your own Rule. The rule is a little bit newer way of doing what you've asked for,
and it doesn't "occupy" the slot of a runner which can be only one.
public final class SampleRule implements TestRule {
#Override public Statement apply(final Statement base,
final Description description) {
return new Statement() {
#Override public void evaluate() throws Throwable {
// do your stuff before actually running the test
try {
base.evaluate(); // This line actually runs the test.
} finally {
// do your stuff after running a test
}
}
};}}
Here is one of numerous guides for writing Rules:

Looks like you should implement your own TestRunner to wrap your custom logic around each test method call. There is an article over at Baelung explaining how this works.

#Before and #After? It won't use closures but should be functionally the same.
https://junit.org/junit4/javadoc/latest/org/junit/Before.html
https://junit.org/junit4/javadoc/latest/org/junit/After.html

EasyMock verifying calls to mock in tearDown method after verify finished

I am seeing inconsistent behaviour in EasyMock tests that I don't understand.
My first test passes..
public class MockATest {
private final AtomicLong aMock = createStrictMock(AtomicLong.class);
#Before
public void setUp() {
aMock.set(101L);
}
#After
public void tearDown() {
aMock.set(999L);
}
#Test
public void testA() {
reset(aMock);
replay(aMock);
// TODO : test stuff here
verify(aMock);
}
}
.. but my second test fails ...
public class MockBTest {
private final List<Long> bMock = createStrictMock(List.class);
#Before
public void setUp() {
bMock.add(101L);
}
#After
public void tearDown() {
bMock.add(999L);
}
#Test
public void testB() {
reset(bMock);
replay(bMock);
// TODO : test stuff here
verify(bMock);
}
}
The failure reason is
Unexpected method call List.add(999)
I have 2 questions really...
Why is the behaviour different for the 2 tests?
Why is the add(999L) that happens in the tearDown method is being verified after the verification in the testB method has already fully completed?
(I know I can make this work by adding another reset(bMock) in after the verify(bMock) but I am not sure whether this is just avoiding the issue)

Why is the behaviour different for the 2 tests?
Because AtomicLong.set is typed void AtomicLong.set(long) so it's a void method. The recording is fine. However, List.add is typed boolean List.add(E) so it's not a void method. The correct way to record a non-void method is to do expect(list.add(101L)).andReturn(true).
Why is the add(999L) that happens in the tearDown method is being verified after the verification in the testB method has already fully completed?
Because it never goes in testB(). EasyMock throws an error on the call to bMock.add(101L) in setUp() so it goes directly to the tearDown which fail as well and hides to exception from setUp().

How to mark #Test as failed in #AfterMethod

I am trying to find out a way if there is any way in TetstNG to mark a test method annotated with #Test as failed inside #AfterMethod.
#Test
public void sampleTest() {
// do some stuff
}
#AfterMethod
public void tearDown() {
// 1st operation
try {
// some operation
} catch(Exception e) {
// mark sampleTest as failed
}
// 2nd operation
try {
// perform some cleanup here
} catch (Exception e) {
// print something
}
}
I have some verification to be done in all tests, which I am doing under 1st try-catch block in tearDown(). If there is an exception in that block, mark the test as failed. Then proceed for next try-catch block.
I cannot reverse the order of try-catch blocks in tearDown() because, 1st block depends on 2nd.

To the best of my knowledge you cannot do it from within #AfterMethod configuration method, because the ITestResult object that gets passed to your configuration method [ Yes you can get access to the test method's result object by adding a parameter ITestResult result to your #AfterMethod annotated method ] is not used to update back the original test method's result.
But you can easily do this if you were to leverage the IHookable interface.
You can get more information on IHookable by referring to the official documentation here.
Here's an example that shows this in action.
import org.testng.IHookCallBack;
import org.testng.IHookable;
import org.testng.ITestResult;
import org.testng.annotations.Test;
public class TestClassSample implements IHookable {
#Test
public void testMethod1() {
System.err.println("testMethod1");
}
#Test
public void failMe() {
System.err.println("failMe");
}
#Override
public void run(IHookCallBack callBack, ITestResult result) {
callBack.runTestMethod(result);
if (result.getMethod().getMethodName().equalsIgnoreCase("failme")) {
result.setStatus(ITestResult.FAILURE);
result.setThrowable(new RuntimeException("Simulating a failure"));
}
}
}
Note: I am using TestNG 7.0.0-beta7 (latest released version as of today)

How to run tearDown type method for a specific test in JUnit class with multiple tests?

I have a junit testCase class with multiple test methods in it ( As requirement , we don't want to create separate class for each test.)
I wanna create a tearDown type method for EACH test method , which will run specifically for that test. Not for ALL test.
My problem is , in many tests i Insert record in database, test it and delete it after test.
But, If a test fails mid way , control don't reaches till end my dummy record ain't deleting.
I think only ONE tearDown() is allowed for one class, and this tearDown() don't know what object/record i created or inserted and what to delete!!!
I want to create a tearDown() or #After method just for one specific test. Something like finally{} in java for each method.
For Eg:
public class TestDummy extends TestCase {
public void testSample1(){
InsertSomeData1();
assertFalse(true);
runTearDown1();
}
public void testSample2(){
InsertSomeData2();
assertFalse(true);
runTearDown2();
}
public void runTearDown1(){
deleteDummyDatafromTestSample1....
}
public void runTearDown2(){
deleteDummyDatafromTestSample2....
}
}
Here control will never go to runTearDown1() or runTearDown2() and I don't a one common tearDown() because it won't know what data I inserted and thats specific to each method.

It seems your test relies on a fixed database, and future tests will break if your current test breaks. What I'd recommend is not to focus on this particular problem (a test-specific tearDown method that runs for each test), but your main problem - borken tests. Before your test run, it should always work with a clean database, and this should be the case for each test. Right now, your first test has a relationship with the second (through the database).
What the right approach would be is that you recreate your database before each test, or at the very least reset it to a basic state. In this case, you'll want a test like this:
public class TestDummy {
// this code runs (once) when this test class is run.
#BeforeClass
public void setupDatabase() {
// code that creates the database schema
}
// this code runs after all tests in this class are run.
#AfterClass
public void teardownDatabase() {
// code that deletes your database, leaving no trace whatsoever.
}
// This code runs before each test case. Use it to, for example, purge the
// database and fill it with default data.
#Before
public void before() {
}
// You can use this method to delete all test data inserted by a test method too.
#After
public void after() {
}
// now for the tests themselves, we should be able to assume the database will
// always be in the correct state, independent from the previous or next test cases.
#Test
public void TestSample2() {
insertSomeData();
assertTrue(someData, isValid());
}
}
Disclaimer: JUnit 4 tests (using annotations), might not be the right annotations, might not even be the right answer(s).

You could have smth like this:
interface DBTest {
void setUpDB();
void test();
void tearDownDB();
}
class DBTestRunner {
void runTest(DBTest test) throws Exception {
test.setUpDB();
try {
test.test();
} finally {
test.tearDownDB();
}
}
}
public void test48() throws Exception {
new DBTestRunner().runTest(new DBTest() {
public void setUpDB() {...}
public void test() {...}
public void tearDownDB() {...}
});
}

#iluxa . Gr8.. Your solution is perfect!!! In one test class i created two tests test48 and test49 (same as required in my code above testSample1 and testSample2) and viola! every test method now gets its own setup() and tearDown. Only this solution looks little complicated as need to use DBTestRunner in each method, but I don't see any better solution. I was thinking Junit may have some direct solution. like #After or tearDown() with some parameter or something.
Tks a lot.

Use MethodRule:
public class MyRule implements MethodRule {
#Override
public Statement apply(final Statement base, FrameworkMethod method, Object target) {
return new Statement() {
#Override
public void evaluate() throws Throwable {
try {
base.evaluate();
} catch (AssertionError e) {
doFail();
} finally {
doAnyway();
}
}
};
}
}
Then declare it in your test class:
public class TestDummy{
public MethodRule rule = new MyRule();
......
}