We Keep Coding

How to stop TestNG from running after a test fail

I am trying to write a test method in TestNG, that after it fails - the entire test suite will stop running.
#Test
public void stopTestingIfThisFailed() throws Exception
{
someTestStesp();
if (softAsserter.isOneFailed()) {
asserter.fail("stopTestingIfThisFailed test Failed");
throw new Exception("Test can't continue, fail here!");
}
}
The exception is being thrown, but other test methods are running.
How to solve this?

You can use the dependsOnMethods or dependsOnGroups annotation parameter in your other test methods:
#Test(dependsOnMethods = {"stopTestingIfThisFailed"})
public void testAnotherTestMehtod() {
}
JavaDoc of the dependsOnMethods parameter:
The list of methods this method depends on. There is no guarantee on the order on which the methods depended upon will be run, but you are guaranteed that all these methods will be run before the test method that contains this annotation is run. Furthermore, if any of these methods was not a SUCCESS, this test method will not be run and will be flagged as a SKIP. If some of these methods have been overloaded, all the overloaded versions will be run.
See https://testng.org/doc/documentation-main.html#dependent-methods

It depends on what you expect (there is no direct support for this in TestNG). You can create ShowStopperException which is thrown in #Test and then in your ITestListener implementation (see docs) you can call System.exit(1 (or whatever number)) when you find this exeption in result but there will be no report and in general it's not good practice. Second option is to have some base class which is parent of all test classes and some context variable which will handle ShowStopperException in #BeforeMethod in parent class and throw SkipException so workflow can be like:
test passed
test passed
showstopper exception in some test
test skipped
test skipped
test skipped
...

I solved the problem like this: after a test that mustn't fail fails - I'm writing data to a temporary text file.
Later, in the next test I added code in the #BeforeClass that checks that data in the former mentioned text file. If a show stopper was found I'm killing the current process.
If a test the "can't" fail actually fails:
public static void saveShowStopper() {
try {
General.createFile("ShowStopper","tempShowStopper.txt");
} catch (ParseException e) {
e.printStackTrace();
}
}
The #BeforeClass validating code:
#BeforeClass(alwaysRun = true)
public void beforeClass(ITestContext testContext, #Optional String step, #Optional String suiteLoopData,
#Optional String group) throws Exception
{
boolean wasShowStopperFound = APIUtils.loadShowStopper();
if (wasShowStopperFound){
Thread.currentThread().interrupt();
return;
}
}

It behaves if you throw a specific exception, SkipException, from the #BeforeSuite setup method.
See (possible dupe)
TestNG - How to force end the entire test suite from the BeforeSuite annotation if a condition is met
If you want to do it from an arbitrary test, it doesn't appear there is a framework mechanism. But you could always flip a flag, and check that flag in the #BeforeTest setup method. Before you jump to that, maybe have a think if you could check once before the whole suite runs, and just abort there (ie #BeforeSuite).

SonarQube issue "Add at least one assertion to this test case" for unit test with assertions?

I'm having issues with SonarQube raising issues with several of my unit tests, prompting the following issue:
Add at least one assertion to this test case.
Each test case resembles this format (where a number of assertions are delegated to a method with common assertions, to avoid duplication):
#Test
public void companyNameOneTooLong() throws Exception {
AddressFormBean formBean = getValidBean();
formBean.setCompanyNameOne("123456789012345678901234567890123456");
assertViolation(validator.validate(formBean), "companyNameOne", "length must be between 0 and 35");
}
private void assertViolation(Set<ConstraintViolation<AddressFormBean>> violations, String fieldname, String message) {
assertThat(violations, hasSize(1));
assertEquals(fieldname, violations.iterator().next().getPropertyPath().iterator().next().getName());
assertEquals(message, violations.iterator().next().getMessage());
}
Now, obviously I could just pull the three assertions out of the private method and put them in the test method - but I'm performing the same checks (on different fields) multiple times.
So, I thought I'd try to emulate the behaviour of the assertion methods, by (re) throwing an AssertionError:
private void assertViolation(Set<ConstraintViolation<AddressFormBean>> violations, String fieldname, String message) throws AssertionError {
try {
assertThat(violations, hasSize(1));
assertEquals(fieldname, violations.iterator().next().getPropertyPath().iterator().next().getName());
assertEquals(message, violations.iterator().next().getMessage());
} catch (AssertionError e) {
throw e;
}
}
Unfortunately, this approach does not work either.
What's special about the JUnit assert methods / what is SonarQube looking for specifically to check that an assertion has been made for each test?
Alternatively - are there other approaches to achieve the same end result (avoiding duplicating the shared assertion code over and over)?

The rule S2699 (Tests should include assertions) from the SonarQube Java Analyzer does not perform cross-procedural analysis and only explore body of methods being identified as test method (usually annotated with #Test).
Consequently, if the only assertions which will be called when executing the test method are done by a dedicated method (to avoid duplication), then the rule will raise an issue. This is a known limitation of the rule and we will deal with it only when we will be able to efficiently perform cross-procedural analysis.
Regarding the issues raised by SonarQube on such cases, you can safely mark them as Won't Fix.
Regarding the detected assertions, the rule consider as assertions the usual assert/fail/verify/expect methods from the following (unit test) frameworks :
JUnit
Fest (1.x & 2.x)
AssertJ
Hamcrest
Mockito
Spring
EasyMock

If you don't expect any exception to be throw from your test, this can be a workaround:
#Test(expected = Test.None.class /* no exception expected */)
Alternatively, you can suppress the warning for the test method/test class:
#SuppressWarnings("squid:S2699")

One thing I have done in the past is to have the helper method return true, and assert on that:
#Test
public void testSomeThings() {
Thing expected = // . . .
Thing actual = service.methodReturningThing(42);
assertTrue(assertViolation(expected, actual));
}
private boolean assertViolation(Thing expected, Thing actual) {
assertEquals(expected.getName(), actual.getName());
assertEquals(expected.getQuest(), actual.getQuest());
assertEquals(expected.getFavoriteColor(), actual.getFavoriteColor());
return true;
}
I hate this, but I hate duplicated code even more.
The other thing we've done at times, is to simply mark any such objections from SonarQube as Won't Fix, but I hate that, too.

Sometimes you don't need to have any code or assertation, for example, the test of load the context of spring boot successfully. In this case, to prevent Sonar issue when you don't expect any exception to be throw from your test, you can use this part of the code:
#Test
void contextLoads() {
Assertions.assertDoesNotThrow(this::doNotThrowException);
}
private void doNotThrowException(){
//This method will never throw exception
}

JUnit: No runnable methods

I am newbie in JUnit. I am fixing JUnit in SonarQube report and also increasing Junit code coverage. I came across of a class where there is no #Test annotation. The Exception thrown is as below:
No runnable methods java.lang.Exception: No runnable methods at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
Test class below:
public class TestLimitPrice {
#BeforeClass
public static void setUpBeforeClass() {
JndiDataManager.init();
}
private LimitPriceBean limitPrice;
private LimitPriceValidator validator;
#Before
public void setUp() {
limitPrice = new LimitPriceBean();
validator = new LimitPriceValidator(limitPrice);
}}
My Question's are :
In Sonar Report is it necessary for every JUnit to have atleast one #Test to pass ?
Will empty #Test is good approach for increasing code coverage ?
If in case any test case is not executing, then assertEquals(true,true) is good practice or should be avoided ?
Update
Sonar Version 4.4.1
JUnit Version 4.12
Java 1.6-45

My Question's are :
In Sonar Report is it necessary for every JUnit to have at least one
#Test to pass ?
I don't understand the question.
Will empty #Test is good approach for increasing code
coverage ?
No, for two reasons.
First, if the #Test method is truly empty, then there's no possibility to increase coverage. So let's assume that you have a #Test method that does execute some of your program code but that contains no assertions.
In this scenario, you've increased your test coverage by executing program code in a #Test but totally subverted the purpose of tests by not making any statements (assertions) about the expected outcome / outputs of that code. Let's say I've got
public int addTwoAndTwo() {
return 2+2;
}
And a corresponding test
#Test public void testAddTwoAndTwo() {
MyClass mc = new MyClass();
my.addTwoAndTwo(); // this method now "covered"
}
Now that addTwoAndTwo is "covered" I'm supposed to be able to maintain it with confidence that as long as the unit tests continue to pass, I haven't broken anything.
So let's do that. My new version is this:
public int addTwoAndTwo() {
return 42;
}
After that change, my unit tests still succeed, so everything must be okay, right? Uhm... no. And that's why you need assertions:
#Test public void testAddTwoAndTwo() {
MyClass mc = new MyClass();
assertThat(mc.addTwoAndTwo()).isEqualTo(4); // this method now truly covered
}
If in case any test case is not executing, then
assertEquals(true,true) is good practice or should be avoided ?
The answer to this question should by now be obvious, but to be explicit, DO NOT DO THIS.

Testing methods which share variables

I have a test :
#Test
testLogin {
String returnValue = login(username, password)
}
Once this test completes I would to test the value of returnValue in a separate test. returnValue is also a method parameter for another method :
#Test
testVal {
performTest(returnValue)
}
To achieve this I could declare returnValue as a global variable and then ensure the tests run in order using #FixMethodOrder(MethodSorters.NAME_ASCENDING) but this seems like a non standard and unclean approach ?
Does this imply that the code which is being tested should be re-factored ?
What are the alternatives ?

Such test
#Test
public void testLogin() {
String returnValue = login(username, password)
}
doesn't have much value. The only thing it tests is that method login doesn't throw an exception and I am pretty sure that it is not what you want to test. Where are the assertions? The following implementation satisfies this test:
public String login(Object a, Object b) {
return null;
}
Your test should look like this:
#Test
public void successfulLoginReturnsExpectedResponseCode() {
String responseCode = login(username, password)
assertThat(responseCode, is(equalTo("SUCCESSFUL_LOGIN")));
}
If the login method does nothing, the return value will be incorrect.
This example uses hamcrest for better diagnostics and readability. You can also use assertEquals.
Regarding your question about tests sharing variables, it is very important that your tests (no matter whether they are unit test, integration tests or acceptance tests) do not share anything and are independent. Otherwise you will run into problems such as failing test because the other test didn't run or the tests were run in different order. You should be able to run any subset of tests in any order.
You should also consider a better name of the test, so it describes what the code under test should be doing. For example successfulLoginReturnsExpectedResponseCode().

JUnit 4.11 get test result in #After

Is there any way I can get the test result in the teardown (#After) method? I'd like to do clean up after the tests depending on the result.
Could not find much details about #After in the junit docs.

The closest thing to what you're asking for would probably be the TestWatcher rule. That won't give you access to a returned result or anything, but you can use it (or create your own TestRule and combined with the Description object, you could annotate your methods differently to indicate what sort of clean-up is necessary.

Yes, if you use TestNG, it is a standard function, your #After method can look like this:
#AfterTest
public void cleanUp( ITestResult result ) {
boolean success = result.isSuccess();
....

If there is no standard possibility (I'm pretty sure there was no possibility in JUnit 3.x), you can just
write a Listener,
push the Listener-events to a static Collection,
and gather them from your #After- Method.

Why not set the result of a test in a class member and then act on it in the #After method?
public enum TestResult {
...
}
public class TestClass {
private TestResult result;
...
#Test
public void aTest() {
// set up test
// call class under test
// assert something and set result based upon outcome
this.result = ...;
}
...
#After
public void teardown() {
// clean up based upon this.result
}
}
I suspect you would not have too many different results and a finite set will suffice.

I am using something alike JamesB suggested. You might get to the point where you have to add timeouts to the tests, then =>>
"setting the result of a test in a class member and then act on it in the #After method" would not always work if you have more than 1 assert. That's is my problem today, when i have testCaces that timeout, but my afterClass is assuming everything went smooth because the most recent assert has passed..