TestNG - configfailurepolicy="continue" is not working for retried test - java

Though configfailurepolicy value is set as "continue", the retried test is getting skipped.
Example:
testng.xml - At suite level, configfailurepolicy value is set as "continue"
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite" parallel="classes" thread-count="1"
configfailurepolicy="continue">
<test name="Front-End">
<classes>
<class
name="com.tests.ExcepionAfterMethodTest">
</class>
</classes>
</test>
</suite>
Test: Throwing an exception in configuration (i.e., #AfterMethod) method to cause failure. And using RetryAnalyzer to retry the failed test.
import org.testng.Assert;
import org.testng.annotations.Test;
public class ExcepionAfterMethodTest{
#BeforeMethod(alwaysRun=true)
public void beforeMethod() {
System.out.println("Before Method");
}
#Test (alwaysRun=true, retryAnalyzer=RetryAnalyzer.class)
public void method() {
System.out.println("Method");
Assert.assertTrue(false);
}
#Test (alwaysRun=true)
public void method1() {
System.out.println("Method1");
}
#AfterMethod(alwaysRun=true)
public void afterMethod() throws Exception {
System.out.println("After Method");
throw new Exception();
}
}
RetryAnalyzer: Will retry once, if a failure occurs.
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
public class RetryAnalyzer implements IRetryAnalyzer {
private int counter = 0;
private int retryLimit = 1;
#Override
public boolean retry(ITestResult result) {
if (counter < retryLimit) {
counter++;
return true;
}
return false;
}
}
Result: Because of failure in #AfterMethod, the test is retried, but the #Test is skipped. Hence, 'Method' is not printed in the result at the 5th line.
Before Method
Method
After Method
Before Method
After Method
Method1
After Method
Expected Result: As configfailurepolicy is configured as "continue", the retried test shouldn't be skipped. So, expecting 'Method' should be printed in the result at the 5th line as below.
Before Method
Method
After Method
Before Method
Method
After Method
Method1
After Method

The issue is fixed with 7.1.0 version of TestNG.
Ref: https://github.com/cbeust/testng/issues/2148

Related

How to avoid executing the method written in #AfterSuite if a single test is executed as 'TestNG Test' or using the 'Run' command provided by plugin

BaseTest Class
public class BaseTest extends Base {
LoginPage login;
Logger logger = Logger.getLogger(Base.class);
#BeforeMethod()
public void setUp() throws IOException, IOException {
logger.info("starting initialisation method from base class");
initialisation();
logger.info("completed initialisation method from base class");
login = new LoginPage();
logger.info("initialising login page and profile page object");
}
#AfterMethod
public void tearDown() {
logger.info("starting tear down method");
Base.getDriver().close();
logger.info("ending tear down method");
}
#AfterSuite
public void sendEmailWithExtentReport() throws IOException {
logger.info("starting sendEmailWithExtentReport");
Utilities.sendJavaMailAfterExecution();
logger.info("ending sendEmailWithExtentReport");
}
LoginTest class
public class LoginTest extends BaseTest {
#Test(priority = 0,description = "Test case to verify login button is present or not")
public void loginPageLoadingTest() throws IOException, InterruptedException {
logger.info("starting loginPageLoadingTest");
Thread.sleep(2000);
Assert.assertEquals(login.verifyLoginButton(), true);
logger.info("ending loginPageLoadingTest");
}
#Test(priority = 1, description = "Test case to verify login page title")
public void loginPageTitleTest() throws IOException, InterruptedException {
logger.info("starting loginPageTitleTest");
Thread.sleep(2000);
Assert.assertEquals(login.getLoginPageTitle(), "Console");
logger.info("ending loginPageTitleTest");
}
}
testng-console.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Suite">
<listeners>
<listener class-name="com.analyzer.MyTransformer" />
<listener class-name="com.utilities.TestListener"></listener>
</listeners>
<test thread-count="5" name="Test">
<classes>
<class name="com.ui.tests.BaseTest" />
<class name="com.ui.tests.LoginTest" />
<class name="com.ui.tests.FilterTest" /> -->
</classes>
</test> <!-- Test -->
</suite> <!-- Suite -->
I need to skip the method in #AfterSuite if i am only executing the single test case from multiple test cases in LoginTest class without using testng.xml file, so that I can verify each test case separately. Right now when individual test is executed the method in #AfterSuite is also getting executed. I want to execute the method in #AfterSuite only when test cases are executed using testng.xml file. How can i add a condition in #AfterSuite to achieve this. Also i am using this command '''mvn clean test -Dbrowsername=chrome''' from command prompt. My testng file name is testng-console.xml .The method in #AfterSuite should only get executed when using the tesng-console.xml. Please help.
You can test your suite name for equality to Default Suite. Say you have testng.xml like:
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd" >
<suite name="Suite1" verbose="1" >
<test name="MyTest" >
<classes>
<class name="click.webelement.Suites" />
</classes>
</test>
</suite>
where you define suite name explicitly. Then you can have the logic like this:
package click.webelement;
import org.testng.ITestContext;
import org.testng.annotations.AfterSuite;
import org.testng.annotations.Test;
public class Suites {
#Test
public void doTest(){
System.out.println("Doing test..");
}
#AfterSuite
public void doingAfterSuite(ITestContext testContext){
if(!testContext.getSuite().getName().equalsIgnoreCase("Default Suite")){
System.out.println("Doing after suite");
}
}
}
Since when you run a single test, Default Suite is created you can just test if default suite is set in test context.
UPDATE: Eclipse plugin sets default suite name as Default suite so that I changed equals to EqualsIgnoreCase to cover both Idea and Eclipse

IAnnotation transform method is not disabling the test in testng

Here my intention is , Based on the before method condition the #Test method needs to enabled or disabled But in the below code even though I am passing the test case name its not getting skipped ? Can anyone suggest me solution?
I need the BeforeTestMethod to check some logic in my actual code and based on that I have to enable the #Test in the class file
public class ListnerClass implements IAnnotationTransformer {
public static String testName;
public void transform(ITestAnnotation iTest, Class testClass, Constructor testConstructor, Method method) {
if(method.getName().equalsIgnoreCase(testName)) {
iTest.setEnabled(false);
}
}
public class TestNGTest3 {
#BeforeMethod
public void setUp(Method result) {
System.out.println("This is the before Method getting name "+result.getName());
if(result.getName().contains("3"))
{
ListnerClass.testName=result.getName();
}
}
#Test
public void testMethod3() {
System.out.println("This is the Method of Method");
}
#Test
public void testMethod4() {
System.out.println("Hi");
}
}
TestNG.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name=" Regression Suite">
<listeners>
<listener class-name="com.listners.ListnerClass" />
</listeners>
<test thread-count="1" name="Test">
<classes>
<class name="com.test.TestNGTest3" />
</classes>
</test> <!-- Test -->
</suite> <!-- Suite -->
Output:
This is the before Method getting name testMethod3
This is the Method of Method
This is the before Method getting name testMethod4
Hi
Finally I have achieve my intention with the help of the below post.
Why is throw new SkipException() skipping all my methods?
Here is an working sample:
public class TestClassTest {
static String name;
static int i=0;
`public static String testmethod() {
List<String> list= new ArrayList<String>();
list.add("Raja");
list.add("Raju");
list.add("Raj");
name=list.get(i);
i++;
return name;
}
public class DependsOnMethodClass extends TestClassTest {
#BeforeMethod()
public void beforecaller(Method m) {
if(!testmethod().equals("Raju")) {
System.out.println("This is the method going to be skipped
"+m.getName());
throw new SkipException("Data provided is not matching");
}
else {
System.out.println("This is the method not skipped"+m.getName());
}
}
#Test
public void getCall() {
System.out.println("This is the getCall method");
}
#Test()
public void getCall1() {
System.out.println("This is the getCall1 method");
}
}
Update of TestNG xml: (Important Change)
Adding the parameter configfailurepolicy="continue" will continue the test even we
are throwing skip exception.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite" configfailurepolicy="continue">
<test thread-count="5" name="Test">
<classes>
<class name="com.testng.newsuite.DependsOnMethodClass"/>
</classes>
</test> <!-- Test -->
</suite> <!-- Suite -->
Output:
This is the method going to be skipped getCall
This is the method not skipped getCall1
This is the getCall1 method

How to configure TestNG to run just a single test case in IntelliJ?

I have two separate packages in my project, one for integration tests and one for unit tests, my testng.xml looks as follows:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd" >
<suite name="All test cases" verbose="1" parallel="classes">
<test name="Integration Tests">
<classes>
<class name="com.sample.integration.ClassC"/>
<class name="com.sample.integration.ClassD"/>
</classes>
</test>
<test name="Unit tests">
<classes>
<class name="com.sample.unit.ClassA"/>
<class name="com.sample.unit.ClassB"/>
</classes>
</test>
</suite>
Class C:
public class ClassC {
#BeforeTest
public void beforeIntegrationTests() {
System.out.println("Before Integration tests");
}
#Test
public void classCMethod() {
System.out.println("Executing class C method");
}
}
Class D:
public class ClassD {
#Test
public void classDMethod() {
System.out.println("Executing class D method");
}
}
Class A:
public class ClassA {
#BeforeTest
public void beforeUnitTests() {
System.out.println("Before unit tests");
}
#Test
public void classAMethod() {
System.out.println("Executing class A method");
}
}
Class B:
public class ClassB {
#Test
public void classBMethod() {
System.out.println("Executing class B method");
}
}
If I run the entire test suite it works as expected as follows:
Before Integration tests
Executing class C method
Executing class D method
Before unit tests
Executing class A method
Executing class B method
However, if I try to either run/debug just classAMethod() from ClassA, it runs beforeUnitTests() [expected] and classAMethod() [expected], however it also runs beforeIntegrationTests() which is not expected. As per the official documentation: #BeforeTest: The annotated method will be run before any test method belonging to the classes inside the <test> tag is run.
How do I configure TestNG and/or IntelliJ to run this correctly?
Side Note: Although I can see that the beforeIntegrationTests() is getting run either by adding a breakpoint in the debug mode or by adding a Thread.sleep in the run mode, the output from this method does not get printed in final console output.
Firstly, is the expectation valid, as in I expect only beforeUnitTests() and classAMethod() to run if I run just the classAMethod().
No, all methods with #BeforeTest annotation will run before execution of method with #Test annotation.
Ideal way to handle this scenario is with groups.
Class A:
public class ClassA {
#BeforeTest(groups="unitTest")
public void beforeUnitTests() {
System.out.println("Before unit tests");
}
#Test(groups="unitTest")
public void classAMethod() {
System.out.println("Executing class A method");
}
}
Class B:
public class ClassB {
#Test(groups="unitTest")
public void classBMethod() {
System.out.println("Executing class B method");
}
}
Class C:
public class ClassC {
#BeforeTest(groups="integrationTest")
public void beforeIntegrationTests() {
System.out.println("Before Integration tests");
}
#Test(groups="integrationTest")
public void classCMethod() {
System.out.println("Executing class C method");
}
}
Class D:
public class ClassD {
#Test(groups="integrationTest")
public void classDMethod() {
System.out.println("Executing class D method");
}
}
testNG XML:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd" >
<suite name="All test cases" verbose="1" parallel="classes">
<groups>
<run>
<include name="unitTest"></include>
</run>
</groups>
<test name="Test Suite">
<classes>
<class name="com.sample.integration.ClassC"/>
<class name="com.sample.integration.ClassD"/>
<class name="com.sample.unit.ClassA"/>
<class name="com.sample.unit.ClassB"/>
</classes>
</test>
</suite>
You can make further configuration to this testNg XML as per your needs.

#AfterMethod being ignored when rerunning test in TestNG

I'm trying to build logic into my test suite so that if a test fails, it automatically gets rerun. When my test fails, it gets rerun as expected, but
my test suite contains an #AfterMethod which seems to be getting ignored when the test is getting rerun.
I've followed the solution given here:
https://www.softwaretestingmaterial.com/run-failed-test-cases-using-testng/
So I created the 2 classes:
package softwareTestingMaterial;
import org.testng.IRetryAnalyzer; import org.testng.ITestResult;
public class RetryFailedTestCases implements IRetryAnalyzer {
private int retryCnt = 0;
private int maxRetryCnt = 2;
public boolean retry(ITestResult result) {
if (retryCnt < maxRetryCnt) {
System.out.println("Retrying " + result.getName() + " again and the count is " + (retryCnt+1));
retryCnt++;
return true;
}
return false;
}
}
And
package softwareTestingMaterial;
import java.lang.reflect.Constructor;
import java.lang.reflect.Method;
import org.testng.IAnnotationTransformer;
import org.testng.IRetryAnalyzer;
import org.testng.annotations.ITestAnnotation;
public class RetryListenerClass implements IAnnotationTransformer {
#Override
public void transform(ITestAnnotation testannotation, Class testClass, Constructor testConstructor, Method testMethod) {
IRetryAnalyzer retry = testannotation.getRetryAnalyzer();
if (retry == null) {
testannotation.setRetryAnalyzer(RetryFailedTestCases.class);
}
}
}
My XML file for executing the tests looks as follows:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="Test Suite 1">
<listeners>
<listener class-name="org.testng.reporters.EmailableReporter2" />
<listener class-name="CustomUtils.RetryListenerClass" />
</listeners>
<test name="Test 1">
<classes>
<class name="WebTests.CheckoutTestWeb"/>
</classes>
</test>
</suite>
When #Test throws exceptions, #AfterMethod may not run. To ensure #AfterMethod to be executed, add alwaysRun attribute which will look like #AfterMethod(alwaysRun = true)

how to keep dependency in testng in different classes webdriver

I have a scenario like two different test classes, second test class result depends on the first one
public class class1
{
#test
public void sometest1()
{
// some functionality
}
}
public class class2
{
#test
public void sometest2()
{
// some different functionality
}
}
Here class2 need to be executed if class1 result is pass else class2 needs to be failed.(if no provision for the same i need to execute the test1 in class1 to be executed always first in selenium grid parallel run)
As said avoid please refer :http://testng.org/doc/documentation-main.html#annotations.
As per above comments, i hope you are trying use depends on for method which is in another class. In such a case you need to provide complete path ie package.class.method.
class1:
public class testingcode {
#Test
public void test1(){
System.out.println("test1");
}
}
Class2:
public class testcase1{
#Test(dependsOnMethods={"com.test.testingcode.test1"})
public void testcase1result() throws InterruptedException{
System.out.println("test");
}
}
Run from TestNG.xml file
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite" parallel="none" preserve-order="true">
<test name="Test">
<classes>
<class name="com.test.testingcode"/>
<class name="com.test.testcase1"/>
</classes>
</test> <!-- Test -->
</suite> <!-- Suite -->
Let me know, if it is not working..
Thank You,
Murali
There are some annotations in testng that allows you the flow that you want.
I have never worked on testng but I think you can find the suitable annotation here :http://testng.org/doc/documentation-main.html#annotations
might be dependsOnGroups help you
You can use groups and dependency in testng to run a test after some test have executed. There is no way(atleast to my knowledge) to fail the Class2 test if Class1 test failed.
public class Class1Test {
#Test(groups="RunFirst")
public void sometest1() {
Assert.fail("just to skip the other test");
}
}
public class Class2Test {
#Test(dependsOnGroups="RunFirst")
public void sometest2() {
System.out.println("I am running");
}
}
Here the Class2Test will run after Class1Test has run and will be skipped if Class1Test failed.
You can use dependsOnMethods like below
#Test
public void test1() {
// Instantiate another case class
// Call your testcase
}
#Test(dependsOnMethods="test1")
public void test2() {
}

Categories