I have a test case like this:
#Listeners({RetryListener.class})
public class TestClass {
#Test
public void test1() throws Exception {
//something
}
#Test(dependsOnMethods = "test1")
public void test2() throws Exception {
//something
}
#Test(dependsOnMethods = "test2")
public void test3() throws Exception {
//something
}
}
As you can see, there are dependencies on the tests. I would like to retry the entire test class once there is anything wrong with the tests.
Is there a way to do it in TestNG?
My RetryListener looks like this:
public class RetryListener extends TestListenerAdapter {
private int count = 0;
#Override
public void onTestFailure(ITestResult result) {
if (result.getMethod().getRetryAnalyzer() != null) {
Reporter.setCurrentTestResult(result);
if(result.getMethod().getRetryAnalyzer().retry(result)) {
count++;
result.setStatus(ITestResult.SKIP);
System.out.println("Error in " + result.getName() + " with status "
+ result.getStatus()+ " Retrying " + count + " of 3 times");
System.out.println("Setting test run attempt status to Skipped");
} else {
count = 0;
System.out.println("Retry limit exceeded for " + result.getName());
}
Reporter.setCurrentTestResult(null);
}
}
#Override
public void onTestSuccess(ITestResult result) {
count = 0;
}
}
Related
Screenshot isn't attached to allure results folder and in the allure report. I don't understand what's the problem. Have tried to add listeners in testng.xml and in the test class above the test name - no difference-> the png file isn't shown in the allure results when the test is failed. What am I doing wrong?
public class AllureReportListener implements ITestListener {
#Attachment(value = "Page screenshot", type = "image/png")
public byte[] saveScreenshotPNG (WebDriver driver) {
return ((TakesScreenshot) driver).getScreenshotAs(OutputType.BYTES);
}
#Override
public void onStart(ITestContext iTestContext) {
System.out.println("Starting Test Suite '" + iTestContext.getName() + "'.......");
iTestContext.setAttribute("WebDriver", BaseTest.getDriver());
}
#Override
public void onFinish(ITestContext iTestContext) {
System.out.println("Finished Test Suite '" + iTestContext.getName() + "'");
}
#Override
public void onTestStart(ITestResult iTestResult) {
System.out.println("Starting Test Method '" + getTestMethodName(iTestResult) + "'");
}
#Override
public void onTestSuccess(ITestResult iTestResult) {
System.out.println("Test Method '" + getTestMethodName(iTestResult) + "' is Passed");
}
#Override
public void onTestFailure(ITestResult iTestResult) {
System.out.println("Test Method '" + getTestMethodName(iTestResult) + "' is Failed");
if (BaseTest.getDriver() != null) {
System.out.println("Screenshot has captured for the Test Method '" + getTestMethodName(iTestResult) + "'");
saveScreenshotPNG(BaseTest.getDriver());
}
}
#Override
public void onTestSkipped(ITestResult iTestResult) {
System.out.println("Test Method '" + getTestMethodName(iTestResult) + "' is Skipped");
}
#Override
public void onTestFailedButWithinSuccessPercentage(ITestResult iTestResult) {
}
private static String getTestMethodName(ITestResult iTestResult) {
return iTestResult.getMethod().getConstructorOrMethod().getName();
}
}
I got 2 tests: addNewVideo and deleteRecentVideo where the second one depends on first one. After first fails, second gets ignored and it's not running. After opening my Extent Report, it looks like this:
I'm expecting to have 2 tests - 1 failed, 1 skipped, but it's not showing properly on the report.
ExtentListener.class
import com.aventstack.extentreports.*;
import com.aventstack.extentreports.markuputils.*;
import org.testng.*;
import java.util.*;
import static constants.FileResources.REPORT_DIR;
public class ExtentListener implements ITestListener {
private static Date d = new Date();
private static String fileName = String.format("%s%s%s%s", d.toString().replaceAll("[: ]", "_"), "_", System.getenv("env"), ".html");
private static ExtentReports extent = ExtentManager.createInstance(REPORT_DIR + fileName);
public static ThreadLocal<ExtentTest> testReport = new ThreadLocal<>();
#Override
public void onTestStart(ITestResult result) {
ExtentTest test = extent.createTest(result.getTestClass().getName() + "." + result.getMethod().getMethodName());
test.assignCategory(result.getTestClass().getName());
testReport.set(test);
}
#Override
public void onTestSuccess(ITestResult result) {
String methodName = result.getMethod().getMethodName();
String logText = methodName + " PASSED";
Markup m = MarkupHelper.createLabel(logText, ExtentColor.GREEN);
testReport.get().pass(m);
}
#Override
public void onTestFailure(ITestResult result) {
String methodName = result.getMethod().getMethodName();
String excepionMessage = Arrays.toString(result.getThrowable().getStackTrace());
testReport.get().fail("<details>" + "<summary>" + "<b>" + "<font color=" + "red>" + "Exception occured: Expand to check details"
+ "</font>" + "</b >" + "</summary>" + excepionMessage.replaceAll(",", "<br>") + "</details>" + " \n");
String failureLog = methodName + " FAILED";
Markup m = MarkupHelper.createLabel(failureLog, ExtentColor.RED);
testReport.get().log(Status.FAIL, m);
}
#Override
public void onTestSkipped(ITestResult result) {
String methodName = result.getMethod().getMethodName();
String logText = "<b>" + methodName + " SKIPPED" + "</b>";
Markup m = MarkupHelper.createLabel(logText, ExtentColor.ORANGE);
testReport.get().skip(m);
}
#Override
public void onTestFailedButWithinSuccessPercentage(ITestResult result) {
}
#Override
public void onStart(ITestContext context) {
}
#Override
public void onFinish(ITestContext context) {
if (extent != null) {
extent.flush();
}
}
}
I've tried implementing IInvokedMethodListener but no success.
Is there any way to show skipped tests properly on ExtentReport4?
Guys I also faced this issue and below code worked for me.
Define the testListener like this and then call this testListener class in your Baseclass by using #listeners(TestListeners.class) before Baseclass starts.
Note : I have used Spark Extentreport
public class TestListeners implements ITestListener {
#override
public void onTestSkipped(ITestResult result) {
Baseclass.extenttest = Baseclass.extent.createTest(result.getMethod().getDescription()).assignCategory("SkipedTest");
Baseclass.extenttest .log(Status.SKIP, result.getThrowable());
Baseclass.extenttest .log(Status.SKIP, result.getMethod().getDescription());
Baseclass.extenttest .log(Status.SKIP, MarkupHelper.createLabel(result.getName(), ExtentColor.YELLOW));
Baseclass.extent.flush();
}
}
In your test class define #Test like below
#Test(description = "Test Case name", dependsOnMethods = { "Name of method on which it depends" })
I tried IInvokeMethodListener2 once again, but it's just a workaround.
If method from dependsOnMethods failed, the test method was not executed at all, therefore it was not logged as skipped. My solution is to tag dependant methods as description and check before invocation if it contains any failed method from test class.
public class MethodListener implements IInvokedMethodListener2 {
#Override
public void beforeInvocation(IInvokedMethod method, ITestResult testResult, ITestContext context) {
List<String> failed = context
.getFailedTests()
.getAllMethods()
.stream()
.map(ITestNGMethod::getMethodName)
.collect(Collectors.toList());
boolean isSkippable = false;
if (failed.size() > 0 && method.getTestMethod().getDescription() != null) {
List<String> methodNames = Arrays.asList(method.getTestMethod().getDescription().split(";"));
isSkippable = failed
.stream()
.anyMatch(methodNames::contains);
}
if (isSkippable) {
throw new SkipException("Skipping " + method.getTestMethod().getMethodName());
}
}
#Override
public void afterInvocation(IInvokedMethod method, ITestResult testResult, ITestContext context) {
}
#Override
public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {
}
#Override
public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
}
}
#Listeners({MethodListener.class, ExtentListener.class})
public class SampleTest extends TestsBase {
#Test(priority = 1)
public void pass() {
System.out.println("pass");
}
#Test(priority = 2)
public void fail1() {
System.out.println("fail1");
Assert.fail();
}
#Test(priority = 3)
public void fail2() {
System.out.println("fail2");
Assert.fail();
}
#Test(priority = 4)
public void fail3() {
System.out.println("fail3");
Assert.fail();
}
#Test(priority = 5)
public void fail4() {
System.out.println("fail4");
Assert.fail();
}
#Test(priority = 6, description = "pass;fail2")
public void skip1() {
System.out.println("skip1");
}
#Test(priority = 7, description = "pass;fail3")
public void skip2() {
System.out.println("skip2");
}
}
Currently I am making the test case repeat 2 times, so how do I print the results as 2 separate results.
I tried using the built-in function to create the text, however, it does now show either "Success" or "Failure".
Currently I have this code:
public class UnitTestRunner {
static JUnitCore junitCore;
static Class<?> testClasses;
public static void main(String[] args) {
System.out.println("Running Junit Test Suite.");
Result result = JUnitCore.runClasses(TestSuite.class);
for (Failure failure : result.getFailures()) {
System.out.println(failure.toString());
}
System.out.println("Successful: " + result.wasSuccessful() +
" ran " + result.getRunCount() + " tests");
}
}
This code is working correctly, but I do not know how to implement this into JUnit.
Can someone please help to show, how to implement this code into JUnit test case.
This will be slightly long answer. For the customized output you have to add your RunListener
You can use following sample implementation for the same.
public class UnitTestRunner {
static JUnitCore junitCore;
static Class<?> testClasses;
public static void main(String[] args) {
System.out.println("Running Junit Test Suite.");
junitCore = new JUnitCore();
junitCore.addListener(new CustomExecutionListener());
Result result = junitCore.run(TestSuite.class);
for (Failure failure : result.getFailures()) {
System.out.println(failure.toString());
}
System.out.println("Successful: " + result.wasSuccessful() + " ran " + result.getRunCount() + " tests");
}
}
And implementation for the RunListener is as follows
import org.junit.runner.Description;
import org.junit.runner.Result;
import org.junit.runner.notification.Failure;
import org.junit.runner.notification.RunListener;
public class CustomExecutionListener extends RunListener {
public void testRunStarted(Description description) throws Exception {
System.out.println("Number of tests to execute: " + description.testCount());
}
public void testRunFinished(Result result) throws Exception {
System.out.println("Number of tests executed: " + result.getRunCount());
}
public void testStarted(Description description) throws Exception {
System.out.println("Starting: " + description.getMethodName());
}
public void testFinished(Description description) throws Exception {
System.out.println("Finished: " + description.getMethodName());
}
public void testFailure(Failure failure) throws Exception {
System.out.println("Failed: " + failure.getDescription().getMethodName());
}
public void testAssumptionFailure(Failure failure) {
System.out.println("Failed: " + failure.getDescription().getMethodName());
}
public void testIgnored(Description description) throws Exception {
System.out.println("Ignored: " + description.getMethodName());
}
}
And by overriding the methods in RunListener you can format you output.
I am trying to implement a javaScript error listener on the Selenium tests built in my Java project.
In order to detect console Java Script errors, I implemented a script in the JavaScript project which will push the console error messages at the bottom of the body, with a certain element containing a specific ID and the error message.
The error Listener I coded looks like:
public class ErrorListener implements Runnable{
WebDriver driver = null;
public ErrorListener (WebDriver driver)
{
this.driver = driver;
}
private void Listen ()
{
try
{
while(!driver.toString().contains("null")) //Chrome is alive
{
List<WebElement> list = driver.findElements(By.xpath("//div[#id='jsError']"));
if(list.size() != 0)
{
WebElement Error = list.get(0);
String ErrorMSG = Error.getText();
if(ErrorMSG != null)
{
Reporter.log("Found a javascript error with code: "+ErrorMSG);
Assert.assertTrue(false); // force the test failure
}
}
}
}catch(Exception ex)
{
ex.printStackTrace();
}
}
#Override
public void run()
{
Listen();
}
}
Which I am calling in my test method such as:
public class TC_NewAccount
{
public WebDriver driver;
public Thread listener;
#BeforeMethod
public void beforeTest()
{
Reporter.log("The test has just begun.");
driver = NewChrome.Generate(true);
ErrorListener errorlog = new ErrorListener(driver);
listener = new Thread(errorlog);
listener.start();
}
#AfterMethod
public void afterClass()
{
Reporter.log("Test has now finished.");
driver.close();
}
#DataProvider
public static Object[][] credentials()
{
return new Object[][] { { "id", "password" }};
}
#Test(dataProvider = "credentials")
public void Execute(String username, String password) throws Exception
{
... whatever the test does
}
}
The problem is, that since I am using the same instance of Chrome to constantly listen to a specific WebElement, I am throwing requests in parallel from both methods (the test and the listener), and the process ends up crashing because of them being at the same time and not being synced (one after another).
I realised that one option could be to check for errors after I do certain action every time, but this would greatly increment my lines of code, and it is only a partial solution because some errors may occur just in terms of milliseconds...
Is there a smooth solution for this problem?
Thanks in advance!
first of all you have to create you TestListener.java file
import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;
public class TestListener implements ITestListener {
#Override
public void onTestStart(ITestResult result) {
System.out.println("Started Test: " + result.getName());
}
#Override
public void onTestSuccess(ITestResult result) {
System.out.println("Finished Test: " + result.getName() + " :PASSED");
}
#Override
public void onTestFailure(ITestResult result) {
System.out.println("Finished Test: " + result.getName() + " :FAILED");
}
#Override
public void onTestSkipped(ITestResult result) {
System.out.println("Finished Test: " + result.getName() + " :SKIPPED");
}
#Override
public void onTestFailedButWithinSuccessPercentage(ITestResult result) {
System.out.println("Finished Test: " + result.getName()
+ " :FAILED BUT WITHIN SUCCESS PERCENTAGE");
}
#Override
public void onStart(ITestContext context) {
// TODO Auto-generated method stub
}
#Override
public void onFinish(ITestContext context) {
// TODO Auto-generated method stub
}
}
you can customize this file according to you . Next what you have to do is you need to mention this listener in your Test class
#Listeners(TestListener.class)
public class TestClass{
#Test(dataProvider = "TestData" , dataProviderClass = DataProviderUtil.class)
public void test(List data) throws Exception {
try{
// your code here
}
catch(Exception e){
Log.exception(e);
}finally {
Log.endTestCase();
}
}
}
My failed Junit test reruns once and stops but i am looking to have it run 3 times. I guess my annotation is screwed in some way. Pls what is the proper format for annotation. Note: I will also appreciate info about correct annotation for testing SuiteClasses. Below is my annotation
#Rule
public Retry retry = new Retry(3);
any help will be appreciated
public class HostTest {
/**
* Test method for {#link com.ibm.ws.ui.collective.internal.rest.resource.Host}.
*/
public class RetryTest {
/**
* #param i
*/
public RetryTest(int i) {
// TODO Auto-generated constructor stub
}
public class Retry extends TestRule {
private final int retryCount;
public Retry(int retryCount) {
this.retryCount = retryCount;
}
#Override
public Statement apply(Statement base, Description description) {
return statement(base, description);
}
private Statement statement(final Statement base, final Description description) {
return new Statement() {
#Override
public void evaluate() throws Throwable {
Throwable caughtThrowable = null;
// implement retry logic here
for (int i = 0; i < retryCount; i++) {
try {
base.evaluate();
return;
} catch (Throwable t) {
caughtThrowable = t;
System.err.println(description.getDisplayName() + ": run " + (i + 1) + " failed");
}
}
System.err.println(description.getDisplayName() + ": giving up after " + retryCount + " failures");
throw caughtThrowable;
}
};
}
}
}
#Test
public void sanityCheck_withHostPathsNull() {
Host h = new Host("myHost");
Map<String, Object> mapFromController = new HashMap<String, Object>();
h.setHostPaths(mapFromController);
assertFalse("it is not empty", h.getHostPaths().isEmpty());
}
}
Are you sure the failed tests were only running once? You might want to put a counter or print statement to confirm.
I'm not sure what you thought the Retry Rule would do but this code:
for (int i = 0; i < retryCount; i++) {
try {
base.evaluate();
return;
} catch (Throwable t) {
caughtThrowable = t;
System.err.println(description.getDisplayName() + ": run " + (i + 1) + " failed");
}
}
means that a test is only retried if it throws anything (meaning the test failed). Therefore, Retry(3) means to run each test and if any fail, then try running the failed tests upto 2 more times to see if it eventually passes. If after the final retry, if it still fails, then that final failure message is rethrown which will fail the test.
If you instead wanted to retry the tests over and over again even if the test passes, then you should get rid of that return statement inside the for loop. But then you have to be careful that you only rethrow the throwable at the end if it's not null. (meaning don't throw if the test passed the last time)
To make the test run over and over no matter if it passes:
#Override
public void evaluate() throws Throwable {
Throwable caughtThrowable = null;
// implement retry logic here
for (int i = 0; i < retryCount; i++) {
try {
base.evaluate();
} catch (Throwable t) {
caughtThrowable = t;
System.err.println(description.getDisplayName() + ": run " + (i + 1) + " failed");
}
}
if(caughtThrowable !=null){
System.err.println(description.getDisplayName() + ": giving up after " + retryCount + " failures");
throw caughtThrowable;
}
}