I have a java file which has 7 junit tests to run. If I run all the tests at once all but 1 passes. If I comment out certain tests and that one test always passes.
Can anybody offer any suggestions as to what could be causing this?
My first thought was something in the test Setup or cleanup but I am not sure what it could be. All I do in the clean up is exit the driver and output the time taken to run the test.
In the setup I set up the driver, the time started, create a firefox profile and read in some data from a properties file to use in the tests.
If it was the setup / cleanup surely the other 6 tests would also be effected? The test that fails is a simple test to check that entering an invalid card type displays an error message on the page.
UPDATE:
I've renamed the test so it runs first and now all 7 pass each time. What could be causing this? Do I need to set something in my test cleanup to get it back to a default state?
My test cleanup:
#After
public void testCleanup() throws IOException {
driver.quit();
endTime = System.currentTimeMillis();
long totalTime = ((endTime - startTime)/1000)/60;
System.out.println();
System.out.println("Test Suite Took: " + totalTime + " Minutes.");
}
Related
If we build and execute selenium (TestNG) suite in Jenkins and if some tests fails; after fixing is there any way to execute only those failed test cases in Jenkins?
Yes, from the docs:
Every time tests fail in a suite, TestNG creates a file called testng-failed.xml in the output directory. This XML file contains the necessary information to rerun only these methods that failed, allowing you to quickly reproduce the failures without having to run the entirety of your tests.
If I understand your use case, you are looking to save the list of failed tests, make changes to the code and re-execute that list, is that correct ? In that case you can store that testng-failed.xml file and use it for the next execution in Jenkins, possibly adding a checkbox to the job that lets you choose wether to use this test suite or the default one.
Please try below to run specific tests (example: testcases you have fixed) with mnetioning below include tag in testNG.xml
<classes>
<class name="test.IndividualMethodsTest">
<methods>
<include name="testMethod" />
</methods>
</class>
</classes>
From my QA and automation perspective, it's always better to run the entire suite from jenkins, if you want to check why they fail you can do it locally. the other option is to parameterize the xml but it is a lot of work, it could be by maven arguments and writing in the xml
Why in Jenkins?, you can build one Testng Retry Listener where it keeps polling the entire test execution and re-executes only Failed testcases.
Refer this below.
public class RetryFailedTestCases implements IRetryAnalyzer {
private int retryCnt = 0;
// Mentioned maxRetryCnt (Maximiun Retry Count) as per your requirement. Here I
// took 3, If any failed testcases then it runs two times
private int maxRetryCnt = 1;
// This method will be called everytime a test fails. It will return TRUE if a
// test fails and need to be retried, else it returns FALSE
public boolean retry(ITestResult result) {
if (retryCnt < maxRetryCnt) {
System.out.println("Retrying " + result.getName() + " again and the count is " + (retryCnt + 1));
retryCnt++;
return true;
}
return false;
}
}
I have a java project with some test files in the following location:
src\test\resources\data\file\daily
I have some Junit test cases that check and assert based on the file modified time.
FileTime modFileTime = Files.getLastModifiedTime(Paths.get(classPathResource.getFile().getPath()));
when I execute the test cases using intellij without maven, my test passes and the modFileTime has time from the past e.g. 16/04/21 19:48
However, my test cases are failing when I run the tests using maven clean test as the file modified timings in target\test-classes\data\file\daily directory get updated timings.
How can I preserve the original file modified timings? or is there a common solution for this?
The method being called with test:
private boolean isFileAvailable(String file) throws IOException {
ClassPathResource classPathResource = new ClassPathResource(file);
boolean exists = Files.exists(Paths.get(classPathResource.getFile().getPath()));
if (exists) {
FileTime modFileTime = Files.getLastModifiedTime(Paths.get(classPathResource.getFile().getPath()));
long modFileMinutes = modFileTime.to(TimeUnit.MINUTES);
long minutes = FileTime.from(Instant.now()).to(TimeUnit.MINUTES);
return minutes - modFileMinutes >= 5;
} else {
return false;
}
}
mvn clean is getting rid of everything in your target/ directory before running the tests, and repopulating it. Hence, the timestamp will change every run. But this also will be the case for an initial (clean) checkout of the project, which you should be doing before any release build, so ... this is a pretty normal thing to be happening.
However, I agree with all the comments -- your test (not posted) doesn't make a lot of sense. If you want to have your test check a file with a relative timestamp, then e.g. set the timestamp on the file to 4 minutes ago and confirm it's not loaded, then set it to 6 minutes ago and confirm it's loaded. You can set the last-modified value on your test file from within the test. This is much more reliable than relying on something in the test execution system (maven) itself, especially if you generate the test file as part of the test (a good idea)
Also: if you only want to load data files older than a certain time, then I doubt you really want to have those be classpath resources. They should probably be loaded from some other known location. I suspect you are trying to solve some problem with cleverness that would be better solved by something from, e.g., https://commons.apache.org/
I've 4 test cases defined using TestNG within my Appium project and all 4 test cases performs almost same code execution besides the difference that all 4 test cases run on different parts of the app I'm automating. The issue I'm facing is within my test case I need to find an element and get its value which works fine in Test Case 1 but fails for Test Case 2, 3 and 4 respectively. However if I comment out Test Case 1, then the code works fine for Test Case 2 but fails for Test Case 3 and 4. If I run code for Test Case 1, 2, 3 or 4 individually, all of them work. I'm not able to understand why the code works only for the first test case in execution but fails to find the element in the next test case.
Note that the only option for me to find element is using absolute xpath value since the app I'm trying to automate does not have any other attribute defined.
Environment
java client 4.1.2
Appium server version: v1.7.2
Selenium: v3.4.0
Desktop OS/version used to run Appium if necessary: Windows 10 x64
Mobile platform/version under test: Fire TV Stick running on Android 5.1.1
Real device or emulator/simulator: Real device
Details
I've provided the code for Test1 below. As mentioned earlier, Test2, Test3 and Test4 are same as Test1. The code below is the one which fails from second Test onwards even though the code works fine when tested individually and the element is definitely present on the screen when the code is executed. However the script returns no element found.
Failing code
strTime1 = driver.findElementByXPath("/hierarchy/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.FrameLayout[1]/android.widget.FrameLayout/android.widget.FrameLayout[3]/android.widget.FrameLayout[3]/android.widget.FrameLayout[2]/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.TextView").getText();
Code To Reproduce Issue
#test //Test Case for Home tab
public void Test1_Home() throws InterruptedException, ParseException{
Thread.sleep(delay3); //Wait for 10 seconds to load the content
driver.findElementByXPath("//android.widget.FrameLayout[#content-desc="Home, Button"]").click(); //Code to click Home button
Thread.sleep(delay2); //Wait for 10 seconds to load the content
((AndroidDriver) driver).pressKeyCode(23); //Code to click first video thumbnail button
Thread.sleep(delay2); //Wait for 10 seconds to load the content
//Video pause verification code starts
((AndroidDriver) driver).pressKeyCode(23); //Pause the video
Thread.sleep(1000); //Wait for 1 seconds
strTime1 = driver.findElementByXPath("/hierarchy/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.FrameLayout[1]/android.widget.FrameLayout/android.widget.FrameLayout[3]/android.widget.FrameLayout[3]/android.widget.FrameLayout[2]/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.TextView").getText();
System.out.println(strTime1);
Thread.sleep(delay1); //Wait for 5 seconds
strTime2 = driver.findElementByXPath("/hierarchy/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.FrameLayout[1]/android.widget.FrameLayout/android.widget.FrameLayout[3]/android.widget.FrameLayout[3]/android.widget.FrameLayout[2]/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.TextView").getText();
System.out.println(strTime2);
time1 = time.parse(strTime1);
time2 = time.parse(strTime2);
System.out.println(time1);
System.out.println(time2);
if(String.valueOf(strTime1).equals(String.valueOf(strTime2)))
{
boolean result = true; //Set result to true
System.out.println("Home: Video playback pause successful");
Assert.assertTrue(result); //Passes the test case
}
else
{
boolean result = false; //Set result to false
System.out.println("Home: Video playback pause failed");
Assert.assertTrue(result); //Fails the test case
}
//Video pause verification code ends
Thread.sleep(delay2); //Allow the video to play for 10 seconds including buffering time
}
Exception stack traces
org.openqa.selenium.NoSuchElementException: An element could not be located on the page using the given search parameters. (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 10.13 seconds
Let me know if any further information is required. Any kind of help would be appreciated
could you please try to locate the element with relative xpath or accesability id?
Also it is always best practice to use latest versions of appium server and java client as they are continuously making changes to the appium.
below is the link for latest java client if you are using maven : https://mvnrepository.com/artifact/io.appium/java-client
I like to customize and display more information for Test suites or tests like Test Run times, for eg: adding more information to below displayed output
===============================================
Demo-Suite
Total tests run: 19, Failures: 1, Skips: 0
===============================================
Any suggestions how to add more to above info like adding Average Test Suite run time etc.,
Here is the solution for you:
Let us assume we have a TestNG script with 3 Testcases, where 1 Testcase passes & 2 Testcases fails.
#Test
public void test1()
{
Assert.assertEquals(12, 13);
}
#Test
public void test2()
{
System.out.println("Testcase 2 Started");
Assert.assertEquals(12, 13, "Dropdown count doesnot match");
System.out.println("Testcase 2 Completed");
}
#Test
public void test3()
{
System.out.println("Testcase 3 Started");
Assert.assertEquals("Hello", "Hello", "Words doesnot match. Please raise a Bug.");
System.out.println("Testcase 3 Completed");
}
So you get the result on the console as: Tests run: 3, Failures: 2, Skips: 0
Now to look at the granular details you can do the following:
a. Move to the tab "Results of running class your_class_name". Here you will observe some more fine prints of the execution in-terms of Default Suite Execution Time, Default Test Execution Time, Time taken for each Individual Test, etc.
b. Now to view more details you can click on the "Open TestNG report" icon located on the top bar of "Results of running class your_class_name". This will provide you a lot more information about Testcase Results & Time taken.
Now if you need more detailed information in the form of Dashboard, Execution Info & System Details, you can integrate "ExtentReports" within TestNG to get some superb graphical representations of your test execution.
Dashboard:
Execution Info:
System Details:
Let me know if this answers your question.
Too long by 3 characters
to be a comment.
TestNG documentation is your friend. You could provide your own implementation. Very basic example here.
Another approach is to use HTML/XML Report generation and to inspect the data from a test run there. It's a bunch of html pages with pretty colors and some data. Sample report here. Also if your project is using Apache Maven than just enable the surefire plug-in. Sample report here.
I have created a automation suite in java selenium using testng frame work.Basically in my suite the scenarios are placed in between before and after method.
I want to take the screen shot for each failed and passed test scenario.For that i have created a separate class and calling it in each script to capture the screenshot before the after Method.
The issue that i am facing here is if the scenario is getting failed the script stopped executing and it is not moving to the take screenshot line of code and so it is not capturing the screenshot for the failed ones.
So i want the take screenshot program to be only placed in after method so before the driver quit it will take the screenshot inspite of the scenario result of pass/fail.
I have written code for nearly 20 scenario using testng.
Can some one tel me the code in the after method only to take the screenshot without so much effecting the code which i have written.
The screenshot program should only be in the after method so it will capture teh screen before driver quit.
Make sure you catch exceptions so you won't exit the code prematurely.
For example, this is how I take screenshots after every test run (passed or failed). I use the exception value (if any) to dynamically name the screenshot files.
First, an example test method:
[DataSource("System.Data.SqlClient", "Data Source=DEV338\\X2B;Initial Catalog=SeleniumDb;Integrated Security=True",
"dbo.F_single_ErkenningAannemerKlasse1", DataAccessMethod.Sequential), TestMethod]
public void erkenningAannemerKlasse1()
{
try
{
frontUser.erkenningAannemerKlasse1(data);
}
catch (Exception e)
{
exception = e.Message + " " + e.StackTrace;
}
}
Then this method executes after every test. Default behaviour in Microsoft UnitTest framework, in this case.
[TestCleanup()]
public void Cleanup()
{
frontUser.takeScreenshot(data.get("testnaam"), data.get("datarow"), exception);
frontUser.closeBrowser();
}