This question already has answers here:
How do I control the order of execution of tests in Maven?
(6 answers)
Closed 7 years ago.
I have a maven project and several test classes.
I want to run these tests in a specific order with the plug-in surefire.
For example, I have:
ClassTest1.java
ClassTest2.java
ClassTest3.java
ClassTest4.java
I want to run the Class 1, then 2, then 3 and finally 4.
How can I specify this in the pom.xml?
One workaround is to set the runOrder parameter to alphabetical and then rename your tests to have alphabetical order.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<runOrder>alphabetical</runOrder>
</configuration>
</plugin>
This isn't recommended, though - unit tests should be independent of each other. The execution order shouldn't matter.
You can do this with the runOrder parameter.
From the documentation:
Defines the order the tests will be run in. Supported values are
"alphabetical", "reversealphabetical", "random", "hourly"
(alphabetical on even hours, reverse alphabetical on odd hours),
"failedfirst", "balanced" and "filesystem".
See the full description here.
Here is an example
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.17</version>
<configuration>
<runOrder>alphabetical</runOrder>
</configuration>
</plugin>
I would suggest you to define the order in testng.xml and then create profile via maven and run it. Here is a code sample you can try out:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Selenium Automation" parallel="false">
<test name="Test1">
<classes>
<class name="some.package.Class1"/>
<class name="some.package.Class2"/>
<class name="some.package.Class3"/>
</classes>
</test>
</suite>
And then in POM.xml you can create a profile as below and refer to the testNG.xml which you want to execute.
<profile>
<id>any id</id>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.12.4</version>
<configuration>
<suiteXmlFiles>
<suiteXmlFile>src/test/resources/testng.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
</plugin/>
</profile>
You could use a JUnitCore to execute the tests in a specific order. Here is a basic implementation.
public static void main(String[] args)
{
List<Class> testCases = new ArrayList<Class>();
//Add test cases
testCases.add(Class1.class);
testCases.add(Class2.class);
for (Class testCase : testCases)
{
runTestCase(testCase);
}
}
private static void runTestCase(Class testCase)
{
Result result = JUnitCore.runClasses(testCase);
for (Failure failure : result.getFailures())
{
System.out.println(failure.toString());
}
}
Is there a reason you need them executed in a specific order? The tests should ideally not have dependencies on each other.
You can create a testsuite class with junit and then use surefire to run this class instead, it will run with the order you give, for example:
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
import org.junit.runners.Suite.SuiteClasses;
#RunWith(Suite.class)
#SuiteClasses({
Test1.class
Test2.class
Test3.class
Test4.class
})
public class TestSuite {
}
Related
Yes, the Java classes which are containing the tests are named correctly. (they are ending with Tests)
Tried to add the following configuration in pom.xml:
<configuration>
<includes>
<include>**/*Test.java</include>
</includes>
</configuration>
Tests are located under the following structure: /src/test/packagename/JavaClassTest.java where packagename is the same package what is under the unit test written under src/main/java path.
I'm using jupiter Junit 5 and maven-surefire-plugin with 2.22.2
And I still get the following error on mvn test:
--- maven-surefire-plugin:2.22.2:test (default-test) # <project-name> ---
[INFO] No tests to run.
What do I do wrong?
Is there any reason why you configure just a certain tests? By default the surefire plugin should access all classes in testRoot and sub directories.
You could also just link a specific file like
src/test/ArchTest.java
to see if it is the "include" in your configuration or something else. I am not sure that the wildecards are working as you expect them to work. See Maven <include> wildcard match on partial folder name .
Based on this you might try out
<configuration>
<includes>
<include>/**/*Test.java</include>
</includes>
</configuration>
Everything with the #Test annotation throws null pointer exception, regardless of the method called. I tried calling driver.findElement() and executing js script with JavascriptExecutor. Both methods ended up with a null pointer.
If I run through the IDE everything works like a charm. Otherwise, only the Before and After annotations are executed adequately
Here's my pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>test</artifactId>
<version>1.0-SNAPSHOT</version>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.9</version>
<configuration>
<parallel>methods</parallel>
<threadCount>1</threadCount>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<!-- https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-java -->
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>4.1.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.testng/testng -->
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>7.5</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/ch.qos.logback/logback-classic -->
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.10</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache-extras.beanshell</groupId>
<artifactId>bsh</artifactId>
<version>2.0b6</version>
</dependency>
</dependencies>
<properties>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
</properties>
</project>
and testng.xml:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Practice Suite">
<test name="Test Basics 1">
<method-selectors>
<method-selector>
<script language="beanshell">
<![CDATA[
String testGroup = System.getProperty("env");
groups.containsKey(testGroup);
]]>
</script>
</method-selector>
</method-selectors>
<classes>
<class name="tests.SignUpTest"/>
</classes>
</test>
</suite>
and finally the two manners I am using to run the test:
mvn -Dwebdriver=chrome -Denv=qa install
mvn -Dwebdriver=chrome -Denv=qa test
EDIT:
BeforeTest and Test:
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.edge.EdgeDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.safari.SafariDriver;
import org.testng.annotations.AfterTest;
import org.testng.annotations.BeforeTest;
public class BaseTest {
protected WebDriver driver;
private WebDriver selectDriver(String driver)
switch (driverString) {
case "chrome":
return new ChromeDriver();
case "firefox":
return new FirefoxDriver();
case "edge":
return new EdgeDriver();
case "safari":
return new SafariDriver();
default:
return new ChromeDriver();
}
}
#BeforeTest(alwaysRun = true)
public void testInit() {
driver = selectDriver(System.getProperty("webdriver"));
}
}
import additions.BaseTest;
import org.testng.Assert;
import org.testng.annotations.Test;
import pages.DashBoardPage;
import pages.LoginPage;
public class LoginTest extends BaseTest {
#Test
public void successfulLogin(){
LoginPage loginPage = new LoginPage(driver);
loginPage
.populateUserName("somename")
.populatePassword("somepassowrd")
.clickLoginButton();
}
}
Since still there is some uncertainty in the question, I'll try to share my thoughts around this.
If I run through the IDE everything works like a charm.
There are several ways how to run this in IDE:
Run the test class.
In this case IDE generates own testng.xml which not match the example (from the question), so it will not contain any method-selector items.
Run the testng.xml
In testng.xml there is method-selector which expects System.getProperty("env") provided.
If you run testng.xml from IDE, you have to provide somehow
System.getProperty("env") via IDE settings, otherwise you'll see org.testng.TestNGException: javax.script.ScriptException error.
If I look at testng.xml, I see method-selector, which filter tests by System.getProperty("env").
And If I look at the test:
#Test
public void successfulLogin(){
LoginPage loginPage = new LoginPage(driver);
I see no any group defined for it. So this test cannot be executed with maven command. And it's unclear how it might throw any exception.
If I look at maven command
mvn -Dwebdriver=chrome -Denv=qa test
I cannot see the arg defining which testng xml suite to execute.
I expect the command like:
mvn test -Dwebdriver=chrome -Denv=qa -Dsurefire.suiteXmlFiles="src/test/resources/tests.xml"
Expected behavior
When we execute the maven command:
mvn test -Dwebdriver=chrome -Denv=qa -Dsurefire.suiteXmlFiles="src/test/resources/tests.xml"
method-selector in testng.xml looks for methods matching the qa group.
and there are no any methods, matched this group.
But there are some configuration methods with alwaysRun = true, and they will be executed anyway.
So
#BeforeTest(alwaysRun = true)
public void testInit() {
is invoked without any tests,
and the output will be:
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.345 s - in TestSuite
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11.867 s
And the test will be executed if we'll add the group:
#Test(groups= {"qa"})
public void successfulLogin(){
I've solved it. It turns out that initialising the driver with the #BeforeTest annotation is not a good idea, especially if your teardown method is marked with #AfterTest. The driver is initialised and used correctly in the first #Test method, but on the second one, the teardown method is already called and the driver becomes null. So I used #BeforeClass and #AfterClass (I might reconsider and use #Before/AfterSuite). Then I added the following configuration to the pom.xml
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.2</version>
<configuration>
<suiteXmlFiles>
<suiteXmlFile>testng.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
</plugin>
</plugins>
</build>
Then I added the package with the tests I wanted to execute, in the testng.xml file, like this:
<packages>
<package name="path.to.tests.*"></package>
</packages>
Finally I called the mvn command using the following syntax:
mvn test -Dsurefire.suiteXmlFiles=testng.xml -Dwebdriver="chrome" -Denv="qa"
This way you don't have to call the path to the testng.xml file, but rather call it out of the pom.xml. As a side note to the last sentence, my testng.xml file is located in the project directory, along with the src folder. If you put it in another folder, I guess it will be good to set the path to this folder in the pom.xml
I hope this is understandable and it will be helpful for anyone who gets stuck with such issue.
Thank again to the others for the clues.
I'm using Cucumber and JUnit together in my selenium automation project. Since i have flaky test cases, i have added retry runs to the project. By adding the class below:
import io.cucumber.junit.Cucumber;
import io.cucumber.junit.CucumberOptions;
#RunWith(Cucumber.class)
#CucumberOptions(features = { "#target/rerun.txt" },
glue = { "stepDefinitions" },
monochrome = true, plugin = { "pretty", "html:target/cucumber-html-reports",
"json:target/cucumber-reports/cucumber.json", "junit:target/cucumber-reports/cucumber.xml",
"json:target/cucumber.json",
"com.aventstack.extentreports.cucumber.adapter.ExtentCucumberAdapter:"})
public class TestRunnerRerunCucumber {
}
I also need to fail the maven build, if any of the test cases are failed except the ones that passed on rerun. But when a test case fails on the first run and passes on the second run, maven build still fails. I have used maven surefire plugin to fail the build:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M3</version>
<configuration>
<includes>
<include>**/*Cucumber.java</include>
</includes>
</configuration>
</plugin>
Is there any configurations or plugins to success the maven build if failed test cases pass on rerun?
I found there is an option to set the parameters to testng xml through surefire plugin, by then the parameter can be sent from command line.
<plugins>
[...]
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M3</version>
<configuration>
<systemPropertyVariables>
<browser>firefox</browser>
</systemPropertyVariables>
</configuration>
</plugin>
[...]
</plugins>
Ref:
https://maven.apache.org/surefire/maven-surefire-plugin/examples/testng.html
https://maven.apache.org/surefire/maven-surefire-plugin/examples/system-properties.html
There is a requirement to print the parameters in testng custom emailable report. Able to print the testng parameters specified in testng xml using the following code. But, not able to print the parameters specified in surefire plugin.
Note: System.getProperty("browser") works here. But, I want to print them without having to specifying the parameter name (say "browser") as below. But below one doesn't work.
Map<String, String> allParameters = context.getCurrentXmlTest().getAllParameters();
for(String parameter: allParameters.keySet()) {
System.out.println(parameter + " : " + allParameters.get(parameter));
}
Example:
import java.util.Map;
import org.testng.ITestContext;
import org.testng.ITestResult;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.BeforeTest;
import org.testng.annotations.Parameters;
import org.testng.annotations.Test;
public class TestNGTest {
ITestContext context;
#BeforeTest
public void beforeTest(ITestContext context) {
this.context = context;
}
#Parameters({"browser"})
#Test
public void method(String browser) {
System.out.println(browser);
Map<String, String> allParameters = context.getCurrentXmlTest().getAllParameters();
for(String parameter: allParameters.keySet()) {
System.out.println(parameter + " : " + allParameters.get(parameter));
}
}
}
Actual Output:
[RemoteTestNG] detected TestNG version 7.0.0
chrome
key : value
===============================================
Suite
Total tests run: 1, Passes: 1, Failures: 0, Skips: 0
===============================================
Expected Output:
[RemoteTestNG] detected TestNG version 7.0.0
chrome
key : value
browser : chrome
===============================================
Suite
Total tests run: 1, Passes: 1, Failures: 0, Skips: 0
===============================================
Testng.xml:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite" parallel="classes" thread-count="4">
<test name="Front-End" group-by-instances="true">
<parameter name="key" value="value"></parameter>
<classes>
<class name="com.ftd.automation.framework.tests.TestNGTest" />
</classes>
</test>
</suite>
Please help me on how to print the testng parameters specified in surefire plugin or passed in command line.
Assuming you are running with command line arguments like,
mvn test -Dbrowser=firefox
then get the parameter by,
import org.testng.annotations.Parameters;
#Parameters("browser")
#Test
public void myTestMethod(String browser){
System.out.println(browser);
}
//or as Test parameter
#Test(parameters = {"browser"})
public void myTestMethod(String browser){
System.out.println(browser);
}
//or System.getProperty() way
#Test
public void myTestMethod(){
System.out.println(System.getProperty("browser"));
}
The above works well. Additionally, if you need to use testng.xml, you can specify the suiteXmlFile like,
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M3</version>
<configuration>
<systemPropertyVariables>
<browser>firefox</browser>
</systemPropertyVariables>
<suiteXmlFiles>
<suiteXmlFile>testng.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
</plugin>
EDIT:
FYI, System.grtProperties() lists all properties, and those set from command line will be there, but there's no way to distinguish those from the other properties added by the system
I have the following plugin configuration in my maven pom.xml
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<version>1.7</version>
<executions>
<execution>
<id>prepare-database</id>
<phase>pre-integration-test</phase>
<configuration>
<target name="run">
<java fork="false" failonerror="yes" classname="Test">
<arg line="arg1 arg2"/>
</java>
</target>
</configuration>
<goals>
<goal>run</goal>
</goals>
</execution>
</executions>
</plugin>
The class is located in a package under src/main/java the package is called com.a.b and the class file is called Test.java
package com.a.b;
public class Test {
public static void main(String[] args)
{
String arg1 = args[0];
String arg2 = args[1];
doStuff(arg1, arg2);
}
public static void doStuff(String arg1, String arg2)
{
System.out.println(arg1);
}
}
When trying to execute the maven build that runs the ant task I get the following error:
An Ant BuildException has occured: Could not find Test. Make sure you
have it in your classpath
I understand that the file can not be found but I have no clue what to do about it. Any help would be highly appreciated.
Take a look at the standard-directory-layout article on the Maven site.
Source folder should be src/main/java (and not src/java/main)
Also, what exactly do you want to do with this ant target ? Do you simply need the fork ? Why can't you run it as a pure unit test, or a maven integration test and forget about the ANT target ?
H2 is well-suited for running in-memory. It only takes a couple of lines to connect to the DB and start testing, all inside the JVM, all using standard jUnit code.
Class.forName("org.h2.Driver");
Connection con = DriverManager.getConnection("jdbc:h2:mem:mytest", "sa", "");
Statement sst = con.createStatement();
sst.executeUpdate(SQL);