I am using TestNG and Maven with surefire plugin to run tests.
I have:
#Test(groups={"groupA"})
TestA{}
#Test
TestB
I would like to have possibility to run:
mvn test
should invoke all test without any group
mvn test -Dgroups=groupA
should invoke only groupA tests ( this works by default but just added here to have it working with previous option )
I tried to configure surefire like:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.16</version>
<configuration>
<excludedGroups>groupA</excludedGroups>
</configuration>
</plugin>
mvn test works as expected, but after
mvn test -Dgroups=groupA
no tests are executed
Edit
I found solution here:
https://labs.consol.de/blog/maven/citrus-and-testng-groups/
<!-- TestNG groups -->
<testGroups></testGroups>
<testGroupsExcluded>groupA</testGroupsExcluded>
<!-- /TestNG groups-->
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.16</version>
<configuration>
<groups>${testGroups}</groups>
<excludedGroups>${testGroupsExcluded}</excludedGroups>
</configuration>
</plugin>
...
<profiles>
<profile>
<id>a-testes</id>
<properties>
<testGroups>a</testGroups>
<testGroupsExcluded></testGroupsExcluded>
</properties>
</profile>
</profiles>
But there is one problem with this solution. It works fine when we want run just always one groups of tests, for example mvn test -P a-tests, but when we will add another group, let's say b-tests, then after mvn test -P a-tests, b-tests only one group will be executed because last defined profile will overide properties... Any ideas how to combine testGroupsExcluded, testGroups properties from multiple profiles?
Edit 2
I just ended with solution
<properties>
<testGroups>unit</testGroups>
</properties>
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.16</version>
<configuration>
<groups>${testGroups}</groups>
</configuration>
</plugin>
But I had to assign all my tests explicitly to groups (all unassigned tests now are 'unit'), but now I can call:
mvn test To invoke all tests marked as unit
mvn test -DtestGroups=groupA, groupB To invoke any group tests...
Dude, have you checked http://testng.org/doc/documentation-main.html#beanshell ?
In surefire plugin attach testng suit config:
<configuration>
<suiteXmlFiles>
<suiteXmlFile>testng.xml</suiteXmlFile>
</suiteXmlFiles>
...
in testng.xml
<suite name="Emap test suite">
<test name="Emap tests">
<method-selectors>
<method-selector>
<script language="beanshell"><![CDATA[
addClassPath("target/test-classes" );
return com.yourpackage.shouldExecuteTest(groups, System.getProperty("groups"));
]]>
In static Java method shouldExecuteTest you can implement any rule you want!
Accoriding to doc, you can use those vars:
java.lang.reflect.Method method: the current test method.
org.testng.ITestNGMethod testngMethod: the description of the current test method
java.util.Map<String, String> groups: a map of the groups the current test method belongs to.
System.getProperty("groups") is just -Dgroups=xxx passed from mvn invocation.
Works like a charm!
Related
I have been searching for this everywhere and whilst there are documentations and answers all over the place, I can't seem to be able to make this work
I am currently working on a multi-modules project with thousands of tests and because these take hours to run, I thought it will be a good idea to run them in parallel(just the tests not the modules)
To achieve this, I found out that jUnit 4.7 or later with maven-surefire-plugin can be used to run tests in parallel
Here's my configuration
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<parallel>classes</parallel>
<threadCount>10</threadCount>
<argLine>-Xverify:none
-enableassertions -Djava.util.Arrays.useLegacyMergeSort=true #{argLine}</argLine>
<useSystemClassLoader>false</useSystemClassLoader>
<includes>
<include>**/*Test.java</include>
<include>**/Test*.java</include>
</includes>
<excludes>
...
...
</plugin>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M3</version>
<configuration>
<includes>
<include>**/*Test.java</include>
<include>**/Test*.java</include>
</includes>
<!-- Required until JDK >=1.7.0u72, see https://code.google.com/p/powermock/issues/detail?id=504 -->
<runOrder>alphabetical</runOrder>
<argLine>#{argLine}</argLine>
</configuration>
</plugin>
Running on Apache Maven 3.3.9
There are all kind of tests, (RunWith, PowerMock,BeforeClass,Before etc)
Here's what I'm doing to test if my configuration is working or not
First I build all the modules
mvn clean install -Pbuild-all -DskipTests
Next I run the specific module tests using the following command line
mvn test -DfailIfNoTests=false -pl module1
The above doesn't seem to be running in parallel as I get stuck in some tests and it doesn't go any further
Furthermore, I tried to run multiple jUnit test classes rather than everything in the module but I'm not sure whether they are running in parallel, I used the following command
mvn test -DfailIfNoTests=false -Dtest=testClass1,testClass2,testClass3 -pl module1
The above finishes in 20 seconds but I'm not sure whether it has run in parallel
Is there anything I'm missing?
My goal: using parameters to switch environments in my tests for instance:
mvn test google -> Tests goes to google site
mvn test bing -> bing site
"I need to feed my tests which environment is the target and it should comes from the pom.xml and using them as arguments."
It would be very useful for teamcity/jenkins integration as well.
Apart from that, I need to use url as variables in my tests. How can I do that?
Profiles can be a solution in pom.xml?
<profiles>
<profile>
<id>google</id>
<properties>
<base.url>http://www.google.com</base.url>
</properties>
</profile>
<profile>
<id>bing</id>
<properties>
<base.url>http://www.bing.com</base.url>
</properties>
</profile>
</profiles>
From build section:
<configuration>
<systemProperties>
<base.url>${base.url}</base.url>
</systemProperties>
</configuration>
But How can I use system properties and overall the approach is good? Thanks!
You can configure the maven-surefire-plugin to include only specific test classes and the run mvn test. By default, mvn will run all these:
"**/Test*.java" - includes all of its subdirectories and all Java filenames that start with "Test".
"**/*Test.java" - includes all of its subdirectories and all Java filenames that end with "Test".
"**/*Tests.java" - includes all of its subdirectories and all Java filenames that end with "Tests".
"**/*TestCase.java" - includes all of its subdirectories and all Java filenames that end with "TestCase".
but you could specify the tests you want to include like this:
<project>
[...]
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.20.1</version>
<configuration>
<includes>
<include>Sample.java</include>
</includes>
</configuration>
</plugin>
</plugins>
</build>
[...]
</project>
or exclude:
<project>
[...]
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.20.1</version>
<configuration>
<excludes>
<exclude>**/TestCircle.java</exclude>
<exclude>**/TestSquare.java</exclude>
</excludes>
</configuration>
</plugin>
</plugins>
</build>
[...]
</project>
Having said that, this is probably not the best design and generally, you should be using some testing framework which you then can configure according to your needs. Few examples are (or combination of): jUnit, TestNG, Cucumber, Spring.
In Cucumber, for example, you can have tags which then you can configure as a part of your test execution. If you use Jenkins, you might have something like this in your build fild:
clean install -Dcucumber.options="--tags #Google
or
clean install -Dcucumber.options="--tags #Bing
In Spring, you can have profiles that you can run like this as Jenkins job:
mvn clean test -Dspring.profiles.active="google"
EDIT
Alternatively, you can define a custom property in your pom like this:
<properties>
<myProperty>command line argument</myProperty>
</properties>
And then pass it from command line like this:
mvn install "-DmyProperty=google"
EDIT2
Providing a -D prefixed value in command line is a way of setting a system property. You can perform this action from Java code itself like this:
Properties props = System.getProperties();
props.setProperty("myPropety", "google");
or simply:
System.setProperty("myPropety", "google");
I am using JUnit's categories to split my tests into different categories and using maven to compile and run my test (surefire and failsafe).
Question is, how to I choose which category of tests are executed from command line?
something like: mvn clean install -DloadTests.
my failsafe plugin:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>${failsafe.plugin.version}</version>
<configuration>
<!--Exclude load tests by default-->
<excludedGroups>com.test.lib.categories.LoadTestCategory</excludedGroups>
</configuration>
</plugin>
You could do that with profiles and specify the plugin configuration in there.
<profiles>
<profile>
<id>noLoadTests</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>${failsafe.plugin.version}</version>
<configuration>
<!--Exclude load tests by default-->
<excludedGroups>com.test.lib.categories.LoadTestCategory</excludedGroups>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>
And then run maven
mvn test -PnoLoadTests
If you only ever need to exclude/include one specific category you could also define a property in the profile and use that in the . For more info you can look here
Edit: The other provided answer is the better one in this case, but profiles allow for various advanced configurations.
According to the documentation, user property is called groups. Therefore this should work:
mvn clean install -Dgroups=com.test.lib.categories.LoadTestCategory
I run a JUnit-based test suite in Maven, some 200 tests in total. Many produce Log4J DEBUG or ERROR lines, and I see them all mashed into a single long file in my surefire-reports directory.
I want to see all this output divided up by test, so I can discover what test produced what output. How can I do this?
The most straightforward solution would be to a call to the following method to the beginning and end of each test:
static void printBeginning(PrintStream stream) {
String methodName = Thread.currentThread().getStackTrace()[0].getMethodName();
stream.format("---- Beginning Test: %s ----%n", methodName);
}
However, if you want to use surefire, the way to do this would be to write your own custom RunListener:
Basically, assign the RunListener in Maven, as in this documentation
<plugins>
[...]
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<properties>
<property>
<name>listener</name>
<value>com.mycompany.MyResultListener,com.mycompany.MyResultListener2</value>
</property>
</configuration>
</plugin>
[...]
</plugins>
Then the code will be run by Surefire.
You can read more about how to do this here: Using JUnit RunListener with Maven
I want to run single test class from command line using Maven and TestNG
Things that doesn't work:
mvn -Dtest=ClassName test
I have defined groups in pom.xml, and this class isn't in one of those groups. So it got excluded on those grounds.
mvn -Dgroups=skipped-group test
mvn -Dsurefire.groups=skipped-group test
when config is
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.7.1</version>
<configuration>
<groups>functest</groups>
</configuration>
</plugin>
Parameters work fine in there are no groups defined in pom.xml.
Similarly, when surefire is configured with
<configuration>
<includes>
<include>**/*UnitTest.java</include>
</includes>
</configuration>
I can add another test with -Dtest parameter, but cannot add group. In any combination, I can narrow down tests to be executed with groups, but not expand them.
What's wrong with my configuration? Is there a way to run a single test or group outside of those defined in pom.xml?
Tried on Ubuntu 10.04 with Maven 2.2.1, TestNG 5.14.6 and Surefire 2.7.1
I didn't test with TestNG 5.12.1 but I can say that running a single test using the test parameter and tests from groups using the groups parameter works with TestNG 5.14.2 (and surefire 2.6) (groups doesn't work in TestNG 5.14)
Here is the pom.xml I'm using:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.stackoverflow</groupId>
<artifactId>Q4159948</artifactId>
<version>1.0-SNAPSHOT</version>
<name>Q4159948</name>
<url>http://maven.apache.org</url>
<properties>
<maven.compiler.source>1.6</maven.compiler.source>
<maven.compiler.target>1.6</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>5.14.2</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.6</version>
<configuration/>
</plugin>
</plugins>
</build>
</project>
With a simple AppTest as follow:
import org.testng.annotations.*;
public class AppTest {
#BeforeClass
public void setUp() {
// code that will be invoked when this test is instantiated
}
#Test(groups = { "fast" })
public void aFastTest() {
System.out.println("Fast test");
}
#Test(groups = { "slow" })
public void aSlowTest() {
System.out.println("Slow test");
}
}
Both
$ mvn test -Dtest=AppTest
and
$ mvn test -Dgroups=slow
produce the expected result.
As I've explained in question, any mention of groups either in pom.xml or on command line resulted in reduction of executed tests count. Only way I've managed to avoid this is by using mavens profiles like this:
<profiles>
<profile>
<id>test-slow</id>
<build>
<plugins>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<groups>slow</groups>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>
and then running tests with
mvn -P test-slow test
I would suggest to try something like
mvn test -Dincludes=rs/magrathea/TestClassName
although I haven't tested this myself.
In order to run a single test you need the following from official documentation
mvn -Dtest=MyFirstTest test
or
mvn -Dtest=MyFirstTest,MySecondTest test
This is tested (and working) on maven 3.
Then you can avoid using the profiles. I had the same problem as I needed to run load test in isolation and using profiler in parallel to get the real figures.
Note: Not sure why but make sure that the switches come before the phase i.e. "-Dtest=MyFirstTest" before "test" otherwise it is not working (Mac OSX)