We have a project with lots of tests running against a database. Many of the tests set up the database using #BeforeClass, and clean it out using #AfterClass, which works fine when we run the tests locally. When Jenkins runs them, it mixes up the order so tests from different classes are intermixed. For instance, A.TestAlpha, B.TestFoo, A.TestGamma. Then inevitably the tests fail because they don't have the correct setup.
My understanding of Jenkins is that it's supposed to run one class of tests at a time. Is there some config somewhere that might tell it to run the tests in this weird way?
(edit)
The same sort of problems occur running tests locally with mvn.
From mvn help:effective-pom
(project)
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.17</version>
<configuration>
<includes>
<include>**/When*.java</include>
<include>**/Test*.java</include>
<include>**/*Test.java</include>
<include>**/*TestCase.java</include>
</includes>
</configuration>
</plugin>
(module)
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.17</version>
<executions>
<execution>
<id>default-test</id>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<includes>
<include>**/When*.java</include>
<include>**/Test*.java</include>
<include>**/*Test.java</include>
<include>**/*TestCase.java</include>
</includes>
</configuration>
</execution>
</executions>
<configuration>
<includes>
<include>**/When*.java</include>
<include>**/Test*.java</include>
<include>**/*Test.java</include>
<include>**/*TestCase.java</include>
</includes>
</configuration>
</plugin>
Thanks to everyone who attempted to help.
The bug apparently lies in TestNG.
How to make sure TestNG runs methods on test classes in succession instead of interleaved?
Related
I have some unit tests under Integration.java. I want Maven to ignore the test results of this class as they fail rarely (due to external server maintenance). But I don't want to ignore them from running as I need them for code coverage.
I've tried this configuration
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<includes>
<include>IntegrationTest.java</include>
</includes>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
</plugin>
This seems to be working by ignoring the failed tests. But the problem with the above configuration is, it is ignoring the failed tests at other classes as well.
I want all the tests under Integration.java needs to be executed but It shouldn't have any impact on the build to determine success/failure.
But, the build should fail if any test cases fail under any other Test classes other than Integration.java
Like #khmarbaise has mentioned, It needs to be handled by maven-failsafe not surefire plugin
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<includes>
<include>*IntegrationTest.java</include>
</includes>
</configuration>
<executions>
<execution>
<id>failsafe-integration-tests</id>
<phase>integration-test</phase>
<goals>
<goal>integration-test</goal>
</goals>
</execution>
</executions>
</plugin>
By using maven-failsafe-plugin I made sure the test cases run and for the good code coverage and the build will not fail even if the upstream is down.
Surefire by default includes all test classes whose name starts with Test/ends with Test/Tests/TestCase.
You can use excludes and includes parameters as per your need :
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.21.0</version>
<configuration>
<excludes>
<exclude>DataTest.java</exclude>
</excludes>
<includes>
<include>DataCheck.java</include>
</includes>
</configuration>
</plugin>
I have 2 sets of tests defined in Maven - integration-test and test.
If I run maven test - my tests run
If I run maven integration-test - both run
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<id>default-test</id>
<configuration>
<forkMode>always</forkMode>
<excludes>
<exclude>**/TC_Integration*</exclude>
</excludes>
<includes>
<include>**/TC_*</include>
</includes>
</configuration>
</execution>
<execution>
<id>integration-test</id>
<phase>integration-test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<excludes>
<exclude>**/TC_Unit*</exclude>
</excludes>
<includes>
<include>**/TC_*</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
What's the best way to make both run all the time? I mainly want maven install to run both, but it's not.
Instead of trying to configure surefire to run both unit and integration tests, configure surefire to run just the unit tests and use the failsafe plugin to run the integration tests.
https://maven.apache.org/surefire/maven-failsafe-plugin/
I have a set of integration tests which I need to run in specific order. So i created a BlahSuite.java inside the same package, and specified the order of classes there. And annotation as following
#RunWith(Suite.class)
#Suite.SuiteClasses({
And I added the plugin into pom as following
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.9</version>
<configuration>
<includes>
<include>**/*Suite.java</include>
</includes>
</configuration>
<executions>
<execution>
<id>integration-test</id>
<goals>
<goal>integration-test</goal>
</goals>
</execution>
<execution>
<id>verify</id>
<goals>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
But still the tests are fired in different orders, feels like the Suite class is fully ignored. Any idea how to fix this ?
I found the answer at Stackoverflow question Run Junit Suite using Maven Command
So what my final setup is I just removed failsafe plugin and added following,
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<includes>
<include>**/*Suite.class</include>
</includes>
</configuration>
</plugin>
I have the following configuration for using maven-surefire-plugin to execute my integration tests and unit tests..
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.16</version>
<executions>
<execution>
<id>unit-tests</id>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<excludes>
<exclude>**/*Test.java</exclude>
<exclude>**/*TestCase.java</exclude>
</excludes>
<includes>
<include>**/MySuite.java</include>
</includes>
</configuration>
</execution>
<execution>
<id>integration-tests</id>
<phase>integration-test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<skip>true</skip>
<includes>
<include>**/BarSuite.java</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
However, when executing the unit tests it appears to run my individual test classes as well as executing the suite which runs those same classes. How can I configure it to only execute that of which i include? i.e the suite? (MySuite)
thanks,
I've noticed that there appears to be an execution with id default-test that occurs even if I don't configure it.
By explicitly configuring an execution with this id, I can control it. In your case telling it to exclude ** might solve your problem.
Without a build log I can't tell for sure, but from your description here is what I suspect. The surefire plugin is bound to the lifecycle by default, for jars, wars, ears. What you have done with your configuration is to add two additional plugin executions, however you did not change the default execution. You should be able to see this by adding -X to the mvn command and counting the number of surefire plugin executions.
To override the default, change the id of the first execution to "default-test" and see if that does the trick.
Maven documentation for overriding the default executions
I have two kinds of Unit tests (not integration test). Because of some strange behaviour with Spring Security I need to run first all normal tests, and later on the security tests.
I am using Junit (so I can not use any TestNG groups).
So what I have done is specify two sets of includes and excludes rules.
<excludes>
<exclude>**/*SecurityTest.java</exclude>
</excludes>
<includes>
<include>**/*Test.java</include>
<include>**/*Tests.java</include>
</includes>
and
<excludes>
</excludes>
<includes>
<include>**/*SecurityTest.java</include>
</includes>
That works if I replace them in my pom by hand so I can have normal or security tests. But of course I want that both kind of tests run in each build.
My first try was to have two complete maven-surefire-plugin configruation. But then maven take only the last of them in account.
My next try was to use two execution definitions, but then surefire seems to ignore the rules at all and run both kind of tests mixed.
So my general question is how to specify two file sets for maven surefire so that they will be executed each after another? And more specific how to specify two different file sets.
the configuration with executions
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.9</version>
<configuration>
<junitArtifactName>junit:junit</junitArtifactName>
<encoding>UTF-8</encoding>
<inputEncoding>UTF-8</inputEncoding>
<outputEncoding>UTF-8</outputEncoding>
<argLine>-Xms256m -Xmx512m -XX:MaxPermSize=128m -ea</argLine>
</configuration>
<executions>
<execution>
<id>normal-tests</id>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<excludes>
<exclude>**/Abstract*.java</exclude>
<exclude>**/*_Roo_*</exclude>
<exclude>**/*SecurityTest.java</exclude>
</excludes>
<includes>
<include>**/*Test.java</include>
<include>**/*Tests.java</include>
</includes>
</configuration>
</execution>
<execution>
<id>security-tests</id>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<excludes>
<exclude>**/Abstract*.java</exclude>
<exclude>**/*_Roo_*</exclude>
</excludes>
<includes>
<include>**/*SecurityTest.java</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
You haven't changed the default surefire execution which is bound to the test phase in the superpom, so it's still running with its default config. The id of that execution is "default-test". You'll either need to override it to unbind it from the test phase or else just use that id for one of your own executions.