Is possible to run same test multiple times with Maven Surefire Plugin? - java

There is a config value rerunFailingTestsCount but I want to run a test method a configurable number of times even it is successful. Are there any options?


I don't think it is possible to configure maven-surefire-plugin to rerun passing tests.
However, you can configure the invocation count of a single test using the TestNG (not JUnit) #Test annotation:
#Test(invocationCount = 5)
public void testSomething() {
}
This will result in the testSomething method being tested 5 times.
If you don't want to go the TestNG route, you can refer to this answer for a solution with JUnit.


If you want to have it configurable by implementing the IInvokedMethodListener beforeInvocation method, something to the effect :
method.getTestMethod().setInvocationCount(Integer.parseInt(System.getProperty("configurablecount")));
System.getProperty can be replaced by however you want to configure it. You can also probably control which tests to set the invocation count to change by passing testnames.


Yes
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
<executions>
<execution>
<id>phase-1</id>
<phase>test</phase>
<configuration>
<skip>false</skip>
<!-- Phase 1 configuration -->
</configuration>
<goals>
<goal>test</goal>
</goals>
</execution>
<execution>
<id>phase-2</id>
<phase>test</phase>
<configuration>
<skip>false</skip>
<!-- Phase 2 configuration -->
</configuration>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

Related

SystemProperties from maven failsafe plugin with Spock Tests

I am trying to get maven failsafe to execute my integration tests, however I am having a problem with the server port being correctly assigned to the system properties.
My failsafe plugin configuration:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.21.0</version>
<executions>
<execution>
<id>integration-tests</id>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
<configuration>
<useSystemClassLoader>false</useSystemClassLoader>
<systemPropertyVariables>
<test.server.port>${tomcat.http.port}</test.server.port>
</systemPropertyVariables>
<skipTests>${skip.integration.tests}</skipTests>
</configuration>
</execution>
</executions>
</plugin>
My spock test configuration:
def "verifies port is being set correctly"() {
expect:
System.getProperties().getProperty('test.server.port') != null
}
When I run the test, test.server.port does not appear in my system properties array. However, if I change it to asdf or anything NOT test.server.port, it then appears in my system properties.

The fix was to use Spring Boot's random port for environment initialization
#SpringBootTest(
classes = OrderQueueApplication.class,
webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT
)

Why Suite classes are not run but the other Tests

I have a set of integration tests which I need to run in specific order. So i created a BlahSuite.java inside the same package, and specified the order of classes there. And annotation as following
#RunWith(Suite.class)
#Suite.SuiteClasses({
And I added the plugin into pom as following
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.9</version>
<configuration>
<includes>
<include>**/*Suite.java</include>
</includes>
</configuration>
<executions>
<execution>
<id>integration-test</id>
<goals>
<goal>integration-test</goal>
</goals>
</execution>
<execution>
<id>verify</id>
<goals>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
But still the tests are fired in different orders, feels like the Suite class is fully ignored. Any idea how to fix this ?

I found the answer at Stackoverflow question Run Junit Suite using Maven Command
So what my final setup is I just removed failsafe plugin and added following,
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<includes>
<include>**/*Suite.class</include>
</includes>
</configuration>
</plugin>

Jenkins and Jacoco Integration Test Coverage

I just added Jacoco on my maven dependencies to run integration tests. Then, I created an integration test to test my controller. For example, I tested my HTTP response codes, the headers and the response resources. After that, I created a profile on maven that starts an embedded tomcat. So, everytime I want to run my integration tests, I just put the profile on the maven goals. However, when I execute the build on Jenkins and Sonar reads the reports from Jacoco, the reports says that I have not tested my controller. The question is: How I tell Jacoco that I have passed through my Controllers, Services and Repositories?
Thanks to all!

Are you getting any Integration Coverage, or just 0%?
It can be quite tricky to set up Integration Test Coverage using maven and Sonar.
Check there is a jacoco file produced when the IT tests are run.
Check your POM set up compared to this...
<properties>
<!-- Jacoco Properties -->
<jacoco.version>0.7.4.201502262128</jacoco.version>
<sonar.java.coveragePlugin>jacoco</sonar.java.coveragePlugin>
<sonar.core.codeCoveragePlugin>jacoco</sonar.core.codeCoveragePlugin>
<sonar.jacoco.itReportPath>${project.basedir}/target/jacoco-it.exec</sonar.jacoco.itReportPath>
<sonar.language>java</sonar.language>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>${jacoco.version}</version>
<executions>
<execution>
<id>prepare-unit-test-agent</id>
<configuration>
</configuration>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<execution>
<id>jacoco-site</id>
<phase>verify</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
<execution>
<id>prepare-it-test-agent</id>
<configuration>
<propertyName>jacoco.agent.argLine</propertyName>
<destFile>${sonar.jacoco.itReportPath}</destFile>
<append>true</append>
</configuration>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.12.4</version>
<configuration>
<argLine>${jacoco.agent.argLine}</argLine>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
Link to GitHub example
BeyondCoding.net

How to do integration tests which do not ends with **/**IT

I have test class which ends with IT i.e SampleClassIT.java.
I have added this class in testng.xml file and used maven-failsafe plugin in pom.xml file.
Plugin used :-
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.17</version>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
Is there any way ,so that all the classes which are not extended with IT can also run as a part of integration tests because i have more that 100 classes that i need to refactor them.I dont need to extend my classes with **/*IT.java and make them run as integration tests using failsafe plugin.

You can explicitly specify what classes to include by
<configuration>
<includes>
<include>**some_pattern*.java</include>
</includes>
</configuration>
make your class at-least to satisfy one pattern

Unable to get Jacoco to work with Powermockito using offline instrumentation

Given that Jacoco doesn't play nicely with PowerMockito when instrumenting "on the fly", I've been trying to configure offline instrumentation in the hope this will give me proper unit test coverage for classes that use PowerMockito.
I've setup my pom as below but I still get zero % coverage on my test class. Any help much appreciated as it's driving me slowly bonkers!
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>mandy</groupId>
<artifactId>jacoco-test</artifactId>
<packaging>war</packaging>
<version>1.0-SNAPSHOT</version>
<name>jacoco-test Maven Webapp</name>
<url>http://maven.apache.org</url>
<properties>
<powermock.version>1.5.4</powermock.version>
<jacoco.version>0.7.1.201405082137</jacoco.version>
</properties>
<dependencies>
<dependency>
<groupId>org.jacoco</groupId>
<artifactId>org.jacoco.agent</artifactId>
<classifier>runtime</classifier>
<version>${jacoco.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.powermock</groupId>
<artifactId>powermock-module-junit4</artifactId>
<version>${powermock.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.powermock</groupId>
<artifactId>powermock-api-mockito</artifactId>
<version>${powermock.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.10</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<source>1.6</source>
<target>1.6</target>
</configuration>
</plugin>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>${jacoco.version}</version>
<executions>
<execution>
<id>instrument</id>
<phase>process-classes</phase>
<goals>
<goal>instrument</goal>
</goals>
</execution>
<execution>
<id>restore-report</id>
<phase>prepare-package</phase>
<goals>
<goal>restore-instrumented-classes</goal>
<goal>report</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.17</version>
<configuration>
<!--<argLine>${argLine}</argLine>-->
<systemPropertyVariables>
<!-- JaCoCo runtime must know where to dump coverage: -->
<jacoco-agent.destfile>target/jacoco.exec</jacoco-agent.destfile>
</systemPropertyVariables>
</configuration>
</plugin>
</plugins>
<finalName>jacoco-test</finalName>
</build>
</project>
here is my class under test:
public class Utils {
private Utils() {
}
public static String say(String s) {
return "hello:"+s;
}
}
here is my test:
#RunWith(PowerMockRunner.class)
#PrepareOnlyThisForTest(Utils.class)
#PowerMockIgnore("org.jacoco.agent.rt.*")
public class UtilsTest {
#Test
public void testSay() throws Exception {
PowerMockito.mockStatic(Utils.class);
Mockito.when(Utils.say(Mockito.anyString())).thenReturn("hello:mandy");
assertEquals("hello:mandy", Utils.say("sid"));
}
}
I run mvn clean install which generates the jacoco.exe
Coverage report (generated from jacoco.exec using an ant script ):-

This pom worked for me:
<build>
<finalName>final-name</finalName>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<systemPropertyVariables>
<jacoco-agent.destfile>target/jacoco.exec</jacoco-agent.destfile>
</systemPropertyVariables>
</configuration>
</plugin>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.7.2.201409121644</version>
<executions>
<execution>
<id>default-instrument</id>
<goals>
<goal>instrument</goal>
</goals>
</execution>
<execution>
<id>default-restore-instrumented-classes</id>
<goals>
<goal>restore-instrumented-classes</goal>
</goals>
</execution>
<execution>
<id>default-report</id>
<phase>prepare-package</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
See this link.

I saw the same behavior, though after following the thread on the GitHub issue it seems to be fixed in 1.6.5, which proved true for me.
Hopefully this will save someone a headache later :).
Working configuration with:
jacoco-maven-plugin 0.7.7.201606060606
powermock 1.6.5
I am not using offline instrumentation.

For me this sample of Offline Instrumentation works well.
But it turns in my case that there is a simplier solution: just do not include the tested class in #PrepareForTest({}) annotation before its declaration.

I used Jacoco offline instrumentation and after execution of test restore ed original classes with help of "restore-instrumented-classes" goal. My JaCoCo configuration look like this:
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.7.9</version>
<executions>
<execution>
<id>default-prepare-agent</id>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<execution>
<id>jacoco-instrument</id>
<goals>
<goal>instrument</goal>
</goals>
</execution>
<execution>
<id>jacoco-restore-instrumented-classes</id>
<goals>
<goal>restore-instrumented-classes</goal>
</goals>
</execution>
<execution>
<id>jacoco-report</id>
<phase>package</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
</executions>
<configuration>
<excludes>
<exclude>*</exclude>
</excludes>
</configuration>
</plugin>

I made it work using the javaagent of PowerMock.
see here: https://github.com/powermock/powermock/wiki/PowerMockAgent
Remove the #RunWith annotations, put the PowerMockRule as described in the link above. Make it public.
Put the following line in the maven-surefire-plugin configuration:
-javaagent:${org.powermock:powermock-module-javaagent:jar}
(used the technique described here : Can I use the path to a Maven dependency as a property?)
<?xml version="1.0" encoding="UTF-8"?>
<project>
<modelVersion>4.0.0</modelVersion>
<groupId>com.stackoverflow</groupId>
<artifactId>q2359872</artifactId>
<version>2.0-SNAPSHOT</version>
<name>q2359872</name>
<properties>
<!-- Must be listed in the dependencies section otherwise it will be null. -->
<my.lib>${org.jmockit:jmockit:jar}</my.lib>
</properties>
<dependencies>
<dependency>
<groupId>org.jmockit</groupId>
<artifactId>jmockit</artifactId>
<version>1.11</version>
</dependency>
</dependencies>
<build>
<defaultGoal>generate-sources</defaultGoal>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<goals>
<goal>properties</goal>
</goals>
</execution>
</executions>
</plugin>
<!-- Example usage: -->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.2</version>
<executions>
<execution>
<goals>
<goal>exec</goal>
</goals>
<phase>generate-sources</phase>
</execution>
</executions>
<configuration>
<executable>echo</executable>
<arguments>
<argument>path to jar=</argument>
<argument>${org.jmockit:jmockit:jar}</argument>
<argument>my.lib=</argument>
<argument>${my.lib}</argument>
</arguments>
</configuration>
</plugin>
<!-- end of Example usage -->
</plugins>
</build>
</project>

I ended up using offline instrumentation and Jacoco (similar to what you have done) in conjunction with sonar and I was able to get the coverage numbers from that.

I was also facing the same issue. I was able to generate report partially. I have used both these tags for my test cases #RunWith(PowerMockRunner.class)
#PrepareForTest({}). And the report was not getting generated for the test cases where i used the above mentioned tags. But for one of the test case there was only this tag #RunWith(PowerMockRunner.class). Somehow the report was getting generated for that case. And also i have never used offline instrumentation. When i tried using the offline instrumentation i was getting error saying that the class was already instrumented. I tried various scenarios and followed various links but could not generate the report. Finally as per the above comment i upgraded my version of powermock from 1.5.5 to 1.6.5 and i was able to generate the report. Following is my pom.xml entry
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.7.5.201505241946</version>
<executions>
<execution>
<id>pre-unit-test</id>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<!-- Sets the path to the file which contains the execution data. -->
<destFile>${basedir}/target/jacoco.exec</destFile>
<!--
Sets the name of the property containing the settings
for JaCoCo runtime agent.
-->
</configuration>
</execution>
<execution>
<!--<id>post-unit-test</id>
<phase>test</phase>-->
<id>default-report</id>
<phase>verify</phase>
<goals>
<goal>report</goal>
</goals>
<configuration>
<!-- Sets the path to the file which contains the execution data. -->
<dataFile>${basedir}/target/jacoco.exec</dataFile>
<!-- Sets the output directory for the code coverage report. -->
<outputDirectory>${basedir}/target/jacoco-ut</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
Following is my entry in pom .xml for maven-surefire-plugin
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<argLine>#{argLine}</argLine>
<skipTests>false</skipTests>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
</plugin>
#{argLine} was set as a property
<properties>
<argLine>-noverify -Xms512m -Xmx1024m -XX:PermSize=128m -XX:MaxPermSize=512m</argLine>
</properties>
And upgraded my powermock version from 1.5.5 to 1.6.5 . Finally i could see my report generation for the classes where i used the following tags in my test cases
#RunWith(PowerMockRunner.class) #PrepareForTest({})

Use below maven plugin code snippet this works fine in jenkins as well as in local and shows full code coverage for PowermockRunner unit tests
<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>${jacoco-maven-plugin.version}</version>
<executions>
<execution>
<id>pre-test</id>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<append>true</append>
</configuration>
</execution>
<execution>
<id>pre-integ-test</id>
<goals>
<goal>prepare-agent-integration</goal>
</goals>
<configuration>
<append>true</append>
</configuration>
</execution>
<execution>
<id>jacoco-instrument</id>
<goals>
<goal>instrument</goal>
</goals>
</execution>
<execution>
<id>jacoco-restore-instrumented-classes</id>
<goals>
<goal>restore-instrumented-classes</goal>
</goals>
</execution>
</executions>
<configuration>
<excludes>
<exclude>*</exclude>
</excludes>
</configuration>
</plugin>
</plugins>
</build>

Code Coverage with JaCoCo explained the reason
The simplest way to use JaCoCo it is — on-the-fly instrumentation with using JaCoCo Java Agent. In this case a class in modified when it is being loaded. You can just run you application with JaCoCo agent and a code coverage is calculated. This way is used by Eclemma and Intellij Idea. But there is a big issue. PowerMock instruments classes also. Javassist is used to modify classes. The main issue is that Javassist reads classes from disk and all JaCoCo changes are disappeared. As result zero code coverage for classes witch are loaded by PowerMock class loader.
We are going to replace Javassist with ByteBuddy (#727) and it should help to resolve this old issue. But right now there is NO WAY TO USE PowerMock with JaCoCo On-the-fly instrumentation. And no workaround to get code coverage in IDE.
JaCoCo Offline Instrumentation can solve this problem.
The sample of Offline Instrumentation fixed my problem .

I have tried these versions of the below softwares; they work together and generate the jacoco report without any extra effort. I use the gradle build tool with powermock, mockito and wiremock.
powermock-api-mockito2:2.0.2
powermock-module-junit4:2.0.2
gradle 5.6.2
The above correctly generates the jacoco coverage report.

I had same issue with JaCoCo On-the-fly and PowerMock. 0% code coverage was generated every time
Found out that JaCoCo version 0.7.7.201606060606 and PowerMock version 1.6.2 are compatible and code coverage is generated successfully.

The report is correct. The provided test leads to no coverage as you mock what happens when calling the method "say". One should never mock the "instance/class under test", only do this for the dependencies and surroundings of the test subject.
In this case: The code return "hello:"+s; is never reached by the test and the constructor of Utils is not called. There is no coverage displayed as there is no test coverage of these lines at all.
Nevertheless using JaCoCo and PowerMock together is not trivial. I am currently looking for how to get it to run, too. Maybe your provided test is just unluckily written and JaCoCo is already working as it should?

Categories