We Keep Coding

SonarQube doesn't increase test coverage as I write more and more tests

I've been facing a really weird problem with a project and the use of SonarQube.
As you can see in the image below, my test coverage is 9.5% of my overall code which is pretty low regarding of the quantity of code I've been writing and will be in the future.
When I first tried to write tests, they were not detected, because I forgot a plugin inside my pom.xml, which I added and is the following
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.8</version>
<executions>
<execution>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<execution>
<id>generate-code-coverage-report</id>
<phase>test</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
</executions>
</plugin>
After that, I got this window on SonarQube.
Nice ! My test are detected and I went to 13.2% on new code and 9.5% overall !
But that's where the real problem started, as I wrote more test and be pushing them to SonarQube, nothing more was shown. The percentage didn't increased, and the line that I thought the tests covered were not.
And the weirdest part is that the test code was on SonarQube ! It was pushed to Sonar but not detected as test code or whatever it should be !
So, I tried to watch as many videos as I could but nothing seems to really work and I just had the feeling that I lost time. I've been trying to code simple test classes on my simplest classes, for example :
Here is my class : Categorie.java
package com.back.projetbdi_groupe1.Entities;
import jakarta.persistence.Entity;
import jakarta.persistence.Id;
#Entity
public class Categorie {
#Id
private String idCategorie;
private String libCateg;
public String getIdCategorie() {
return idCategorie;
}
public void setIdCategorie(String idCategorie) {
this.idCategorie = idCategorie;
}
public String getLibCateg() {
return libCateg;
}
public void setLibCateg(String libCateg) {
this.libCateg = libCateg;
}
}
And my test class : CategorieTest.java
package com.back.projetbdi_groupe1.entities;
import com.back.projetbdi_groupe1.Entities.Categorie;
import org.junit.Test;
import static org.junit.jupiter.api.Assertions.assertEquals;
public class CategorieTest {
#Test
public void testGetIdCategorie(){
Categorie categorie = new Categorie();
categorie.setIdCategorie("1");
assertEquals("1",categorie.getIdCategorie());
}
#Test
public void testGetLibCateg(){
Categorie categorie = new Categorie();
categorie.setLibCateg("categ");
assertEquals("categ",categorie.getLibCateg());
}
}
You can see it in sonar :
But :
You can see that nothing is covered.
So, I wanted to know if I'm not testing the right way, or is it a SonarQube bug ? Or my pom.xml is incomplete ? I will put what I found useful about the pom.xml below.
Oh, and to " push " to SonarQube the code, I'm using the following command : mvn clean verify sonar:sonar -Dsonar.login=MyFabulousAndLongToken
Rest of the pom.xml :
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<jettyVersion>9.4.3.v20170317</jettyVersion>
<jettyServletVersion>9.4.3.v20170317</jettyServletVersion>
<sonar.host.url>http://im2ag-sonar.u-ga.fr:9000/</sonar.host.url>
<java.version>17</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>5.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<version>5.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-library</artifactId>
<version>2.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.jayway.jsonpath</groupId>
<artifactId>json-path</artifactId>
<version>2.5.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13.1</version>
<scope>test</scope>
</dependency>

I don't have a definitive answer, just some things that should be checked.
It's a little odd to use the "clean" goal on the same mvn command line you run "sonar:sonar", but it's possible that can work. Inspect the output from this command and verify that it's running all of your unit tests.
When Jacoco and Surefire are properly integrated, it should generate an html page you can visit to inspect the Jacoco results. It might be in "target/jacoco_report". If that is there (it's possible it might be in a different subdirectory), then view that in your browser and see if there are tests missing from that report. SonarQube only integrates the jacoco data that is produced.
In the SonarQube results, you can also see the list of unit tests that are executed. See if "CategorieTest" is in that list.

So, to resolve my own problem I did nothing but testing my code "less wrongly" if I could say.
That's what the previous example of testing looks like now !
package com.back.projetbdi_groupe1.Entities;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.BeforeAll;
import static org.junit.jupiter.api.Assertions.assertEquals;
class CategorieTest {
/**
* Methods under test:
*
* <ul>
* <li>default or parameterless constructor of {#link Categorie}
* <li>{#link Categorie#setIdCategorie(String)}
* <li>{#link Categorie#setLibCateg(String)}
* <li>{#link Categorie#getIdCategorie()}
* <li>{#link Categorie#getLibCateg()}
* </ul>
*/
static Categorie actualCategorie;
#BeforeAll
public static void init() {
actualCategorie = new Categorie();
actualCategorie.setIdCategorie("1");
actualCategorie.setLibCateg("Lib Categ");
}
#Test
void testGetIdCategorie() {
assertEquals("1", actualCategorie.getIdCategorie());
}
#Test
void testGetLibCateg() {
assertEquals("Lib Categ", actualCategorie.getLibCateg());
}
}
And in Sonar :
Everything is green, im H A P P Y.

Unit testing java maven

I have a maven project that looks like this:
If I run in the terminal:
mvn test
it will build the application and run the 3 tests:
SocialMultiplicationApplicationTests
MultiplicationServiceTest
RandomGeneratorServiceTest
but not RandomGeneratorServiceImplTest.
If I try to explicitly run this class:
mvn test -Dtest=RandomGeneratorServiceImplTest
I get:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.2:test (default-test)
on project social-multiplication: No tests were executed! (Set -DfailIfNoTests=false to ignore this error.) -> [Help 1]
Here's my pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.7.3</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>microservices.book</groupId>
<artifactId>social-multiplication</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>social-multiplication</name>
<description>Social Multiplication App</description>
<properties>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-test</artifactId>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
</dependency>
<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
Here's the test class that won't run:
package microservices.book.multiplication.service;
import static org.assertj.core.api.Assertions.assertThat;
import org.junit.Before;
import org.junit.Test;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
public class RandomGeneratorServiceImplTest {
private RandomGeneratorServiceImpl randomGeneratorServiceImp;
#Before
public void setUp(){
randomGeneratorServiceImp = new RandomGeneratorServiceImpl();
}
#Test
public void generateRandomFactorIsBetweenExpectedLimits() {
List<Integer> randomFactors = IntStream.range(0,1000)
.map(i -> randomGeneratorServiceImp.generateRandomFactor())
.boxed()
.collect(Collectors.toList());
for(Integer i: randomFactors){
assertThat(i).isIn(
IntStream.range(11, 100).
boxed().collect(Collectors.toList()));
}
}
}
EDIT: SOLUTION
The proble was indeed the conflict between junit4 and junit5.
I chose to move my tests to junit5, hence I replaced #Before with #BeforeAll and added #TestInstance(Lifecycle.PER_CLASS) on top of the RandomGeneratorServiceImplTest class definition.
Couple links I found useful:
junit docs
differences between junit4 and junit5

The reason lies in mixing JUnit 5 and JUnit 4.
The maven-surefire-plugin picks only one strategy to execute tests and favors JUnit 5 in this case.
Either replace the dependency to junit by junit-vintage-engine:
<!-- <dependency>-->
<!-- <groupId>junit</groupId>-->
<!-- <artifactId>junit</artifactId>-->
<!-- <scope>test</scope>-->
<!-- </dependency>-->
<dependency>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
<scope>test</scope>
</dependency>
Or adapt your tests to use JUnit 5 instead (as for example in RandomGeneratorServiceImplTest):
...
//import org.junit.Before;
//import org.junit.Test;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
...
public class RandomGeneratorServiceImplTest {
...
// #Before
#BeforeEach
public void setUp(){
...
}
// #Test can stay as is, only import needs to be updated
#Test
public void generateRandomFactorIsBetweenExpectedLimits() {
...
}
}

Try adding this plugin to your build section in pom.xml:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
</plugin>

ClassNotFoundException when running JUnit test in Maven

I'm currently trying to set up automated testing for a Maven project, but I've run into a problem. When running my tests using mvn test, I get the following result:
-------------------------------------------------------------------------------
Test set: no.digipat.patornat.servlets.ServletTests
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.398 s <<< FAILURE! - in no.digipat.patornat.servlets.ServletTests
no.digipat.patornat.servlets.ServletTests Time elapsed: 0.368 s <<< ERROR!
java.lang.NoClassDefFoundError: com/mongodb/OperationExecutor
Caused by: java.lang.ClassNotFoundException: com.mongodb.OperationExecutor
I get the same error when running the tests in Eclipse.
This is my pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>no.digipat.patornat</groupId>
<artifactId>backend</artifactId>
<packaging>war</packaging>
<version>0.0.1-SNAPSHOT</version>
<name>Pat or Nat Backend</name>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.target>11</maven.compiler.target>
<maven.compiler.source>11</maven.compiler.source>
</properties>
<repositories>
<repository>
<id>cytomine-uliege-Cytomine-java-client</id>
<url>https://packagecloud.io/cytomine-uliege/Cytomine-java-client/maven2</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.lordofthejars</groupId>
<artifactId>nosqlunit-mongodb</artifactId>
<version>1.0.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.github.fakemongo</groupId>
<artifactId>fongo</artifactId>
<version>2.2.0-RC2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.github.stefanbirkner</groupId>
<artifactId>system-rules</artifactId>
<version>1.19.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<version>3.0.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>be.cytomine.client</groupId>
<artifactId>cytomine-java-client</artifactId>
<version>2.0.7-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongo-java-driver</artifactId>
<version>3.12.1</version>
<scope>compile</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M4</version>
<configuration>
<includes>
<include>ServletTests.java</include>
</includes>
</configuration>
</plugin>
</plugins>
</build>
</project>
The relevant Java files:
ServletTests.java:
package no.digipat.patornat.servlets;
import org.junit.BeforeClass;
import org.junit.ClassRule;
import org.junit.contrib.java.lang.system.EnvironmentVariables;
import org.junit.rules.RuleChain;
import org.junit.rules.TestRule;
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
import org.junit.runners.Suite.SuiteClasses;
import static com.lordofthejars.nosqlunit.mongodb.InMemoryMongoDb.InMemoryMongoRuleBuilder.newInMemoryMongoDbRule;
#RunWith(Suite.class)
#SuiteClasses({MyTest.class})
public class ServletTests {
private static final EnvironmentVariables environmentVariables = new EnvironmentVariables();
#ClassRule
public static final TestRule chain = RuleChain
.outerRule(newInMemoryMongoDbRule().build())
.around(environmentVariables);
#BeforeClass
public static void setUpClass() {
environmentVariables.set("MY_VARIABLE", "some value");
}
}
MyTest.java:
package no.digipat.patornat.servlets;
import static org.junit.Assert.*;
import org.junit.Test;
public class MyTest {
#Test
public void test() {
fail("Not yet implemented");
}
}
Any ideas on how to solve this? I tried running mvn clean (which previously helped me fix a similar issue), but to no avail.

I think you need the maven-compiler-plugin in your pom.xml
<build>
<!-- put here the path of your test source directory -->
<testSourceDirectory>src/test</testSourceDirectory>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>${maven.compiler.source}</source>
<target>${maven.compiler.target}</target>
</configuration>
</plugin>
</plugins>
...

It seems that the issue was that the latest versions of Fongo and mongo-java-driver are incompatible. When I change the dependencies of Fongo and/or the Mongo Java driver to older versions (I specifically tried versions 2.1.0 and 3.6.3, respectively), the error disappears. However, since this solution seems rather fragile and inflexible, the best bet is probably to switch to an alternative to Fongo. According to this GitHub comment, mongo-java-server could be a good option. A perhaps more robust alternative, and the one I'll probably end up using, is to use a "real" test database. This article has information about a couple of ways to do this.

Intellij doesn't recognise Mockito class as Test class

i use IntellJ Ultimate and have problems to run a Mockito test class.
If I use a class like that
import org.junit.jupiter.api.Test;
class FactoryTest {
#Test
void createEntry() {
}
}
I can straight "run" this class.
If I have this:
import static org.mockito.Mockito.when;
import org.junit.Assert;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.runners.MockitoJUnitRunner;
// #RunWith attaches a runner with the test class to initialize the test data
#RunWith(MockitoJUnitRunner.class)
public class MathApplicationTester {
//#InjectMocks annotation is used to create and inject the mock object
#InjectMocks
MathApplication mathApplication = new MathApplication();
//#Mock annotation is used to create the mock object to be injected
#Mock
CalculatorService calcService;
#Test
public void testAdd(){
//add the behavior of calc service to add two numbers
when(calcService.add(10.0,20.0)).thenReturn(30.00);
//test the add functionality
Assert.assertEquals(mathApplication.add(10.0, 20.0),30.0,0);
}
}
I have to create another class before and run it which seems not correct to me because in that tutorial there is nothing written about an exlucisv class which starts the test http://www.vogella.com/tutorials/Mockito/article.html#testing-with-mock-objects
public class TestRunner {
public static void main(String[] args) {
Result result = JUnitCore.runClasses(MathApplicationTester.class);
for (Failure failure : result.getFailures()) {
System.out.println(failure.toString());
}
System.out.println(result.wasSuccessful());
}
}
My POM:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>1</groupId>
<artifactId>2</artifactId>
<version>1.0-SNAPSHOT</version>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<version>RELEASE</version>
</dependency>
<dependency>
<groupId>1</groupId>
<artifactId>2</artifactId>
<version>1.0-SNAPSHOT</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>RELEASE</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.mockito/mockito-all -->
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>1.9.5</version>
</dependency>
</dependencies>
</project>

MockitoJUnitRunner is already deprecated.
After getting rid of it, you can initialize your mocks with a #Before-annotated method:
#Before
public void setup() {
MockitoAnnotations.initMocks(this);
}

Accessing compiled resources in a Pax Exam contained test

I've been trying to run a JMH benchmark test in an OSGI container created with Pax Exam (similarly to how it's described in this SO answer, but with Pax Exam thrown into the mix as well). But am having some problems getting the JMH generated resource files created during the compilation to load.
After the tests have been compiled, the following can be found in the target/test-classes directory:
$ ls -l target/test-classes/META-INF
BenchmarkList CompilerHints
In a test I use some code (that I cannot change) that (effectively) looks for the file like so:
getClass().getClassLoader().getResources("META-INF/BenchmarkList");
Running this fails (returns 0 results) and I get the following error:
java.lang.RuntimeException: ERROR: Unable to find the resource: /META-INF/BenchmarkList
at org.openjdk.jmh.runner.AbstractResourceReader.getReaders(AbstractResourceReader.java:96)
at org.openjdk.jmh.runner.BenchmarkList.find(BenchmarkList.java:104)
at org.openjdk.jmh.runner.Runner.internalRun(Runner.java:228)
at org.openjdk.jmh.runner.Runner.run(Runner.java:178)
at com.company.project.performance.MyBenchmarkTest.launchBenchmark(MyBenchmarkTest.java:145)
I've tried creating a bundle containing the file, like so:
streamBundle(bundle()
.add("META-INF/BenchmarkList", new FileInputStream("target/test-classes/META-INF/BenchmarkList"))
.build()),
Which does create a JAR with the following contents:
META-INF/MANIFEST.MF
META-INF/
META-INF/BenchmarkList
But the problem persists. How could I make the resource file available for JMH?
An MCVE:
pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.company.project</groupId>
<artifactId>performance</artifactId>
<version>0.0.1</version>
<packaging>jar</packaging>
<name>MCVE for JMH+PaxExam issue</name>
<prerequisites>
<maven>3.0</maven>
</prerequisites>
<properties>
<apache-servicemix.version>5.4.0</apache-servicemix.version>
<junit.version>4.11</junit.version>
<jmh.version>1.10.1</jmh.version>
<pax-exam.version>4.4.0</pax-exam.version>
<tinybundles.version>2.1.0</tinybundles.version>
<maven-sunfire-report-plugin.version>2.18.1</maven-sunfire-report-plugin.version>
<maven-depends-plugin.version>1.2</maven-depends-plugin.version>
<maven-compiler-plugin.version>3.3</maven-compiler-plugin.version>
<javac.target>1.7</javac.target>
</properties>
<dependencies>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-core</artifactId>
<version>${jmh.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-generator-annprocess</artifactId>
<version>${jmh.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.ops4j.pax.exam</groupId>
<artifactId>pax-exam</artifactId>
<version>${pax-exam.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.ops4j.pax.exam</groupId>
<artifactId>pax-exam-container-karaf</artifactId>
<version>${pax-exam.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.ops4j.pax.exam</groupId>
<artifactId>pax-exam-junit4</artifactId>
<version>${pax-exam.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.servicemix</groupId>
<artifactId>apache-servicemix</artifactId>
<version>${apache-servicemix.version}</version>
<scope>test</scope>
<type>zip</type>
</dependency>
<dependency>
<groupId>org.ops4j.pax.tinybundles</groupId>
<artifactId>tinybundles</artifactId>
<version>${tinybundles.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.servicemix.tooling</groupId>
<artifactId>depends-maven-plugin</artifactId>
<version>${maven-depends-plugin.version}</version>
<executions>
<execution>
<id>generate-depends-file</id>
<goals>
<goal>generate-depends-file</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-sunfire-report-plugin.version}</version>
<dependencies>
<dependency>
<groupId>org.apache.maven.surefire</groupId>
<artifactId>surefire-junit47</artifactId>
<version>${maven-sunfire-report-plugin.version}</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>${maven-compiler-plugin.version}</version>
<configuration>
<source>${javac.target}</source>
<target>${javac.target}</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
src/test/java/com/company/project/performance/MyBenchmarkTest.java:
package com.company.project.performance;
import static org.ops4j.pax.exam.CoreOptions.junitBundles;
import static org.ops4j.pax.exam.CoreOptions.maven;
import static org.ops4j.pax.exam.CoreOptions.options;
import static org.ops4j.pax.exam.CoreOptions.streamBundle;
import static org.ops4j.pax.exam.CoreOptions.wrappedBundle;
import static org.ops4j.pax.exam.karaf.options.KarafDistributionOption.karafDistributionConfiguration;
import static org.ops4j.pax.exam.karaf.options.KarafDistributionOption.keepRuntimeFolder;
import static org.ops4j.pax.tinybundles.core.TinyBundles.bundle;
import static org.ops4j.pax.tinybundles.core.TinyBundles.withBnd;
import java.io.File;
import java.io.FileInputStream;
import java.util.concurrent.TimeUnit;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;
import org.ops4j.pax.exam.Configuration;
import org.ops4j.pax.exam.MavenUtils;
import org.ops4j.pax.exam.Option;
import org.ops4j.pax.exam.junit.PaxExam;
import org.ops4j.pax.exam.options.MavenArtifactUrlReference;
#RunWith(PaxExam.class)
public class MyBenchmarkTest
{
public static final String BENCHMARK_LIST = "META-INF/BenchmarkList";
#Benchmark
public void measureThroughput() throws InterruptedException
{
TimeUnit.MILLISECONDS.sleep(100);
}
#Configuration
public Option[] config() throws Exception
{
String karafVersion = MavenUtils.getArtifactVersion("org.apache.karaf", "apache-karaf");
MavenArtifactUrlReference servicemixUrl = maven()
.groupId("org.apache.servicemix")
.artifactId("apache-servicemix")
.versionAsInProject()
.type("zip");
return options(karafDistributionConfiguration()
.frameworkUrl(servicemixUrl)
.useDeployFolder(false)
.karafVersion(karafVersion)
.unpackDirectory(new File("target/exam")),
keepRuntimeFolder(),
junitBundles(),
wrappedBundle(maven("org.openjdk.jmh", "jmh-core")),
streamBundle(bundle()
.add(BENCHMARK_LIST, new FileInputStream("target/test-classes/" + BENCHMARK_LIST))
.build(withBnd())));
}
#Test
public void launchBenchmark() throws Exception
{
Options opts = new OptionsBuilder()
.include("com.company.project.performance.*")
.warmupIterations(1)
.forks(1)
.build();
new Runner(opts).run();
}
}

The issue is clearly an import export issue.
As your test which is using your meta data, isn't aware of those extra data.
The bundle you generated with the streamBundle, needs to add an extra header information actually exporting those extra data.
streamBundle(bundle()
.add(BENCHMARK_LIST, new FileInputStream("target/test-classes/" + BENCHMARK_LIST))
.set(Constants.EXPORT_PACKAGE, BENCHMARK_LIST)
and in your test you'll need to make sure you are actually importing it.
#ProbeBuilder
public TestProbeBuilder probeConfiguration(TestProbeBuilder probe) {
//make sure the needed imports are there.
probe.setHeader(Constants.IMPORT_PACKAGE, "*,"+BENCHMARK_LIST);
return probe;
}
on the other hand it might just be a lot better to actually try to add those extra data into the test-bundle (your test class is generated into a bundle on the fly)
Therefore the following should be added to the configuration
.metaInfResource(BENCHMARK_LIST)