I'm trying to run Cucumber's feature files in parallel using Cucumber's CLI Runner and I'm currently stuck trying to figure out how to make JUnit #BeforeClass hook to work with the CLI Runner.
At the moment, my working Runner class looks like this:
#RunWith(Cucumber.class)
#CucumberOptions(
plugin = {
"pretty",
"html:target/reports/basic/report.html",
"json:target/reports/cluecumber/cucumber.json",
"timeline:target/reports/timeline"
},
tags = "#RegressionTests",
snippets = SnippetType.CAMELCASE,
stepNotifications = true,
features = "classpath:features",
glue = "my.steps.package")
public class RegressionTestsIT {
#BeforeClass
public static void setup() {
ContextHolder.setupTestContext();
}
}
And my CLI command looks like this:
java -cp "target/test-jar-with-dependencies.jar" io.cucumber.core.cli.Main -p "pretty" -p "html:target/reports/basic/report.html" -p "json:target/reports/cluecumber/cucumber.json" -p "timeline:target/reports/timeline" --threads 10 -g "my.steps.package" target/test-classes/features
What happens is that I get a NullPointerException at the tests because TestContext was not properly set up as the hook was not executed.
I tried to include both the Runner's package and the Runner class itself as glue and it didn't work.
Also tried to make my Runner extend io.cucumber.core.cli.Main and then execute my Runner in the CLI and not surprisingly it did not work either, sadly still getting NPE.
Although this issue is related to the CLI Runner use, I'm content with any answer that might help me run multiple feature files in parallel whatever the method.
Using JUnit Rules
Cucumber supports JUnit's #ClassRule, #BeforeClass, and #AfterClass annotations. These will be executed before and after all scenarios. Using these is not recommended as it limits portability between different runners; they may not execute correctly when using the command line, IntelliJ IDEA, or Cucumber-Eclipse. Instead it is recommended to use Cucumber's hooks.
When using the CLI, JUnit is not involved at all so you can not use any of JUnit annotations. However since Cucumber v7 you can use #BeforeAll and #AfterAll to declare methods that executed before and after all scenarios.
package io.cucumber.example;
import io.cucumber.java.AfterAll;
import io.cucumber.java.BeforeAll;
public class StepDefinitions {
#BeforeAll
public static void beforeAll() {
// Runs before all scenarios
}
#AfterAll
public static void afterAll() {
// Runs after all scenarios
}
}
JUnit #BeforeClass didn't work for me. Since I'm kinda in a hurry with this, I didn't bother keep on trying to make it work. I don't really need to run the command in a pipeline at the moment, so I was completely fine in running it on IntelliJ as long as it was running in parallel.
My solution was creating a custom CLI Runner that runs the context configuration before Cucumber's CLI run method.
public class CLIRunner {
public static void main(String[] args) {
ContextHolder.setupTestContext();
io.cucumber.core.cli.Main.run(
new String[] {
"-p", "pretty",
"-p", "html:target/reports/basic/report.html",
"-p", "json:target/reports/cluecumber/cucumber.json",
"-p", "timeline:target/reports/timeline",
"-g", "my.steps.package",
"classpath:features",
"--threads", "10"
}, Thread.currentThread().getContextClassLoader());
}
}
Related
I have a Spring Boot 2.5.4 project with some #SpringBootTest tests and some #Cucumber tests. I am using gradle to build.
I have noticed that my build is failing depending on where it's executed, and I found that it actually depended on the order the tests get executed, so I have a problem in my tests : if #SpringBootTest runs first then it's passing. if #Cucumber tests run first then it fails - probably because the H2 DB doesn't get fully reset in between.
Now, I would like to temporarily control the execution order, so that I can reproduce the issue consistently to fix the data dependency between my tests.
I am trying to use junit.jupiter.testclass.order.default property with value org.junit.jupiter.api.ClassOrderer$ClassName but it's not working.
I've put my 2 tests in a Juint5 #Suite mentioning the 2 tests in the #SelectClasses and changing their order, but even like that, it's not working - my feeling is that it's because there are actually 2 test runners, Junit Jupiter and Cucumber. Sometimes when I change something that doesn't seem related, the execution order changes :
I'm overriding Junit version to latest, hoping that it helps (and Junit5 Suite is available), but it doesn't help :
ext['junit-jupiter.version']='5.9.2'
I am using Cucumber 6.11.0.
My gradle test task is simply
test {
useJUnitPlatform()
finalizedBy(project.tasks.jacocoTestReport)
}
Is there a way to configure in my build the order in which the test runners get executed, in a consistent and reproduceable manner ?
Thanks
Gradle uses the JUnit Platform Launcher API to run the tests. You can do the same thing. And to ensure the order remains the same, you would invoke the platform twice within the same JVM.
This should allow you to reproduce your problem as Spring keeps the application running until the JVM exits.
import static org.junit.platform.engine.discovery.DiscoverySelectors.selectClass;
import my.project.spring.MySpringBootTest;
import my.project.cucumber.MyCucumberTest;
import org.junit.platform.launcher.Launcher;
import org.junit.platform.launcher.LauncherDiscoveryRequest;
import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder;
import org.junit.platform.launcher.core.LauncherFactory;
import org.junit.platform.launcher.listeners.SummaryGeneratingListener;
import org.junit.platform.launcher.listeners.TestExecutionSummary;
public class ManualLaunch {
public static void main(String[] args) {
var cucumberSummary= runCucumberTest();
var jupiterSummary= runJupiterTest();
System.out.println("Jupiter failures : "+jupiterSummary.getTestsFailedCount());
System.out.println("Cucumber failures : "+cucumberSummary.getTestsFailedCount());
}
private static TestExecutionSummary runCucumberTest() {
LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request()
// Configure the request for Cucumber here
.selectors(
selectClass(MyCucumberTest.class)
)
.build();
return launchAndGetSummary(request);
}
private static TestExecutionSummary runJupiterTest() {
LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request()
.selectors(
selectClass(MySpringBootTest.class)
)
.build();
return launchAndGetSummary(request);
}
private static TestExecutionSummary launchAndGetSummary(LauncherDiscoveryRequest request){
Launcher launcher = LauncherFactory.create();
SummaryGeneratingListener listener = new SummaryGeneratingListener();
launcher.registerTestExecutionListeners(listener);
launcher.execute(request);
return listener.getSummary();
}
}
If you don't know how to run a main method from Gradle, you could also use JUnit to run JUnit. But then you do have to make sure the tests don't target themselves.
What is the recommended solution to run a test multiple times with different system properties via command line?
What I would like to do could look like this:
gradle clean test --tests -Dmyproperty=foo my.fancy.test.TestClass --tests -Dmyproperty=bar my.fancy.test.TestClass
I need this parameter for the setup code in the #BeforeAll method.
As you use junit you can use #ParameterizedTest which allows you to repeat the same junit test with different parameters.
You would need to move the code for the setup from the "BeforeAll" somewhere else.
static Stream<String> testArgs() { return Stream.of(""," ", null,"12345"); }
#ParameterizedTest
#MethodSource("testArgs")
public void testByArgs(String arg){
// Your test with asserts based on args here ...
}
You need to also wire those parameters up with tests' JVM through Gradle's test dsl:
// build.gradle
test {
systemProperty 'myproperty', System.getProperty('myproperty')
}
// in your tests use it like:
#BeforeAll
fun setUp() {
System.getProperty("myproperty")
}
Basicailly this answer.
My team received ownership of a webapp. Tests are written with junit suites and serenity. Good things, there a good test coverage. Problem come when you need to run that single test/scenario that is still failing and you need to wait >30min to run everything.
How can I run a single scenario of this suite using mvn command line?
From code editor, it's hard to start single scenario as both suite and test classes contains important initialization code.
I've also tried argument '-Dtest=T1Test#T1Scenario1' without success.
Code snipplet:
#RunWith(Suite.class)
#Suite.SuiteClasses({
UserConfigASuite.class,
UserConfigBSuite.class,
UserConfigCSuite.class
})
public class AllTestSuite {
}
#RunWith(Suite.class)
#Suite.SuiteClasses({
T1Test.class,
T2Test.class,
//... Lots of other tests
})public class UserConfigASuite {
#BeforeClass
public static void beforeClass() {
//Required init code
}
#AfterClass
public static void afterClass() {
//Cleanup after test suite
}
}
#RunWith(SerenityRunner.class)
public class T1Test {
#Test
#Title("T1: scenario 1")
public void T1Scenario1() {
}
//... Lots of other scenarios
}
Just confirm first that your using supported surefire and junit version. For more details refer https://maven.apache.org/surefire/maven-surefire-plugin/examples/single-test.html
In case your using maven failsafe plugin then the syntax will vary little bit. Something like this
mvn -Dit.test=ITCircle#test* verify
Refer https://maven.apache.org/surefire/maven-failsafe-plugin/examples/single-test.html for more details.
How do I start a simple Vert.x server from inside IntelliJ IDEA?
My build.gradle is as below:
apply plugin: 'java'
version = '3.0.0'
repositories {
mavenCentral()
}
dependencies {
compile 'io.vertx:vertx-core:3.0.0'
}
My Vertx-server, MyVertex.java is as below:
package com.example;
import io.vertx.core.AbstractVerticle;
import io.vertx.core.Future;
public class MyVerticle extends AbstractVerticle {
#Override
public void start(Future<Void> fut) {
vertx.createHttpServer()
.requestHandler(r -> r.response().end("<h1>Hello</h1>"))
.listen(8081);
}
}
And my IntelliJ run configuration is as below, with io.vertx.core.Starteras main class:
But when I run it with my run configuration I get this error message:
Error: Could not find or load main class run
Is the VM option (in Run configuration) run something I need to install and add to my path or how do I get started with Vert.x-server development?
I'm using vertx 3.2.1 and it's complaining about io.vertx.core.Starter. It's deprecated now. So, one should use io.vertx.core.Launcher.
This is an example of launching via intellij with the option of specifying a config JSON file:
Main Class: io.vertx.core.Launcher
VM Options: <up to you, or leave blank>
Program Arguments: run com.app.verticle.MyVerticle -conf /path/to/my_config.json
When using a logging framework it will be added in VM Options as below.
Log4j with either log4j or slf4j delgate:
-Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.Log4jLogDelegateFactory -Dlog4j.configuration=log4j.xml
-Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -Dlog4j.configuration=log4j.xml
Logback:
-Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -Dlogback.configurationFile=logback.xml
Simply add this to your MyVerticle (or a separate class):
import io.vertx.core.Launcher;
...
public static void main(final String[] args) {
Launcher.executeCommand("run", MyVerticle.class.getName());
}
Then simply Ctrl+Shift+F10 to run it and IntelliJ will automatically create the Run Configuration.
Ah, my mistake:
run com.example.MyVerticle should be the value of Program arguments: and not as VM options in the IntelliJ IDEA Run configuration.
You can simply add a main and use deployVerticle() and then from there in IntelliJ you can Run or Debug it easily.
With deployVerticle, you can pass a new instance of your main/bootstrap verticle or you can pass yourMainVerticle.class
public class VertxVerticleMain {
public static void main(String[] args) {
Vertx vertx = Vertx.vertx();
vertx.deployVerticle(new MyVerticle());
//vertx.deployVerticle(MyVerticle.class);
}
}
You have to use this: org.vertx.java.platform.impl.cli.Starter as your Main Class in IntelliJ IDEA; and if you are using arguments and things like that you might want to use something like: runmod <groupId>~<artifactId>~<version> [-conf src/main/resources/your_config.json -cp]
Have a look at this project.
For Vert.x 3.0.0 you have to use this: io.vertx.core.Starter as your Main Class and run com.example.other.AnyVerticle as your Program arguments.
is there a way to specify the JVM version in a junit test, when multiple java types are available on an OS ?
I'd like to do something like this:
#JVM=1.5
public void test1(){
run();
}
#JVM=1.7
public void test2(){
run();
}
etc..
No, you'll have to run the JUnit tests multiple times with multiple VMs. You can then use Assume to only run the tests for the correct VM:
public void test1(){
// not compiled or tested
Assume.assumeThat(System.getProperty("java.version"), CoreMatchers.startsWith("1.5"));
run();
}
public void test2(){
Assume.assumeThat(System.getProperty("java.version"), CoreMatchers.startsWith("1.7"));
run();
}
Please note that I haven't compiled this, so there may be errors. Then, in your build tool, run these tests twice in two different VMs.
On JUnit 5, you can use EnabledOnJre and DisabledOnJre annotations.
#EnabledOnJre(JRE.JAVA_8)
void testMyMethodInJava8() {
...
}
#DisabledOnJre(JRE.JAVA_8)
void testMyMethodInOtherVersions() {
...
}
You can also apply these to a whole test class:
#EnabledOnJre(JRE.JAVA_11)
public class MyClassTest {
...
}
You still need to run the test suite separately for each JVM, but this allows you to skip the tests that will not work in one or the other.
JUnit test cases are just a bunch of Java classes, not a standalone program. So they would still be confined within the same JVM. AFAIK there is no simple way to do what you intend to do. What you can do is separate the test cases into test suites for each java version you want to test for. Then change the java version in the project settings and run the appropriate test suites.
Also there is a junit.jvm property, but I am not sure if you can change it mid-execution. Have you taken a look at it?