I need to be able to restart currently executing cucumber test case or scenario so that if it is about to be marked as failed in the #After phase it will restart immediately. I have looked into:
Flaky Test handler Plugin
ExtendedCucumberOptions
TestRule (JUnit) - Retry Rule
but none of the above have worked for me, or simply did not do what I am seeking to achieve. The behavior I am looking for is as follows:
The test starts with cucumber runner class for example:
#RunWith(Cucumber.class)
#CucumberOptions(tags = "#MyTag"
, format = {"pretty", "html:target/MyTest/cucumber", "json:target/MyTest/cucumber.json"}
, glue = {"my.ui.reportsGlue"})
public class RunMyTest {
}
Cucumber Steps definitions where the retry logic should happen:
#Before("#MyTag")
public void setUp() {
// test set up methods
}
#After("#MyTag")
public void stop(Scenario scenario) {
if (scenario.isFailed() && retryCount <=nTimes) {
nTimes++;
// currently running scenario should restart from the beginning here
}
}
Any help would be greatly appreciated.
Related
I have a Spring Boot 2.5.4 project with some #SpringBootTest tests and some #Cucumber tests. I am using gradle to build.
I have noticed that my build is failing depending on where it's executed, and I found that it actually depended on the order the tests get executed, so I have a problem in my tests : if #SpringBootTest runs first then it's passing. if #Cucumber tests run first then it fails - probably because the H2 DB doesn't get fully reset in between.
Now, I would like to temporarily control the execution order, so that I can reproduce the issue consistently to fix the data dependency between my tests.
I am trying to use junit.jupiter.testclass.order.default property with value org.junit.jupiter.api.ClassOrderer$ClassName but it's not working.
I've put my 2 tests in a Juint5 #Suite mentioning the 2 tests in the #SelectClasses and changing their order, but even like that, it's not working - my feeling is that it's because there are actually 2 test runners, Junit Jupiter and Cucumber. Sometimes when I change something that doesn't seem related, the execution order changes :
I'm overriding Junit version to latest, hoping that it helps (and Junit5 Suite is available), but it doesn't help :
ext['junit-jupiter.version']='5.9.2'
I am using Cucumber 6.11.0.
My gradle test task is simply
test {
useJUnitPlatform()
finalizedBy(project.tasks.jacocoTestReport)
}
Is there a way to configure in my build the order in which the test runners get executed, in a consistent and reproduceable manner ?
Thanks
Gradle uses the JUnit Platform Launcher API to run the tests. You can do the same thing. And to ensure the order remains the same, you would invoke the platform twice within the same JVM.
This should allow you to reproduce your problem as Spring keeps the application running until the JVM exits.
import static org.junit.platform.engine.discovery.DiscoverySelectors.selectClass;
import my.project.spring.MySpringBootTest;
import my.project.cucumber.MyCucumberTest;
import org.junit.platform.launcher.Launcher;
import org.junit.platform.launcher.LauncherDiscoveryRequest;
import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder;
import org.junit.platform.launcher.core.LauncherFactory;
import org.junit.platform.launcher.listeners.SummaryGeneratingListener;
import org.junit.platform.launcher.listeners.TestExecutionSummary;
public class ManualLaunch {
public static void main(String[] args) {
var cucumberSummary= runCucumberTest();
var jupiterSummary= runJupiterTest();
System.out.println("Jupiter failures : "+jupiterSummary.getTestsFailedCount());
System.out.println("Cucumber failures : "+cucumberSummary.getTestsFailedCount());
}
private static TestExecutionSummary runCucumberTest() {
LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request()
// Configure the request for Cucumber here
.selectors(
selectClass(MyCucumberTest.class)
)
.build();
return launchAndGetSummary(request);
}
private static TestExecutionSummary runJupiterTest() {
LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request()
.selectors(
selectClass(MySpringBootTest.class)
)
.build();
return launchAndGetSummary(request);
}
private static TestExecutionSummary launchAndGetSummary(LauncherDiscoveryRequest request){
Launcher launcher = LauncherFactory.create();
SummaryGeneratingListener listener = new SummaryGeneratingListener();
launcher.registerTestExecutionListeners(listener);
launcher.execute(request);
return listener.getSummary();
}
}
If you don't know how to run a main method from Gradle, you could also use JUnit to run JUnit. But then you do have to make sure the tests don't target themselves.
I'm trying to run Cucumber's feature files in parallel using Cucumber's CLI Runner and I'm currently stuck trying to figure out how to make JUnit #BeforeClass hook to work with the CLI Runner.
At the moment, my working Runner class looks like this:
#RunWith(Cucumber.class)
#CucumberOptions(
plugin = {
"pretty",
"html:target/reports/basic/report.html",
"json:target/reports/cluecumber/cucumber.json",
"timeline:target/reports/timeline"
},
tags = "#RegressionTests",
snippets = SnippetType.CAMELCASE,
stepNotifications = true,
features = "classpath:features",
glue = "my.steps.package")
public class RegressionTestsIT {
#BeforeClass
public static void setup() {
ContextHolder.setupTestContext();
}
}
And my CLI command looks like this:
java -cp "target/test-jar-with-dependencies.jar" io.cucumber.core.cli.Main -p "pretty" -p "html:target/reports/basic/report.html" -p "json:target/reports/cluecumber/cucumber.json" -p "timeline:target/reports/timeline" --threads 10 -g "my.steps.package" target/test-classes/features
What happens is that I get a NullPointerException at the tests because TestContext was not properly set up as the hook was not executed.
I tried to include both the Runner's package and the Runner class itself as glue and it didn't work.
Also tried to make my Runner extend io.cucumber.core.cli.Main and then execute my Runner in the CLI and not surprisingly it did not work either, sadly still getting NPE.
Although this issue is related to the CLI Runner use, I'm content with any answer that might help me run multiple feature files in parallel whatever the method.
Using JUnit Rules
Cucumber supports JUnit's #ClassRule, #BeforeClass, and #AfterClass annotations. These will be executed before and after all scenarios. Using these is not recommended as it limits portability between different runners; they may not execute correctly when using the command line, IntelliJ IDEA, or Cucumber-Eclipse. Instead it is recommended to use Cucumber's hooks.
When using the CLI, JUnit is not involved at all so you can not use any of JUnit annotations. However since Cucumber v7 you can use #BeforeAll and #AfterAll to declare methods that executed before and after all scenarios.
package io.cucumber.example;
import io.cucumber.java.AfterAll;
import io.cucumber.java.BeforeAll;
public class StepDefinitions {
#BeforeAll
public static void beforeAll() {
// Runs before all scenarios
}
#AfterAll
public static void afterAll() {
// Runs after all scenarios
}
}
JUnit #BeforeClass didn't work for me. Since I'm kinda in a hurry with this, I didn't bother keep on trying to make it work. I don't really need to run the command in a pipeline at the moment, so I was completely fine in running it on IntelliJ as long as it was running in parallel.
My solution was creating a custom CLI Runner that runs the context configuration before Cucumber's CLI run method.
public class CLIRunner {
public static void main(String[] args) {
ContextHolder.setupTestContext();
io.cucumber.core.cli.Main.run(
new String[] {
"-p", "pretty",
"-p", "html:target/reports/basic/report.html",
"-p", "json:target/reports/cluecumber/cucumber.json",
"-p", "timeline:target/reports/timeline",
"-g", "my.steps.package",
"classpath:features",
"--threads", "10"
}, Thread.currentThread().getContextClassLoader());
}
}
I have a scenario in my feature file that contains two tags
#tag1, #tag2
Scenario: This is a test
Given I open the website
When I log in
Then the account page is open
Now #tag is part of a cucumber serenity runner and this works in isolation (before the implementation of #tag2):
#RunWith(CucumberWithSerenity.class)
#CucumberOptions(features="...features/", glue = {".../steps"}, tags = {"#tag1"} )
public class Tag1Runner extends Hooks {
#BeforeClass
public static void startAppium() {
AppiumServerController.startAppiumServer();
}
#AfterClass
public static void stopAppium() {
AppiumServerController.stopAppiumServer();
}
Now what I also want to do is run #tag2, but this only runs after the test has been completed. This is because the method that occurs in #tag2 should not be part of the scenario to test, but should complete a clean up after the test has ran.
Now if I include the following method below, when I run the feature file as it is displayed above, nothing happens.
How can I implement #tag2 to work as mentioned?
public class Hooks{
#After("#tag2")
public void completeCleanUp() {
//code to perform clean up...
}
}
If you are trying to run an After hook after each scenario tagged with "#tag1" in your example, you should mark the After hook with "#tag1". For more information on tagged hooks, please refer to the Cucumber documentation.
My team received ownership of a webapp. Tests are written with junit suites and serenity. Good things, there a good test coverage. Problem come when you need to run that single test/scenario that is still failing and you need to wait >30min to run everything.
How can I run a single scenario of this suite using mvn command line?
From code editor, it's hard to start single scenario as both suite and test classes contains important initialization code.
I've also tried argument '-Dtest=T1Test#T1Scenario1' without success.
Code snipplet:
#RunWith(Suite.class)
#Suite.SuiteClasses({
UserConfigASuite.class,
UserConfigBSuite.class,
UserConfigCSuite.class
})
public class AllTestSuite {
}
#RunWith(Suite.class)
#Suite.SuiteClasses({
T1Test.class,
T2Test.class,
//... Lots of other tests
})public class UserConfigASuite {
#BeforeClass
public static void beforeClass() {
//Required init code
}
#AfterClass
public static void afterClass() {
//Cleanup after test suite
}
}
#RunWith(SerenityRunner.class)
public class T1Test {
#Test
#Title("T1: scenario 1")
public void T1Scenario1() {
}
//... Lots of other scenarios
}
Just confirm first that your using supported surefire and junit version. For more details refer https://maven.apache.org/surefire/maven-surefire-plugin/examples/single-test.html
In case your using maven failsafe plugin then the syntax will vary little bit. Something like this
mvn -Dit.test=ITCircle#test* verify
Refer https://maven.apache.org/surefire/maven-failsafe-plugin/examples/single-test.html for more details.
Integration test executed by cucumber tends to leave behind context that causes problems with subsequent tests. Obvious solution appeared to be Spring's #DirtiesContext, but instead of tearing down the context after all the cucumber features have been run, it does this after each and every scenario, thus making the test execution time rather lengthy.
Tried also with #TestExecutionListeners, but no luck.
#RunWith( SpringJUnit4ClassRunner.class )
#ContextConfiguration( classes = { MyApplication.class, MyTestComponent.class }, loader = SpringApplicationContextLoader.class )
#ActiveProfiles( { "test", "someotherprofile" } )
#DirtiesContext( classMode = DirtiesContext.ClassMode.AFTER_CLASS )
#WebIntegrationTest( randomPort = true )
public class StepDefs extends StepDefUtils {
// givens, whens, thens
Am I trying to use DirtiesContext in an unsupported way?
As previous answer said the scenarios get compiled and run as separate classes stopping DirtiesContext from working and there are no per feature hooks in cucumber for same reason.
Workaround is to put tags in scenarios and have a class with hook detect these and conditionally dirty the context during the afterTestClass method.
The tag lets you control when context gets dirtied for example if want each feature to have fresh context then mark last scenario with tag, or can have many time per feature as and when needed.
public class CucumberFeatureDirtyContextTestExecutionListener extends AbstractTestExecutionListener{
private static boolean dirtyContext = false;
#After("#DirtyContextAfter")
public void afterDirtyContext(){
dirtyContext = true;
}
#Override public void afterTestClass(TestContext testContext) throws Exception {
if (dirtyContext) {
testContext.markApplicationContextDirty(HierarchyMode.EXHAUSTIVE);
testContext.setAttribute(DependencyInjectionTestExecutionListener.REINJECT_DEPENDENCIES_ATTRIBUTE, TRUE);
dirtyContext = false;
}
}
}
Mark scenarios with tag
#DirtyContextAfter
Scenario: My scenario
On steps class register the listener with spring
#TestExecutionListeners(listeners = {DependencyInjectionTestExecutionListener.class, CucumberFeatureDirtyContextTestExecutionListener.class})
Make sure the listener is in cucumber glue so after hook is registerd
Could not get it working on beforeClass as the context is already set up so have to do on afterClass.
Cucumber test methods are compiled into different test classes so #DirtiesContext( classMode = DirtiesContext.ClassMode.AFTER_CLASS ) will be run after each test method.
Unfortunately I don't see any DirtiesContext mode which suits your needs.
I would search for some cucumber listener and manually make spring context dirty trough it.