I have a Spring Boot 2.5.4 project with some #SpringBootTest tests and some #Cucumber tests. I am using gradle to build.
I have noticed that my build is failing depending on where it's executed, and I found that it actually depended on the order the tests get executed, so I have a problem in my tests : if #SpringBootTest runs first then it's passing. if #Cucumber tests run first then it fails - probably because the H2 DB doesn't get fully reset in between.
Now, I would like to temporarily control the execution order, so that I can reproduce the issue consistently to fix the data dependency between my tests.
I am trying to use junit.jupiter.testclass.order.default property with value org.junit.jupiter.api.ClassOrderer$ClassName but it's not working.
I've put my 2 tests in a Juint5 #Suite mentioning the 2 tests in the #SelectClasses and changing their order, but even like that, it's not working - my feeling is that it's because there are actually 2 test runners, Junit Jupiter and Cucumber. Sometimes when I change something that doesn't seem related, the execution order changes :
I'm overriding Junit version to latest, hoping that it helps (and Junit5 Suite is available), but it doesn't help :
ext['junit-jupiter.version']='5.9.2'
I am using Cucumber 6.11.0.
My gradle test task is simply
test {
useJUnitPlatform()
finalizedBy(project.tasks.jacocoTestReport)
}
Is there a way to configure in my build the order in which the test runners get executed, in a consistent and reproduceable manner ?
Thanks
Gradle uses the JUnit Platform Launcher API to run the tests. You can do the same thing. And to ensure the order remains the same, you would invoke the platform twice within the same JVM.
This should allow you to reproduce your problem as Spring keeps the application running until the JVM exits.
import static org.junit.platform.engine.discovery.DiscoverySelectors.selectClass;
import my.project.spring.MySpringBootTest;
import my.project.cucumber.MyCucumberTest;
import org.junit.platform.launcher.Launcher;
import org.junit.platform.launcher.LauncherDiscoveryRequest;
import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder;
import org.junit.platform.launcher.core.LauncherFactory;
import org.junit.platform.launcher.listeners.SummaryGeneratingListener;
import org.junit.platform.launcher.listeners.TestExecutionSummary;
public class ManualLaunch {
public static void main(String[] args) {
var cucumberSummary= runCucumberTest();
var jupiterSummary= runJupiterTest();
System.out.println("Jupiter failures : "+jupiterSummary.getTestsFailedCount());
System.out.println("Cucumber failures : "+cucumberSummary.getTestsFailedCount());
}
private static TestExecutionSummary runCucumberTest() {
LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request()
// Configure the request for Cucumber here
.selectors(
selectClass(MyCucumberTest.class)
)
.build();
return launchAndGetSummary(request);
}
private static TestExecutionSummary runJupiterTest() {
LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request()
.selectors(
selectClass(MySpringBootTest.class)
)
.build();
return launchAndGetSummary(request);
}
private static TestExecutionSummary launchAndGetSummary(LauncherDiscoveryRequest request){
Launcher launcher = LauncherFactory.create();
SummaryGeneratingListener listener = new SummaryGeneratingListener();
launcher.registerTestExecutionListeners(listener);
launcher.execute(request);
return listener.getSummary();
}
}
If you don't know how to run a main method from Gradle, you could also use JUnit to run JUnit. But then you do have to make sure the tests don't target themselves.
Related
I'm trying to run Cucumber's feature files in parallel using Cucumber's CLI Runner and I'm currently stuck trying to figure out how to make JUnit #BeforeClass hook to work with the CLI Runner.
At the moment, my working Runner class looks like this:
#RunWith(Cucumber.class)
#CucumberOptions(
plugin = {
"pretty",
"html:target/reports/basic/report.html",
"json:target/reports/cluecumber/cucumber.json",
"timeline:target/reports/timeline"
},
tags = "#RegressionTests",
snippets = SnippetType.CAMELCASE,
stepNotifications = true,
features = "classpath:features",
glue = "my.steps.package")
public class RegressionTestsIT {
#BeforeClass
public static void setup() {
ContextHolder.setupTestContext();
}
}
And my CLI command looks like this:
java -cp "target/test-jar-with-dependencies.jar" io.cucumber.core.cli.Main -p "pretty" -p "html:target/reports/basic/report.html" -p "json:target/reports/cluecumber/cucumber.json" -p "timeline:target/reports/timeline" --threads 10 -g "my.steps.package" target/test-classes/features
What happens is that I get a NullPointerException at the tests because TestContext was not properly set up as the hook was not executed.
I tried to include both the Runner's package and the Runner class itself as glue and it didn't work.
Also tried to make my Runner extend io.cucumber.core.cli.Main and then execute my Runner in the CLI and not surprisingly it did not work either, sadly still getting NPE.
Although this issue is related to the CLI Runner use, I'm content with any answer that might help me run multiple feature files in parallel whatever the method.
Using JUnit Rules
Cucumber supports JUnit's #ClassRule, #BeforeClass, and #AfterClass annotations. These will be executed before and after all scenarios. Using these is not recommended as it limits portability between different runners; they may not execute correctly when using the command line, IntelliJ IDEA, or Cucumber-Eclipse. Instead it is recommended to use Cucumber's hooks.
When using the CLI, JUnit is not involved at all so you can not use any of JUnit annotations. However since Cucumber v7 you can use #BeforeAll and #AfterAll to declare methods that executed before and after all scenarios.
package io.cucumber.example;
import io.cucumber.java.AfterAll;
import io.cucumber.java.BeforeAll;
public class StepDefinitions {
#BeforeAll
public static void beforeAll() {
// Runs before all scenarios
}
#AfterAll
public static void afterAll() {
// Runs after all scenarios
}
}
JUnit #BeforeClass didn't work for me. Since I'm kinda in a hurry with this, I didn't bother keep on trying to make it work. I don't really need to run the command in a pipeline at the moment, so I was completely fine in running it on IntelliJ as long as it was running in parallel.
My solution was creating a custom CLI Runner that runs the context configuration before Cucumber's CLI run method.
public class CLIRunner {
public static void main(String[] args) {
ContextHolder.setupTestContext();
io.cucumber.core.cli.Main.run(
new String[] {
"-p", "pretty",
"-p", "html:target/reports/basic/report.html",
"-p", "json:target/reports/cluecumber/cucumber.json",
"-p", "timeline:target/reports/timeline",
"-g", "my.steps.package",
"classpath:features",
"--threads", "10"
}, Thread.currentThread().getContextClassLoader());
}
}
I'm automating my rest api and searching for a performance testing framework to use with my junit5. I came across zerocode tdd but it is not helping it's giving error and all the tests are falling under failure. My test method is proper and works when it invoked normally by junit jupiter. When I use zerocodeLoadRunner for the same thing it is not working.
import org.jsmart.zerocode.core.domain.LoadWith;
import org.jsmart.zerocode.core.domain.TestMapping;
import org.jsmart.zerocode.core.runner.parallel.ZeroCodeLoadRunner;
import org.junit.runner.RunWith;
#LoadWith("loadConfig.properties")
#TestMapping(testClass = MyTest.class, testMethod = "myMethod")
#RunWith(ZeroCodeLoadRunner.class)
public class LoadTest {
}
The error message I am getting is as follows.
2019-09-09 12:35:57,191 [main] ERROR org.jsmart.zerocode.core.runner.parallel.ZeroCodeLoadRunner - myPackage.LoadTest.myMethod Failed. See target/logs -or- junit granular failure report(csv) -or- fuzzy search and filter report(html) for details
The dependency used for this is as follows
<dependency>
<groupId>org.jsmart</groupId>
<artifactId>zerocode-tdd-jupiter</artifactId>
<version>1.3.8</version>
</dependency>
I don't want to use any testing tool so I'm going with this.
The setup which you have done works for JUnit4(but not for JUnit5 as JUnit5 does not support Runners). For JUnit5, you need to use the following extention of zerocode.
#ExtendWith({ParallelLoadExtension.class})
This JUnit5-Jupiter-Parallel-Load-Extension page has the precise details. Please try the below way:
e.g.
#LoadWith("load_generation.properties")
#ExtendWith({ParallelLoadExtension.class})
public class JUnit5LoadCommonLoadTest {
#Test
#DisplayName("testing parallel load for X, Y and Z scenarios")
#TestMappings({
#TestMapping(testClass = JUnit5Test.class, testMethod = "testX"),
#TestMapping(testClass = JUnit5Test.class, testMethod = "testY"),
#TestMapping(testClass = JUnit5MoreTest.class, testMethod = "testZ")
})
public void testLoad_xyz() {
// This space remains empty
}
}
You can visit this Github JUnit5 Load examples for JUnit5 Jupiter load tests and also how to run them via JUnit5 Suite.
I second the answer above the issue is you are setup using JUnit4 style tests and need to setup for JUnit5 - I have tested this as recently as the last few weeks and works exactly as specified above.
One item to note is that in the properties file I have found you need to set seconds 15-20% higher than threads (ie threads 20 / seconds 25). If you do not then I very commonly see failures in the zerocode test
I need to be able to restart currently executing cucumber test case or scenario so that if it is about to be marked as failed in the #After phase it will restart immediately. I have looked into:
Flaky Test handler Plugin
ExtendedCucumberOptions
TestRule (JUnit) - Retry Rule
but none of the above have worked for me, or simply did not do what I am seeking to achieve. The behavior I am looking for is as follows:
The test starts with cucumber runner class for example:
#RunWith(Cucumber.class)
#CucumberOptions(tags = "#MyTag"
, format = {"pretty", "html:target/MyTest/cucumber", "json:target/MyTest/cucumber.json"}
, glue = {"my.ui.reportsGlue"})
public class RunMyTest {
}
Cucumber Steps definitions where the retry logic should happen:
#Before("#MyTag")
public void setUp() {
// test set up methods
}
#After("#MyTag")
public void stop(Scenario scenario) {
if (scenario.isFailed() && retryCount <=nTimes) {
nTimes++;
// currently running scenario should restart from the beginning here
}
}
Any help would be greatly appreciated.
In my project there are 2 types of tests: simple unit tests and acceptance tests with UI automation. All acceptance tests are extending a common base class:
import org.junit.Test;
public class MyAcceptanceTest extends AbstractAcceptanceTest {
#Test
public void someTest() {...}
}
Is it possible with gradle to filter tests so that only the acceptance tests are executed?
My main goal is that with gradle build only the normal unit tests are executed (all test classes except those that extend from AbstractAcceptanceTest).
The second goal is to have a separate task that only executes the acceptance tests (only test classes that extend AbstractAcceptanceTest). However, it would also be ok to execute all tests with this task (no filtering).
As far as I've found in the docs and by googling, filtering is only possible by file name patterns. But maybe there is some groovy magic that allows filtering like this?
EDIT
I've found a solution that works but feels a little hacky:
I've added a #Category annotation from JUnit to the base class (AbstractAcceptanceTest). The category needs an identifier class for which I've created a class com.example.AcceptanceTest. In my gradle.build file I'm using this configuration:
task acceptanceTests (type: Test) {
test {
useJUnit {
includeCategories 'com.example.AcceptanceTest'
}
}
}
test {
useJUnit {
excludeCategories 'com.example.AcceptanceTest'
}
}
This way the acceptance tests aren't executed in normal build but only with gradle acceptanceTests.
When writing code that interacts with external resources (such as using a web service or other network operation), I often structure the classes so that it can also be "stubbed" using a file or some other input method. So then I end up using the stubbed implementation to test other parts of the system and then one or two tests that specifically test calling the web service.
The problem is I don't want to be calling these external services either from Jenkins or when I run all of the tests for my project (e.g. "gradle test"). Some of the services have side effects, or may not be accessible to all developers.
Right now I just uncomment and then re-comment the #Test annotation on these particular test methods to enable and disable them. Enable it, run it manually to check it, then remember to comment it out again.
// Uncomment to test external service manually
//#Test
public void testSomethingExternal() {
Is there is a better way of doing this?
EDIT: For manual unit testing, I use Eclipse and am able to just right-click on the test method and do Run As -> JUnit test. But that doesn't work without the (uncommented) annotation.
I recommend using junit categories. See this blog for details : https://community.oracle.com/blogs/johnsmart/2010/04/25/grouping-tests-using-junit-categories-0.
Basically, you can annotate some tests as being in a special category and then you can set up a two test suites : one that runs the tests of that category and one that ignores tests in that category (but runs everything else)
#Category(IntegrationTests.class)
public class AccountIntegrationTest {
#Test
public void thisTestWillTakeSomeTime() {
...
}
#Test
public void thisTestWillTakeEvenLonger() {
....
}
}
you can even annotate individual tests"
public class AccountTest {
#Test
#Category(IntegrationTests.class)
public void thisTestWillTakeSomeTime() {
...
}
Anytime I see something manually getting turned on or off I cringe.
As far as I can see you use gradle and API for JUnit says that annotation #Ignore disables test. I will add gradle task which will add #Ignore for those tests.
If you're just wanting to disable tests for functionality that hasn't been written yet or otherwise manually disable some tests temporarily, you can use #Ignore; the tests will be skipped but still noted in the report.
If you are wanting something like Spring Profiles, where you can define rulesets for which tests get run when, you should either split up your tests into separate test cases or use a Filter.
You can use #Ignore annotation to prevent them from running automatically during test. If required, you may trigger such Ignored tests manually.
#Test
public void wantedTest() {
return checkMyFunction(10);
}
#Ignore
#Test
public void unwantedTest() {
return checkMyFunction(11);
}
In the above example, unwantedTest will be excluded.