I have been experiencing something really weird. Maybe someone can explain me where I am making mistake.
I have a following scenario in a feature file
#DeleteUserAfterTest
Scenario: Testing a functionality
Given admin exists
When a user is created
Then the user is verified
My #After method in Hooks class looks like following
#After
public void tearDown(Scenario scenario) {
if (scenario.isFailed()) {
final byte[] screenshot = ((TakesScreenshot) driver)
.getScreenshotAs(OutputType.BYTES);
scenario.embed(screenshot, "image/png"); //stick it in the report
}
driver.quit();
}
I am using the following method in my step definition to delete the created user based on tag passed in the Test Scenario as follows:
#After("#DeleteUserAfterTest")
public void deleteUser(){
//Do fucntionalities to delete user
}
My test runner looks something like this:
import io.cucumber.testng.AbstractTestNGCucumberTests;
import io.cucumber.testng.CucumberOptions;
#CucumberOptions(
plugin = {"pretty","com.aventstack.extentreports.cucumber.adapter.ExtentCucumberAdapter:", "json:target/cucumber-report/TestResult.json"},
monochrome = false,
features = "src/test/resources/features/IntegrationScenarios.feature",
tags="#DeleteUserAfterTest",
glue="Steps")
public class IntegrationTest extends AbstractTestNGCucumberTests {
}
However, when I launch the test case, sometimes my user is deleted in the After("#DeleteUserAfterTest") but sometimes my test does not recognises the tagged After at all. It directly goes to After method in my Hooks class and quits the driver. Maybe someone has encountered this problem or knows a workaround!
Method order is not defined in Java. So you have to tell Cucumber in which order you hooks should be executed. Higher numbers are run first (before hooks are the other way around).
#After(order = 500)
public void tearDown(Scenario scenario) {
}
#After(value = "#DeleteUserAfterTest", order = 1000)
public void deleteUser(){
}
I have a maven java project in the latest Intellij IDEA with Testng plugin. I want to run specific groups in a test class without manually creating an xml config file for this.
Can I do that through the run configuration menu instead ? Unfortunately, the menu only lets me select groups, classes, methods etc. but not the groups inside a particular class.
Let's assume you have the following code sample:
import org.testng.annotations.*;
public class Test1 {
#Test(groups = { "functest", "checkintest" })
public void testMethod1() {
}
#Test(groups = {"functest", "checkintest"} )
public void testMethod2() {
}
#Test(groups = { "functest" })
public void testMethod3() {
}
}
To run only checkintest group go to Run Configuration, create new TestNG config -> select Test kind "Group" -> select checkintest group.
Only testMethod1, testMethod2 will be run.
Suppose I develop an extension which disallows test method names to start with an uppercase character.
public class DisallowUppercaseLetterAtBeginning implements BeforeEachCallback {
#Override
public void beforeEach(ExtensionContext context) {
char c = context.getRequiredTestMethod().getName().charAt(0);
if (Character.isUpperCase(c)) {
throw new RuntimeException("test method names should start with lowercase.");
}
}
}
Now I want to test that my extension works as expected.
#ExtendWith(DisallowUppercaseLetterAtBeginning.class)
class MyTest {
#Test
void validTest() {
}
#Test
void TestShouldNotBeCalled() {
fail("test should have failed before");
}
}
How can I write a test to verify that the attempt to execute the second method throws a RuntimeException with a specific message?
Another approach could be to use the facilities provided by the new JUnit 5 - Jupiter framework.
I put below the code which I tested with Java 1.8 on Eclipse Oxygen. The code suffers from a lack of elegance and conciseness but could hopefully serve as a basis to build a robust solution for your meta-testing use case.
Note that this is actually how JUnit 5 is tested, I refer you to the unit tests of the Jupiter engine on Github.
public final class DisallowUppercaseLetterAtBeginningTest {
#Test
void testIt() {
// Warning here: I checked the test container created below will
// execute on the same thread as used for this test. We should remain
// careful though, as the map used here is not thread-safe.
final Map<String, TestExecutionResult> events = new HashMap<>();
EngineExecutionListener listener = new EngineExecutionListener() {
#Override
public void executionFinished(TestDescriptor descriptor, TestExecutionResult result) {
if (descriptor.isTest()) {
events.put(descriptor.getDisplayName(), result);
}
// skip class and container reports
}
#Override
public void reportingEntryPublished(TestDescriptor testDescriptor, ReportEntry entry) {}
#Override
public void executionStarted(TestDescriptor testDescriptor) {}
#Override
public void executionSkipped(TestDescriptor testDescriptor, String reason) {}
#Override
public void dynamicTestRegistered(TestDescriptor testDescriptor) {}
};
// Build our test container and use Jupiter fluent API to launch our test. The following static imports are assumed:
//
// import static org.junit.platform.engine.discovery.DiscoverySelectors.selectClass
// import static org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder.request
JupiterTestEngine engine = new JupiterTestEngine();
LauncherDiscoveryRequest request = request().selectors(selectClass(MyTest.class)).build();
TestDescriptor td = engine.discover(request, UniqueId.forEngine(engine.getId()));
engine.execute(new ExecutionRequest(td, listener, request.getConfigurationParameters()));
// Bunch of verbose assertions, should be refactored and simplified in real code.
assertEquals(new HashSet<>(asList("validTest()", "TestShouldNotBeCalled()")), events.keySet());
assertEquals(Status.SUCCESSFUL, events.get("validTest()").getStatus());
assertEquals(Status.FAILED, events.get("TestShouldNotBeCalled()").getStatus());
Throwable t = events.get("TestShouldNotBeCalled()").getThrowable().get();
assertEquals(RuntimeException.class, t.getClass());
assertEquals("test method names should start with lowercase.", t.getMessage());
}
Though a little verbose, one advantage of this approach is it doesn't require mocking and execute the tests in the same JUnit container as will be used later for real unit tests.
With a bit of clean-up, a much more readable code is achievable. Again, JUnit-Jupiter sources can be a great source of inspiration.
If the extension throws an exception then there's not much a #Test method can do since the test runner will never reach the #Test method. In this case, I think, you have to test the extension outside of its use in the normal test flow i.e. let the extension be the SUT.
For the extension provided in your question, the test might be something like this:
#Test
public void willRejectATestMethodHavingANameStartingWithAnUpperCaseLetter() throws NoSuchMethodException {
ExtensionContext extensionContext = Mockito.mock(ExtensionContext.class);
Method method = Testable.class.getMethod("MethodNameStartingWithUpperCase");
Mockito.when(extensionContext.getRequiredTestMethod()).thenReturn(method);
DisallowUppercaseLetterAtBeginning sut = new DisallowUppercaseLetterAtBeginning();
RuntimeException actual =
assertThrows(RuntimeException.class, () -> sut.beforeEach(extensionContext));
assertThat(actual.getMessage(), is("test method names should start with lowercase."));
}
#Test
public void willAllowTestMethodHavingANameStartingWithAnLowerCaseLetter() throws NoSuchMethodException {
ExtensionContext extensionContext = Mockito.mock(ExtensionContext.class);
Method method = Testable.class.getMethod("methodNameStartingWithLowerCase");
Mockito.when(extensionContext.getRequiredTestMethod()).thenReturn(method);
DisallowUppercaseLetterAtBeginning sut = new DisallowUppercaseLetterAtBeginning();
sut.beforeEach(extensionContext);
// no exception - good enough
}
public class Testable {
public void MethodNameStartingWithUpperCase() {
}
public void methodNameStartingWithLowerCase() {
}
}
However, your question suggests that the above extension is only an example so, more generally; if your extension has a side effect (e.g. sets something in an addressable context, populates a System property etc) then your #Test method could assert that this side effect is present. For example:
public class SystemPropertyExtension implements BeforeEachCallback {
#Override
public void beforeEach(ExtensionContext context) {
System.setProperty("foo", "bar");
}
}
#ExtendWith(SystemPropertyExtension.class)
public class SystemPropertyExtensionTest {
#Test
public void willSetTheSystemProperty() {
assertThat(System.getProperty("foo"), is("bar"));
}
}
This approach has the benefit of side stepping the potentially awkward setup steps of: creating the ExtensionContext and populating it with the state required by your test but it may come at the cost of limiting the test coverage since you can really only test one outcome. And, of course, it is only feasible if the extension has a side effect which can be evaulated in a test case which uses the extension.
So, in practice, I suspect you might need a combination of these approaches; for some extensions the extension can be the SUT and for others the extension can be tested by asserting against its side effect(s).
After trying the solutions in the answers and the question linked in the comments, I ended up with a solution using the JUnit Platform Launcher.
class DisallowUppercaseLetterAtBeginningTest {
#Test
void should_succeed_if_method_name_starts_with_lower_case() {
TestExecutionSummary summary = runTestMethod(MyTest.class, "validTest");
assertThat(summary.getTestsSucceededCount()).isEqualTo(1);
}
#Test
void should_fail_if_method_name_starts_with_upper_case() {
TestExecutionSummary summary = runTestMethod(MyTest.class, "InvalidTest");
assertThat(summary.getTestsFailedCount()).isEqualTo(1);
assertThat(summary.getFailures().get(0).getException())
.isInstanceOf(RuntimeException.class)
.hasMessage("test method names should start with lowercase.");
}
private TestExecutionSummary runTestMethod(Class<?> testClass, String methodName) {
SummaryGeneratingListener listener = new SummaryGeneratingListener();
LauncherDiscoveryRequest request = request().selectors(selectMethod(testClass, methodName)).build();
LauncherFactory.create().execute(request, listener);
return listener.getSummary();
}
#ExtendWith(DisallowUppercaseLetterAtBeginning.class)
static class MyTest {
#Test
void validTest() {
}
#Test
void InvalidTest() {
fail("test should have failed before");
}
}
}
JUnit itself will not run MyTest because it is an inner class without #Nested. So there are no failing tests during the build process.
Update
JUnit itself will not run MyTest because it is an inner class without #Nested. So there are no failing tests during the build process.
This is not completly correct. JUnit itself would also run MyTest, e.g. if "Run All Tests" is started within the IDE or within a Gradle build.
The reason why MyTest was not executed is because I used Maven and I tested it with mvn test. Maven uses the Maven Surefire Plugin to execute tests. This plugin has a default configuration which excludes all nested classes like MyTest.
See also this answer about "Run tests from inner classes via Maven" and the linked issues in the comments.
JUnit 5.4 introduced the JUnit Platform Test Kit which allows you to execute a test plan and inspect the results.
To take a dependency on it from Gradle, it might look something like this:
testImplementation("org.junit.platform:junit-platform-testkit:1.4.0")
And using your example, your extension test could look something like this:
import org.junit.jupiter.api.extension.ExtendWith
import org.junit.jupiter.api.fail
import org.junit.platform.engine.discovery.DiscoverySelectors
import org.junit.platform.testkit.engine.EngineTestKit
import org.junit.platform.testkit.engine.EventConditions
import org.junit.platform.testkit.engine.TestExecutionResultConditions
internal class DisallowUpperCaseExtensionTest {
#Test
internal fun `succeed if starts with lower case`() {
val results = EngineTestKit
.engine("junit-jupiter")
.selectors(
DiscoverySelectors.selectMethod(ExampleTest::class.java, "validTest")
)
.execute()
results.tests().assertStatistics { stats ->
stats.finished(1)
}
}
#Test
internal fun `fail if starts with upper case`() {
val results = EngineTestKit
.engine("junit-jupiter")
.selectors(
DiscoverySelectors.selectMethod(ExampleTest::class.java, "TestShouldNotBeCalled")
)
.execute()
results.tests().assertThatEvents()
.haveExactly(
1,
EventConditions.finishedWithFailure(
TestExecutionResultConditions.instanceOf(java.lang.RuntimeException::class.java),
TestExecutionResultConditions.message("test method names should start with lowercase.")
)
)
}
#ExtendWith(DisallowUppercaseLetterAtBeginning::class)
internal class ExampleTest {
#Test
fun validTest() {
}
#Test
fun TestShouldNotBeCalled() {
fail("test should have failed before")
}
}
}
I have viewed the following similar questions and they are not applicable:
Cucumber - implementing missing steps in java
cucumber not able to recognize the implemented methods
also viewed https://github.com/cucumber/cucumber/issues but no hits
I am running Java 1.8 with cucumber, when I hover over the steps in .feature file i can see there implementation but when i run the test i get:
You can implement missing steps with the snippets below:
#Given("^I am on the Login Page$")
public void i_am_on_the_Login_Page() throws Throwable {
// Write code here that turns the phrase above into concrete actions
throw new PendingException();
}
#When("^I enter the non existent email$")
public void i_enter_the_non_existent_email() throws Throwable {
// Write code here that turns the phrase above into concrete actions
throw new PendingException();
}
#When("^I enter the false password$")
public void i_enter_the_false_password() throws Throwable {
// Write code here that turns the phrase above into concrete actions
throw new PendingException();
}
#When("^I click on it$")
public void i_click_on_it() throws Throwable {
// Write code here that turns the phrase above into concrete actions
throw new PendingException();
}
#Then("^I still on the page$")
public void i_still_on_the_page() throws Throwable {
// Write code here that turns the phrase above into concrete actions
throw new PendingException();
}
I thought may be i did some thing wrong in project configurations but when I also imported the cucumber-java-skeleton from https://github.com/cucumber/cucumber-java-skeleton
when I populate the skelton project with my own implementation as below i get the above same error when i run the test but i still can see the implementation when i hover over steps in .feature file:
#Given("^I am on the Login Page$")
public void i_am_on_the_Login_Page() throws Throwable {
// Write code here that turns the phrase above into concrete actions
}
#When("^I enter the non existent email$")
public void i_enter_the_non_existent_email() throws Throwable {
// Write code here that turns the phrase above into concrete actions
}
#When("^I enter the false password$")
public void i_enter_the_false_password() throws Throwable {
// Write code here that turns the phrase above into concrete actions
}
#When("^I click on it$")
public void i_click_on_it() throws Throwable {
// Write code here that turns the phrase above into concrete actions
}
#Then("^I still on the page$")
public void i_still_on_the_bb() throws Throwable {
// Write code here that turns the phrase above into concrete actions
}
.feature file:
Feature: Login
Scenario: Testing with false credentials
Given I am on the Login Page
When I enter the non existent email
And I enter the false password
And I click on it
Then I still on the page
I am using Netbeans latest version with the same pom.xml file as the skelton project.
I have used tidy Gherkin to get the Gherkin translate into Java code
is there any other way to get Gherkin translated into Java specially Java 8 other than Tidy gherkin?? maybe that is the issue??
Thank you for you help in advance
I am away from my computer so I don't have a way to check this, but have your Gherkin match you method names exactly.
I'm having difficult pointing to my stepDefinitions. It exists, it has stuff in it like:
#Given("^Mark \"(.*)\" flew a kite$")
public void mark_flew_a_kite(String kiteName) throws Throwable {
kiteName = kiteName;
}
#When("^The wind blows (\\d+)st (\\d+)$")
public void the_wind_blows() throws Throwable {
json = callKiteService(200);
}
Please note I've changed all the actual method names for stackoverflow and inadvertantly made some of the regex not match. But it matches in the actual code!
But the test output says that none of the test steps were found and that I need to write them all.
Feature: Mark flies kite
Scenario: Mark flew a kite on a windy day [90m# WebServiceTest.feature:3[0m
[33mGiven [0m[33mMark flew a kite[0m
[33mWhen [0m[33mMark flies a kite on Dec 1st 2015[0m
[33mThen [0m[33mThe kite should fly up[0m
1 Scenarios ([33m1 undefined[0m)
3 Steps ([33m3 undefined[0m)
0m0.000s
You can implement missing steps with the snippets below:
#Given("^Mark flew a kite$")
public void mark_flew_a_kite(String arg1, int arg2, int arg3, int arg4) throws Throwable {
// Write code here that turns the phrase above into concrete actions
throw new PendingException();
}
#When("^The wind blows (\\d+)st (\\d+)$")
public void the_wind_blows(int arg1, int arg2) throws Throwable {
// Write code here that turns the phrase above into concrete actions
throw new PendingException();
}
#Then("^The kite should fly up$")
public void kite_should_fly(DataTable arg1) throws Throwable {
// Write code here that turns the phrase above into concrete actions
// For automatic transformation, change DataTable to one of
// List<YourType>, List<List<E>>, List<Map<K,V>> or Map<K,V>.
// E,K,V must be a scalar (String, Integer, Date, enum etc)
throw new PendingException();
}
I just ended up copying the files into a new cucumber java skeleton and using that instead of the old project. I couldn't find any differences between the two projects other than that the skeleton worked and I have no idea where it was configured that my features had to be located in src/main/resources/Features
I did a search for feature and found nothing.
Link to skeleton