We Keep Coding

How do I test code calling static factory methods with or without Mocks?

I have a dilemma trying to solve the following test. The class is a toggle created through the Togglz library. I'm catching the feature manager method execution because I'm using a JDBCStateReporsitory to read the toggle value and if something goes wrong with the DB I have to be able to return the default value of the toggle with the #EnabledByDefault annotation.
#Slf4j
public enum PocToggle {
#EnabledByDefault
USE_MY_FEATURE;
public boolean isActive() {
FeatureManager featureManager = FeatureContext.getFeatureManager();
try {
return featureManager.isActive(this);
} catch (RuntimeException ignored) {
if (log.isWarnEnabled()) {
log.warn(String.format("Failed to retrieve feature '%s' state", this.name()));
}
FeatureMetaData metaData = featureManager.getMetaData(this);
FeatureState featureState = metaData.getDefaultFeatureState();
return featureState.isEnabled();
}
}
}
I have no clue of how to do it because an object created by an inner utility static method doesn't allow me to Stub or Mock it. I just created the true and false paths of the test, but the test trying to cover the exception path is not working throwing me a Expected exception of type 'java.lang.IllegalStateException', but no exception was thrown message.
class PocToggleSpecification extends Specification {
#Rule
private TogglzRule toggleRule = TogglzRule.allEnabled(PocToggle.class)
def "Should toggle to use my feature when it is enabled"() {
when:
toggleRule.enable(USE_MY_FEATURE)
then:
USE_MY_FEATURE.isActive()
}
def "Should toggle to not to use my feature when it is disabled"() {
when:
toggleRule.disable(USE_MY_FEATURE)
then:
!USE_MY_FEATURE.isActive()
}
def "Should throw an exception when something goes wrong"() {
given:
toggleRule.enable(USE_MY_FEATURE)
FeatureManager featureManager = Stub()
featureManager.isActive() >> { throw new IllegalStateException() }
def featureContext = Spy(FeatureContext)
featureContext.getFeatureManager() >> featureManager
when:
USE_MY_FEATURE.isActive()
then:
thrown IllegalStateException
}
}
Could you please help me to solve this kind of test?

Well, first of all your test would not run as expected because your toggle class catches runtime exceptions and IllegalStateException is a runtime exception, so it will never be thrown.
Secondly, Spock cannot mock static methods for Java classes, only for Groovy classes.
So if you do not want to fiddle around with PowerMock inside of Spock - mocking static methods is always a bad smell - you still have the option of making your toggle class more testable by making the feature manager injectable via a package-scoped setter method and then use that method from the test. Try this example:
package de.scrum_master.stackoverflow;
import org.togglz.core.Feature;
import org.togglz.core.annotation.EnabledByDefault;
import org.togglz.core.context.FeatureContext;
import org.togglz.core.manager.FeatureManager;
import org.togglz.core.metadata.FeatureMetaData;
import org.togglz.core.repository.FeatureState;
public enum PocToggle implements Feature {
#EnabledByDefault
USE_MY_FEATURE;
private FeatureManager customFeatureManager;
void setFeatureManager(FeatureManager featureManager) {
this.customFeatureManager = featureManager;
}
public boolean isActive() {
FeatureManager featureManager = customFeatureManager != null
? customFeatureManager
: FeatureContext.getFeatureManager();
try {
return featureManager.isActive(this);
} catch (RuntimeException ignored) {
System.err.println(String.format("Failed to retrieve feature '%s' state", this.name()));
FeatureMetaData metaData = featureManager.getMetaData(this);
FeatureState featureState = metaData.getDefaultFeatureState();
return featureState.isEnabled();
}
}
}
package de.scrum_master.stackoverflow
import org.junit.Rule
import org.togglz.junit.TogglzRule
import org.togglz.testing.TestFeatureManager
import spock.lang.Specification
import static PocToggle.USE_MY_FEATURE
class PocToggleTest extends Specification {
#Rule
TogglzRule toggleRule = TogglzRule.allEnabled(PocToggle.class)
def "Feature is active when enabled"() {
when:
toggleRule.enable(USE_MY_FEATURE)
then:
USE_MY_FEATURE.isActive()
}
def "Feature is inactive when disabled"() {
when:
toggleRule.disable(USE_MY_FEATURE)
then:
!USE_MY_FEATURE.isActive()
}
def "Feature defaults to active upon feature manager error"() {
setup: "inject error-throwing feature manager into Togglz rule"
def featureManagerSpy = Spy(TestFeatureManager, constructorArgs: [PocToggle]) {
isActive(_) >> { throw new IllegalStateException() }
}
when: "feature is disabled and feature manager throws an error"
toggleRule.disable(USE_MY_FEATURE)
USE_MY_FEATURE.featureManager = featureManagerSpy
then: "feature is reported to be active by default"
USE_MY_FEATURE.isActive()
cleanup: "reset Togglz rule feature manager"
USE_MY_FEATURE.featureManager = null
}
}
Running the last test, you will see the log message Failed to retrieve feature 'USE_MY_FEATURE' state as expected. My test coverage tool also shows that it works:
Update 2018-01-17: solution variant using PowerMock (tested with 1.6.6 and 1.7.3)
Okay, I needed PowerMock for another reason and quickly gave your code a spin.
Disclaimer: I do prefer the first solution above, i.e. refactoring with regard to dependency injection instead of dirty tricks with PowerMock, but for what it is worth, here is how to do it.
The Java class has the dependency injection removed again, should be identical to the OP's original code.
package de.scrum_master.stackoverflow;
import org.togglz.core.Feature;
import org.togglz.core.annotation.EnabledByDefault;
import org.togglz.core.context.FeatureContext;
import org.togglz.core.manager.FeatureManager;
import org.togglz.core.metadata.FeatureMetaData;
import org.togglz.core.repository.FeatureState;
public enum PocToggle implements Feature {
#EnabledByDefault
USE_MY_FEATURE;
public boolean isActive() {
FeatureManager featureManager = FeatureContext.getFeatureManager();
try {
return featureManager.isActive(this);
} catch (RuntimeException ignored) {
System.err.println(String.format("Failed to retrieve feature '%s' state", this.name()));
FeatureMetaData metaData = featureManager.getMetaData(this);
FeatureState featureState = metaData.getDefaultFeatureState();
return featureState.isEnabled();
}
}
}
See PowerMock wiki for an explanation. BTW, Sputnik is Spock's JUnit runner.
package de.scrum_master.stackoverflow
import org.junit.Rule
import org.junit.runner.RunWith
import org.powermock.core.classloader.annotations.PrepareForTest
import org.powermock.modules.junit4.PowerMockRunner
import org.powermock.modules.junit4.PowerMockRunnerDelegate
import org.spockframework.runtime.Sputnik
import org.togglz.core.context.FeatureContext
import org.togglz.junit.TogglzRule
import org.togglz.testing.TestFeatureManager
import spock.lang.Specification
import static PocToggle.USE_MY_FEATURE
import static org.powermock.api.mockito.PowerMockito.mockStatic
import static org.powermock.api.mockito.PowerMockito.when
#RunWith(PowerMockRunner.class)
#PowerMockRunnerDelegate(Sputnik.class)
#PrepareForTest([FeatureContext.class])
class PocToggleTest extends Specification {
#Rule
TogglzRule toggleRule = TogglzRule.allEnabled(PocToggle.class)
// ...
def "Feature defaults to active upon feature manager error (power-mocked)"() {
setup: "inject error-throwing feature manager into Togglz rule"
def featureManagerSpy = Spy(TestFeatureManager, constructorArgs: [PocToggle]) {
isActive(_) >> { throw new IllegalStateException() }
}
mockStatic(FeatureContext)
when(FeatureContext.getFeatureManager()).thenReturn(featureManagerSpy)
when: "feature is disabled and feature manager throws an error"
toggleRule.disable(USE_MY_FEATURE)
then: "feature is reported to be active by default"
USE_MY_FEATURE.isActive()
}
}
Update 2 (2018-01-19): I want to mention one more issue: Maybe you noticed that I use a Spy instead of a Stub or Mock. This is because your code catches an exception thrown by the feature manager (FM), but then in the catch block uses the FM again. This can be dangerous because if the FM is broken due to network or database failure, it might also fail upon your next call. So while the test gives you 100% code coverage, it does not guarantee that your application behaves as expected in production or that you are testing the right thing.

I think that the reason is that since the method FeatureContext.getFeatureManager() is static, you just can't place a spy, if you'll put a breakpoint at the line of creation of FeatureManager stub inside the test and check out in debugger the address of the created object, and then place a breakpoint in a line inside the test that tries to use this feature manager, you'll see that these are two different objects, so the expectation to throw an exception is not applicable here.
In terms of a solution:
I suggest getting rid of static code as much as you can since it's not really unit testable. For example, you could create an interface like this (ok you use enum here, but it could be easily refactored to a class used from within an enum so that you could test it):
public interface FeatureManagerProvider {
FeatureManager getFeatureManager();
}
public class DefaultFeatureManagerProviderImpl implements FeatureManagerProvider {
.... // inject this in real use cases
}
With this abstraction, the code could be refactored like this:
public class Activator {
private FeatureManagerProvider featureManagerProvider;
public Activator(FeatureManagerProvider provider) {
this.featureManagerProvider = provider;
}
public boolean isActive() {
...
FeatureManager fm = featureManagerProvider.getFeatureManager();
...
}
}
During the test, you could supply a stub for FeatureManagerProvider and check all the interactions.

Check that JUnit Extension throws specific Exception

Suppose I develop an extension which disallows test method names to start with an uppercase character.
public class DisallowUppercaseLetterAtBeginning implements BeforeEachCallback {
#Override
public void beforeEach(ExtensionContext context) {
char c = context.getRequiredTestMethod().getName().charAt(0);
if (Character.isUpperCase(c)) {
throw new RuntimeException("test method names should start with lowercase.");
}
}
}
Now I want to test that my extension works as expected.
#ExtendWith(DisallowUppercaseLetterAtBeginning.class)
class MyTest {
#Test
void validTest() {
}
#Test
void TestShouldNotBeCalled() {
fail("test should have failed before");
}
}
How can I write a test to verify that the attempt to execute the second method throws a RuntimeException with a specific message?

Another approach could be to use the facilities provided by the new JUnit 5 - Jupiter framework.
I put below the code which I tested with Java 1.8 on Eclipse Oxygen. The code suffers from a lack of elegance and conciseness but could hopefully serve as a basis to build a robust solution for your meta-testing use case.
Note that this is actually how JUnit 5 is tested, I refer you to the unit tests of the Jupiter engine on Github.
public final class DisallowUppercaseLetterAtBeginningTest {
#Test
void testIt() {
// Warning here: I checked the test container created below will
// execute on the same thread as used for this test. We should remain
// careful though, as the map used here is not thread-safe.
final Map<String, TestExecutionResult> events = new HashMap<>();
EngineExecutionListener listener = new EngineExecutionListener() {
#Override
public void executionFinished(TestDescriptor descriptor, TestExecutionResult result) {
if (descriptor.isTest()) {
events.put(descriptor.getDisplayName(), result);
}
// skip class and container reports
}
#Override
public void reportingEntryPublished(TestDescriptor testDescriptor, ReportEntry entry) {}
#Override
public void executionStarted(TestDescriptor testDescriptor) {}
#Override
public void executionSkipped(TestDescriptor testDescriptor, String reason) {}
#Override
public void dynamicTestRegistered(TestDescriptor testDescriptor) {}
};
// Build our test container and use Jupiter fluent API to launch our test. The following static imports are assumed:
//
// import static org.junit.platform.engine.discovery.DiscoverySelectors.selectClass
// import static org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder.request
JupiterTestEngine engine = new JupiterTestEngine();
LauncherDiscoveryRequest request = request().selectors(selectClass(MyTest.class)).build();
TestDescriptor td = engine.discover(request, UniqueId.forEngine(engine.getId()));
engine.execute(new ExecutionRequest(td, listener, request.getConfigurationParameters()));
// Bunch of verbose assertions, should be refactored and simplified in real code.
assertEquals(new HashSet<>(asList("validTest()", "TestShouldNotBeCalled()")), events.keySet());
assertEquals(Status.SUCCESSFUL, events.get("validTest()").getStatus());
assertEquals(Status.FAILED, events.get("TestShouldNotBeCalled()").getStatus());
Throwable t = events.get("TestShouldNotBeCalled()").getThrowable().get();
assertEquals(RuntimeException.class, t.getClass());
assertEquals("test method names should start with lowercase.", t.getMessage());
}
Though a little verbose, one advantage of this approach is it doesn't require mocking and execute the tests in the same JUnit container as will be used later for real unit tests.
With a bit of clean-up, a much more readable code is achievable. Again, JUnit-Jupiter sources can be a great source of inspiration.

If the extension throws an exception then there's not much a #Test method can do since the test runner will never reach the #Test method. In this case, I think, you have to test the extension outside of its use in the normal test flow i.e. let the extension be the SUT.
For the extension provided in your question, the test might be something like this:
#Test
public void willRejectATestMethodHavingANameStartingWithAnUpperCaseLetter() throws NoSuchMethodException {
ExtensionContext extensionContext = Mockito.mock(ExtensionContext.class);
Method method = Testable.class.getMethod("MethodNameStartingWithUpperCase");
Mockito.when(extensionContext.getRequiredTestMethod()).thenReturn(method);
DisallowUppercaseLetterAtBeginning sut = new DisallowUppercaseLetterAtBeginning();
RuntimeException actual =
assertThrows(RuntimeException.class, () -> sut.beforeEach(extensionContext));
assertThat(actual.getMessage(), is("test method names should start with lowercase."));
}
#Test
public void willAllowTestMethodHavingANameStartingWithAnLowerCaseLetter() throws NoSuchMethodException {
ExtensionContext extensionContext = Mockito.mock(ExtensionContext.class);
Method method = Testable.class.getMethod("methodNameStartingWithLowerCase");
Mockito.when(extensionContext.getRequiredTestMethod()).thenReturn(method);
DisallowUppercaseLetterAtBeginning sut = new DisallowUppercaseLetterAtBeginning();
sut.beforeEach(extensionContext);
// no exception - good enough
}
public class Testable {
public void MethodNameStartingWithUpperCase() {
}
public void methodNameStartingWithLowerCase() {
}
}
However, your question suggests that the above extension is only an example so, more generally; if your extension has a side effect (e.g. sets something in an addressable context, populates a System property etc) then your #Test method could assert that this side effect is present. For example:
public class SystemPropertyExtension implements BeforeEachCallback {
#Override
public void beforeEach(ExtensionContext context) {
System.setProperty("foo", "bar");
}
}
#ExtendWith(SystemPropertyExtension.class)
public class SystemPropertyExtensionTest {
#Test
public void willSetTheSystemProperty() {
assertThat(System.getProperty("foo"), is("bar"));
}
}
This approach has the benefit of side stepping the potentially awkward setup steps of: creating the ExtensionContext and populating it with the state required by your test but it may come at the cost of limiting the test coverage since you can really only test one outcome. And, of course, it is only feasible if the extension has a side effect which can be evaulated in a test case which uses the extension.
So, in practice, I suspect you might need a combination of these approaches; for some extensions the extension can be the SUT and for others the extension can be tested by asserting against its side effect(s).

After trying the solutions in the answers and the question linked in the comments, I ended up with a solution using the JUnit Platform Launcher.
class DisallowUppercaseLetterAtBeginningTest {
#Test
void should_succeed_if_method_name_starts_with_lower_case() {
TestExecutionSummary summary = runTestMethod(MyTest.class, "validTest");
assertThat(summary.getTestsSucceededCount()).isEqualTo(1);
}
#Test
void should_fail_if_method_name_starts_with_upper_case() {
TestExecutionSummary summary = runTestMethod(MyTest.class, "InvalidTest");
assertThat(summary.getTestsFailedCount()).isEqualTo(1);
assertThat(summary.getFailures().get(0).getException())
.isInstanceOf(RuntimeException.class)
.hasMessage("test method names should start with lowercase.");
}
private TestExecutionSummary runTestMethod(Class<?> testClass, String methodName) {
SummaryGeneratingListener listener = new SummaryGeneratingListener();
LauncherDiscoveryRequest request = request().selectors(selectMethod(testClass, methodName)).build();
LauncherFactory.create().execute(request, listener);
return listener.getSummary();
}
#ExtendWith(DisallowUppercaseLetterAtBeginning.class)
static class MyTest {
#Test
void validTest() {
}
#Test
void InvalidTest() {
fail("test should have failed before");
}
}
}
JUnit itself will not run MyTest because it is an inner class without #Nested. So there are no failing tests during the build process.
Update
JUnit itself will not run MyTest because it is an inner class without #Nested. So there are no failing tests during the build process.
This is not completly correct. JUnit itself would also run MyTest, e.g. if "Run All Tests" is started within the IDE or within a Gradle build.
The reason why MyTest was not executed is because I used Maven and I tested it with mvn test. Maven uses the Maven Surefire Plugin to execute tests. This plugin has a default configuration which excludes all nested classes like MyTest.
See also this answer about "Run tests from inner classes via Maven" and the linked issues in the comments.

JUnit 5.4 introduced the JUnit Platform Test Kit which allows you to execute a test plan and inspect the results.
To take a dependency on it from Gradle, it might look something like this:
testImplementation("org.junit.platform:junit-platform-testkit:1.4.0")
And using your example, your extension test could look something like this:
import org.junit.jupiter.api.extension.ExtendWith
import org.junit.jupiter.api.fail
import org.junit.platform.engine.discovery.DiscoverySelectors
import org.junit.platform.testkit.engine.EngineTestKit
import org.junit.platform.testkit.engine.EventConditions
import org.junit.platform.testkit.engine.TestExecutionResultConditions
internal class DisallowUpperCaseExtensionTest {
#Test
internal fun `succeed if starts with lower case`() {
val results = EngineTestKit
.engine("junit-jupiter")
.selectors(
DiscoverySelectors.selectMethod(ExampleTest::class.java, "validTest")
)
.execute()
results.tests().assertStatistics { stats ->
stats.finished(1)
}
}
#Test
internal fun `fail if starts with upper case`() {
val results = EngineTestKit
.engine("junit-jupiter")
.selectors(
DiscoverySelectors.selectMethod(ExampleTest::class.java, "TestShouldNotBeCalled")
)
.execute()
results.tests().assertThatEvents()
.haveExactly(
1,
EventConditions.finishedWithFailure(
TestExecutionResultConditions.instanceOf(java.lang.RuntimeException::class.java),
TestExecutionResultConditions.message("test method names should start with lowercase.")
)
)
}
#ExtendWith(DisallowUppercaseLetterAtBeginning::class)
internal class ExampleTest {
#Test
fun validTest() {
}
#Test
fun TestShouldNotBeCalled() {
fail("test should have failed before")
}
}
}

JUnit5 - How to get test result in AfterTestExecutionCallback

I write JUnit5 Extension. But I cannot find way how to obtain test result.
Extension looks like this:
import org.junit.jupiter.api.extension.AfterTestExecutionCallback;
import org.junit.jupiter.api.extension.TestExtensionContext;
public class TestResultExtension implements AfterTestExecutionCallback {
#Override
public void afterTestExecution(TestExtensionContext context) throws Exception {
//How to get test result? SUCCESS/FAILED
}
}
Any hints how to obtain test result?

This work for me:
public class RunnerExtension implements AfterTestExecutionCallback {
#Override
public void afterTestExecution(ExtensionContext context) throws Exception {
Boolean testResult = context.getExecutionException().isPresent();
System.out.println(testResult); //false - SUCCESS, true - FAILED
}
}
#ExtendWith(RunnerExtension.class)
public abstract class Tests {
}

As other answers point out, JUnit communicates failed tests with exceptions, so an AfterTestExecutionCallback can be used to gleam what happened. Note that this is error prone as extension running later might still fail the test.
Another way to do that is to register a custom TestExecutionListener. Both of these approaches are a little roundabout, though. There is an issue that tracks a specific extension point for reacting to test results, which would likely be the most straight-forward answer to your question. If you can provide a specific use case, it would be great if you could head over to #542 and leave a comment describing it.

You can use SummaryGeneratingListener from org.junit.platform.launcher.listeners
It contains MutableTestExecutionSummary field, which implements TestExecutionSummary interface, and this way you can obtain info about containers, tests, time, failures etc.
You can create custom listener, for example:
Create class that extends SummaryGeneratingListener
public class ResultAnalyzer extends SummaryGeneratingListener {
#Override
public void testPlanExecutionFinished(TestPlan testPlan) {
//This method is invoked after all tests in all containers is finished
super.testPlanExecutionFinished(testPlan);
analyzeResult();
}
private void analyzeResult() {
var summary = getSummary();
var failures = summary.getFailures();
//Do something
}
}
Register listener by creating file
src\main\resources\META-INF\services\org.junit.platform.launcher.TestExecutionListener
and specify your implementation in it
path.to.class.ResultAnalyzer
Enable auto-detection of extensions, set parameter
-Djunit.jupiter.extensions.autodetection.enabled=true
And that's it!
Docs
https://junit.org/junit5/docs/5.0.0/api/org/junit/platform/launcher/listeners/SummaryGeneratingListener.html
https://junit.org/junit5/docs/5.0.0/api/org/junit/platform/launcher/listeners/TestExecutionSummary.html
https://junit.org/junit5/docs/current/user-guide/#extensions-registration-automatic

I have only this solution:
String testResult = context.getTestException().isPresent() ? "FAILED" : "OK";
It seems that it works well. But I am not sure if it will work correctly in all situations.

Fails in JUnit are propagated with exceptions. There are several exceptions, which indicate various types of errors.
So an exception in TestExtensionContext#getTestException() indicates an error. The method can't manipulate actual test results, so depending on your use case you might want to implement TestExecutionExceptionHandler, which allows you to swallow exceptions, thus changing whether a test succeeded or not.

You're almost there.
To implement a test execution callback and get the test result for logging (or generating a report) you can do the following:
import org.junit.jupiter.api.extension.AfterTestExecutionCallback;
import org.junit.jupiter.api.extension.ExtensionContext;
public class TestResultExtension implements AfterTestExecutionCallback
{
#Override
public void afterTestExecution(ExtensionContext context) throws Exception
{
// check the context for an exception
Boolean passed = context.getExecutionException().isEmpty();
// if there isn't, the test passed
String result = passed ? "PASSED" : "FAILED";
// now that you have the result, you can do whatever you want
System.out.println("Test Result: " + context.getDisplayName() + " " + result);
}
}
And then you just add the TestResultExtension using the #ExtendWith() annotation for your test cases:
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import static org.junit.jupiter.api.Assertions.assertTrue;
#ExtendWith(TestResultExtension.class)
public class SanityTest
{
#Test
public void testSanity()
{
assertTrue(true);
}
#Test
public void testInsanity()
{
assertTrue(false);
}
}
It's a good idea to extend a base test that includes the extension
import org.junit.jupiter.api.extension.ExtendWith;
#ExtendWith(TestResultExtension.class)
public class BaseTest
{}
And then you don't need to include the annotation in every test:
public class SanityTest extends BaseTest
{ //... }

How to get the exception that was thrown when a Cucumber test failed in Java?

I can perform actions on test failure by using:
#After
public void afterTest(Scenario scenario) {
if (scenario.isFailed()) {
/*Do stuff*/
}
}
However some of the actions I need to perform depend on the Exception that was thrown and in what context it was thrown. Is there a way to get the Throwable that caused the test to fail? For example in JUnit I would do this by extending TestWatcher and add a rule to my tests:
#Override
protected void failed(Throwable e, Description description) {
/*Do stuff with e*/
}
However the cucumber-junit iplementation does not allow the use of rules, so this solution would not work with Cucumber.
I don't think I need to explain why accessing a thrown exception on test failure would be useful, however I will still provide an Example:
My test environment is not always stable, so my tests might fail unexpectedly at any moment (there's no specific place I can try to catch the exception since it could occur at any time). When this happens I need the test to reschedule for another attempt, and log the incident so that we can get some good statistical data on the environment instability (when, how frequent, how long etc.)

The problem with the work around suggested by Frank Escobar:
By using reflection to reach into a frameworks internals you're depending on implementation details. This is a bad practice, when ever the framework changes its implementation your code may break as you will observe in Cucumber v5.0.0.
Hooks in Cucumber are designed to manipulate the test execution context before and after a scenario. They're not made to report on the test execution itself. Reporting is cross cutting concern and best managed by using the plugin system.
For example:
package com.example;
import io.cucumber.plugin.ConcurrentEventListener;
import io.cucumber.plugin.event.EventPublisher;
import io.cucumber.plugin.event.Result;
import io.cucumber.plugin.event.Status;
import io.cucumber.plugin.event.TestCase;
import io.cucumber.plugin.event.TestCaseFinished;
public class MyTestListener implements ConcurrentEventListener {
#Override
public void setEventPublisher(EventPublisher publisher) {
publisher.registerHandlerFor(TestCaseFinished.class, this::handleTestCaseFinished);
}
private void handleTestCaseFinished(TestCaseFinished event) {
TestCase testCase = event.getTestCase();
Result result = event.getResult();
Status status = result.getStatus();
Throwable error = result.getError();
String scenarioName = testCase.getName();
String id = "" + testCase.getUri() + testCase.getLine();
System.out.println("Testcase " + id + " - " + status.name());
}
}
When using JUnit 4 and TestNG you can activate this plugin using:
#CucumberOptions(plugin="com.example.MyTestListener")
With JUnit 5 you add it to junit-platform.properties:
cucumber.plugin=com.example.MyTestListener
Or if you are using the CLI
--plugin com.example.MyTestListener

I've implemented this method using reflections. You can't access directly to steps errors (stack trace). I've created this static method which allows you to access to "stepResults" attribute and then you can iterate and get the error and do whatever you want.
import cucumber.runtime.ScenarioImpl;
import gherkin.formatter.model.Result;
import org.apache.commons.lang3.reflect.FieldUtils;
import java.lang.reflect.Field;
import java.util.ArrayList;
#After
public void afterScenario(Scenario scenario) {
if (scenario.isFailed())
logError(scenario);
}
private static void logError(Scenario scenario) {
Field field = FieldUtils.getField(((ScenarioImpl) scenario).getClass(), "stepResults", true);
field.setAccessible(true);
try {
ArrayList<Result> results = (ArrayList<Result>) field.get(scenario);
for (Result result : results) {
if (result.getError() != null)
LOGGER.error("Error Scenario: {}", scenario.getId(), result.getError());
}
} catch (Exception e) {
LOGGER.error("Error while logging error", e);
}
}

You can to this by writing your own custom implementation of Formatter & Reporter interface. The empty implementation of Formatter is the NullFormatter.java which you can extend. You will need to provide implementations for the Reporter interface.
The methods which would be of interest will be the result() of the Reporter interface and possibly the done() method of Formatter. The result() has the Result object which has the exceptions.
You can look at RerunFormatter.java for clarity.
Github Formatter source
public void result(Result result) {
//Code to create logs or store to a database etc...
result.getError();
result.getErrorMessage();
}
You will need to add this class(com.myimpl.CustomFormRep) to the plugin option.
plugin={"pretty", "html:report", "json:reports.json","rerun:target/rerun.txt",com.myimpl.CustomFormRep}
More details on custom formatters.
You can use the rerun plugin to get a list of failed scenarios to run again. Not sure about scheduling a run of failed tests, code to create a batch job or schedule one on your CI tool.

This is the workaround for cucumber-java version 4.8.0 using reflection.
import cucumber.api.Result;
import io.cucumber.core.api.Scenario;
import io.cucumber.core.logging.Logger;
import io.cucumber.core.logging.LoggerFactory;
import io.cucumber.java.After;
import org.apache.commons.lang3.ClassUtils;
import org.apache.commons.lang3.reflect.FieldUtils;
import java.io.IOException;
import java.lang.reflect.Field;
import java.net.URL;
import java.util.ArrayList;
#After
public void afterScenario(Scenario scenario) throws IOException {
if(!scenario.getStatus().isOk(true)){
logError(scenario);
}
}
private static void logError(Scenario scenario) {
try {
Class clasz = ClassUtils.getClass("cucumber.runtime.java.JavaHookDefinition$ScenarioAdaptor");
Field fieldScenario = FieldUtils.getField(clasz, "scenario", true);
fieldScenario.setAccessible(true);
Object objectScenario = fieldScenario.get(scenario);
Field fieldStepResults = objectScenario.getClass().getDeclaredField("stepResults");
fieldStepResults.setAccessible(true);
ArrayList<Result> results = (ArrayList<Result>) fieldStepResults.get(objectScenario);
for (Result result : results) {
if (result.getError() != null) {
LOGGER.error(String.format("Error Scenario: %s", scenario.getId()), result.getError());
}
}
} catch (Exception e) {
LOGGER.error("Error while logging error", e);
}
}

For cucumber-js https://www.npmjs.com/package/cucumber/v/6.0.3
import { After } from 'cucumber'
After(async function(scenario: any) {
const exception = scenario.result.exception
if (exception) {
this.logger.log({ level: 'error', message: '-----------StackTrace-----------' })
this.logger.log({ level: 'error', message: exception.stack })
this.logger.log({ level: 'error', message: '-----------End-StackTrace-----------' })
}
})

After a lot of experimentation I now removed the Before/After-Annotations and rely on Cucumber-Events instead. They contain the TestCase (which is what the Scenario-class wraps) and a Result where you can call getError(); to get the Throwable.
Here is a simple example to get it working
import io.cucumber.plugin.EventListener;
import io.cucumber.plugin.event.EventPublisher;
import io.cucumber.plugin.event.Result;
import io.cucumber.plugin.event.Status;
import io.cucumber.plugin.event.TestCase;
import io.cucumber.plugin.event.TestCaseFinished;
import io.cucumber.plugin.event.TestCaseStarted;
import org.openqa.selenium.WebDriver;
public class TestCaseListener implements EventListener {
#Override
public void setEventPublisher(final EventPublisher publisher) {
publisher.registerHandlerFor(TestCaseStarted.class, this::onTestCaseStarted);
publisher.registerHandlerFor(TestCaseFinished.class, this::onTestCaseFinished);
}
public void onTestCaseStarted(TestCaseStarted event) {
TestCase testCase = event.getTestCase();
System.out.println("Starting " + testCase.getName());
// Other stuff you did in your #Before-Method.
// ...
}
private void onTestCaseFinished(final TestCaseFinished event) {
TestCase testCase = event.getTestCase();
System.out.println("Finished " + testCase.getName());
Result result = event.getResult();
if (result.getStatus() == Status.FAILED) {
final Throwable error = result.getError();
error.printStackTrace();
}
// Other stuff you did in your #After-Method.
// ...
}
}
All that's left to do is to register this class as a Cucumber-Plugin.
I did this by modifying my #CucumberOptions-annotation:
#CucumberOptions(plugin = {"com.example.TestCaseListener"})
I find this much cleaner than all of this reflection-madness, however it requires a lot more code-changes.
Edit
I don't know why, but this caused a lot of tests to randomly fail in a multithreaded environment.
I tried to figure it out, but now also use the ugly reflections mentioned in this thread:
public class SeleniumUtils {
private static final Logger log = LoggerFactory.getLogger(SeleniumUtils.class);
private static final Field field = FieldUtils.getField(Scenario.class, "delegate", true);
private static Method getError;
public static Throwable getError(Scenario scenario) {
try {
final TestCaseState testCase = (TestCaseState) field.get(scenario);
if (getError == null) {
getError = MethodUtils.getMatchingMethod(testCase.getClass(), "getError");
getError.setAccessible(true);
}
return (Throwable) getError.invoke(testCase);
} catch (Exception e) {
log.warn("error receiving exception", e);
}
return null;
}
}

If you just want to massage the result being sent to the report then you can extend the CucumberJSONFormatter and override the result method like this:
public class CustomReporter extends CucumberJSONFormatter {
CustomReporter(Appendable out) {
super(out);
}
/**
* Truncate the error in the output to the testresult.json file.
* #param result the error result
*/
#Override
void result(Result result) {
String errorMessage = null;
if (result.error) {
errorMessage = "Error: " + truncateError(result.error);
}
Result myResult = new Result(result.status, result.duration, errorMessage);
// Log the truncated error to the JSON report
super.result(myResult);
}
}
Then set the plugin option to:
plugin = ["com.myimpl.CustomReporter:build/reports/json/testresult.json"]

Alternative of mocking a static method present in some jar

I know that if I need to mock a static method, this indicates that my design has some issue, but in my case this does not seem to be a design issue.
BundleContext bundleContext = FrameworkUtil.getBundle(ConfigService.class).getBundleContext();
Here FrameworkUtil is a class present in an api jar. Using it in code cant be a design issue.
my problem here is while running this line
FrameworkUtil.getBundle(ConfigService.class);
returns null So my question, is there any way by which I can replace that null at runtime
I am using Mockito framewrok and my project does not allow me to use powermock.
if I use
doReturn(bundle).when(FrameworkUtil.class)
in this way getBundle method is not visible since its a static method.

You are correct that is not a design issue on your part. Without PowerMock, your options become a bit murkier, though.
I would suggest creating a non-static wrapper for the FrameworkUtil class that you can inject and mock.
Update: (David Wallace)
So you add a new class to your application, something like this
public class UtilWrapper {
public Bundle getBundle(Class<?> theClass) {
return FrameworkUtil.getBundle(theClass);
}
}
This class is so simple that you don't need to unit test it. As a general principle, you should only EVER write unit tests for methods that have some kind of logic to them - branching, looping or exception handling. One-liners should NOT be unit tested.
Now, within your application code, add a field of type UtilWrapper, and a setter for it, to every class that currently calls FrameworkUtil.getBundle. Add this line to the construtor of each such class
utilWrapper = new UtilWrapper();
And replace every call to FrameworkUtil.getBundle with utilWrapper.getBundle.
Now in your test, you make a mock UtilWrapper and stub it to return whatever Bundle you like.
when(mockUtilWrapper.getBundle(ConfigService.class)).thenReturn(someBundleYouMade);
and for the class that you're testing, call setUtilWrapper(mockUtilWrapper) or whatever. You don't need this last step if you're using #InjectMocks.
Now your test should all hang together, but using your mocked UtilWrapper instead of the one that relies on FrameworkUtil.

unit test
package x;
import static org.junit.Assert.*;
import org.junit.Before;
import org.junit.Test;
import org.mockito.Mockito;
public class GunTest {
#Before
public void setUp() throws Exception {
}
#Test
public void testFireTrue() {
final Gun unit = Mockito.spy(new Gun());
Mockito.doReturn(5).when(unit).getCount();
assertTrue(unit.fire2());
}
#Test
public void testFireFalse() {
final Gun unit = Mockito.spy(new Gun());
Mockito.doReturn(15).when(unit).getCount();
assertFalse(unit.fire2());
}
}
the unit:
fire calls the static method directly,
fire2 factors out the static call to a protected method:
package x;
public class Gun {
public boolean fire() {
if (StaticClass.getCount() > 10) {
return false;
}
else {
return true;
}
}
public boolean fire2() {
if (getCount() > 10) {
return false;
}
else {
return true;
}
}
protected int getCount() {
return StaticClass.getCount();
}
}