We Keep Coding

Check that JUnit Extension throws specific Exception

Suppose I develop an extension which disallows test method names to start with an uppercase character.
public class DisallowUppercaseLetterAtBeginning implements BeforeEachCallback {
#Override
public void beforeEach(ExtensionContext context) {
char c = context.getRequiredTestMethod().getName().charAt(0);
if (Character.isUpperCase(c)) {
throw new RuntimeException("test method names should start with lowercase.");
}
}
}
Now I want to test that my extension works as expected.
#ExtendWith(DisallowUppercaseLetterAtBeginning.class)
class MyTest {
#Test
void validTest() {
}
#Test
void TestShouldNotBeCalled() {
fail("test should have failed before");
}
}
How can I write a test to verify that the attempt to execute the second method throws a RuntimeException with a specific message?

Another approach could be to use the facilities provided by the new JUnit 5 - Jupiter framework.
I put below the code which I tested with Java 1.8 on Eclipse Oxygen. The code suffers from a lack of elegance and conciseness but could hopefully serve as a basis to build a robust solution for your meta-testing use case.
Note that this is actually how JUnit 5 is tested, I refer you to the unit tests of the Jupiter engine on Github.
public final class DisallowUppercaseLetterAtBeginningTest {
#Test
void testIt() {
// Warning here: I checked the test container created below will
// execute on the same thread as used for this test. We should remain
// careful though, as the map used here is not thread-safe.
final Map<String, TestExecutionResult> events = new HashMap<>();
EngineExecutionListener listener = new EngineExecutionListener() {
#Override
public void executionFinished(TestDescriptor descriptor, TestExecutionResult result) {
if (descriptor.isTest()) {
events.put(descriptor.getDisplayName(), result);
}
// skip class and container reports
}
#Override
public void reportingEntryPublished(TestDescriptor testDescriptor, ReportEntry entry) {}
#Override
public void executionStarted(TestDescriptor testDescriptor) {}
#Override
public void executionSkipped(TestDescriptor testDescriptor, String reason) {}
#Override
public void dynamicTestRegistered(TestDescriptor testDescriptor) {}
};
// Build our test container and use Jupiter fluent API to launch our test. The following static imports are assumed:
//
// import static org.junit.platform.engine.discovery.DiscoverySelectors.selectClass
// import static org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder.request
JupiterTestEngine engine = new JupiterTestEngine();
LauncherDiscoveryRequest request = request().selectors(selectClass(MyTest.class)).build();
TestDescriptor td = engine.discover(request, UniqueId.forEngine(engine.getId()));
engine.execute(new ExecutionRequest(td, listener, request.getConfigurationParameters()));
// Bunch of verbose assertions, should be refactored and simplified in real code.
assertEquals(new HashSet<>(asList("validTest()", "TestShouldNotBeCalled()")), events.keySet());
assertEquals(Status.SUCCESSFUL, events.get("validTest()").getStatus());
assertEquals(Status.FAILED, events.get("TestShouldNotBeCalled()").getStatus());
Throwable t = events.get("TestShouldNotBeCalled()").getThrowable().get();
assertEquals(RuntimeException.class, t.getClass());
assertEquals("test method names should start with lowercase.", t.getMessage());
}
Though a little verbose, one advantage of this approach is it doesn't require mocking and execute the tests in the same JUnit container as will be used later for real unit tests.
With a bit of clean-up, a much more readable code is achievable. Again, JUnit-Jupiter sources can be a great source of inspiration.

If the extension throws an exception then there's not much a #Test method can do since the test runner will never reach the #Test method. In this case, I think, you have to test the extension outside of its use in the normal test flow i.e. let the extension be the SUT.
For the extension provided in your question, the test might be something like this:
#Test
public void willRejectATestMethodHavingANameStartingWithAnUpperCaseLetter() throws NoSuchMethodException {
ExtensionContext extensionContext = Mockito.mock(ExtensionContext.class);
Method method = Testable.class.getMethod("MethodNameStartingWithUpperCase");
Mockito.when(extensionContext.getRequiredTestMethod()).thenReturn(method);
DisallowUppercaseLetterAtBeginning sut = new DisallowUppercaseLetterAtBeginning();
RuntimeException actual =
assertThrows(RuntimeException.class, () -> sut.beforeEach(extensionContext));
assertThat(actual.getMessage(), is("test method names should start with lowercase."));
}
#Test
public void willAllowTestMethodHavingANameStartingWithAnLowerCaseLetter() throws NoSuchMethodException {
ExtensionContext extensionContext = Mockito.mock(ExtensionContext.class);
Method method = Testable.class.getMethod("methodNameStartingWithLowerCase");
Mockito.when(extensionContext.getRequiredTestMethod()).thenReturn(method);
DisallowUppercaseLetterAtBeginning sut = new DisallowUppercaseLetterAtBeginning();
sut.beforeEach(extensionContext);
// no exception - good enough
}
public class Testable {
public void MethodNameStartingWithUpperCase() {
}
public void methodNameStartingWithLowerCase() {
}
}
However, your question suggests that the above extension is only an example so, more generally; if your extension has a side effect (e.g. sets something in an addressable context, populates a System property etc) then your #Test method could assert that this side effect is present. For example:
public class SystemPropertyExtension implements BeforeEachCallback {
#Override
public void beforeEach(ExtensionContext context) {
System.setProperty("foo", "bar");
}
}
#ExtendWith(SystemPropertyExtension.class)
public class SystemPropertyExtensionTest {
#Test
public void willSetTheSystemProperty() {
assertThat(System.getProperty("foo"), is("bar"));
}
}
This approach has the benefit of side stepping the potentially awkward setup steps of: creating the ExtensionContext and populating it with the state required by your test but it may come at the cost of limiting the test coverage since you can really only test one outcome. And, of course, it is only feasible if the extension has a side effect which can be evaulated in a test case which uses the extension.
So, in practice, I suspect you might need a combination of these approaches; for some extensions the extension can be the SUT and for others the extension can be tested by asserting against its side effect(s).

After trying the solutions in the answers and the question linked in the comments, I ended up with a solution using the JUnit Platform Launcher.
class DisallowUppercaseLetterAtBeginningTest {
#Test
void should_succeed_if_method_name_starts_with_lower_case() {
TestExecutionSummary summary = runTestMethod(MyTest.class, "validTest");
assertThat(summary.getTestsSucceededCount()).isEqualTo(1);
}
#Test
void should_fail_if_method_name_starts_with_upper_case() {
TestExecutionSummary summary = runTestMethod(MyTest.class, "InvalidTest");
assertThat(summary.getTestsFailedCount()).isEqualTo(1);
assertThat(summary.getFailures().get(0).getException())
.isInstanceOf(RuntimeException.class)
.hasMessage("test method names should start with lowercase.");
}
private TestExecutionSummary runTestMethod(Class<?> testClass, String methodName) {
SummaryGeneratingListener listener = new SummaryGeneratingListener();
LauncherDiscoveryRequest request = request().selectors(selectMethod(testClass, methodName)).build();
LauncherFactory.create().execute(request, listener);
return listener.getSummary();
}
#ExtendWith(DisallowUppercaseLetterAtBeginning.class)
static class MyTest {
#Test
void validTest() {
}
#Test
void InvalidTest() {
fail("test should have failed before");
}
}
}
JUnit itself will not run MyTest because it is an inner class without #Nested. So there are no failing tests during the build process.
Update
JUnit itself will not run MyTest because it is an inner class without #Nested. So there are no failing tests during the build process.
This is not completly correct. JUnit itself would also run MyTest, e.g. if "Run All Tests" is started within the IDE or within a Gradle build.
The reason why MyTest was not executed is because I used Maven and I tested it with mvn test. Maven uses the Maven Surefire Plugin to execute tests. This plugin has a default configuration which excludes all nested classes like MyTest.
See also this answer about "Run tests from inner classes via Maven" and the linked issues in the comments.

JUnit 5.4 introduced the JUnit Platform Test Kit which allows you to execute a test plan and inspect the results.
To take a dependency on it from Gradle, it might look something like this:
testImplementation("org.junit.platform:junit-platform-testkit:1.4.0")
And using your example, your extension test could look something like this:
import org.junit.jupiter.api.extension.ExtendWith
import org.junit.jupiter.api.fail
import org.junit.platform.engine.discovery.DiscoverySelectors
import org.junit.platform.testkit.engine.EngineTestKit
import org.junit.platform.testkit.engine.EventConditions
import org.junit.platform.testkit.engine.TestExecutionResultConditions
internal class DisallowUpperCaseExtensionTest {
#Test
internal fun `succeed if starts with lower case`() {
val results = EngineTestKit
.engine("junit-jupiter")
.selectors(
DiscoverySelectors.selectMethod(ExampleTest::class.java, "validTest")
)
.execute()
results.tests().assertStatistics { stats ->
stats.finished(1)
}
}
#Test
internal fun `fail if starts with upper case`() {
val results = EngineTestKit
.engine("junit-jupiter")
.selectors(
DiscoverySelectors.selectMethod(ExampleTest::class.java, "TestShouldNotBeCalled")
)
.execute()
results.tests().assertThatEvents()
.haveExactly(
1,
EventConditions.finishedWithFailure(
TestExecutionResultConditions.instanceOf(java.lang.RuntimeException::class.java),
TestExecutionResultConditions.message("test method names should start with lowercase.")
)
)
}
#ExtendWith(DisallowUppercaseLetterAtBeginning::class)
internal class ExampleTest {
#Test
fun validTest() {
}
#Test
fun TestShouldNotBeCalled() {
fail("test should have failed before")
}
}
}

JUnitCore.run() is ignoring #BeforeAll annotations

I am working in Eclipse Oxygen.1a (4.7.1a) with the JUnit 5 library, and it seems like none of my annotated methods are running correctly when I run a test class using JUnitCore.
For example, if I call the following class using JUnitCore.run(TestClass.class) :
public class TestClass {
#BeforeAll
public static void beforeAll() {
System.out.println("In TestClass.beforeAll");
}
#Test
public void testMethod() {
System.out.println("In TestClass.testMethod");
}
#AfterAll
public static void afterAll() {
System.out.println("In TestClass.afterAll");
}
}
There is no output to System.out, and the Result object says that 1 test failed. Implying that none of these methods ran. I can use a JUnit Launcher to run the test class like so:
final LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request()
.selectors(selectClass(TestClass.class)).build();
final Launcher launcher = LauncherFactory.create();
launcher.execute(request);
However, this does not give me any feedback about how many tests passed/failed, it seems to just run them. I can't find much documentation on using JUnitCore with JUnit5, is there something newer that I should be working with?

You should not use JUnitCore with JUnit 5, but instead use the platform launcher API as you did in the second part of your analysis.
launcher#execute(.) does not return any value, hence you need to use a listener to aggregate the results from the execution of your tests, as per the JUnit 5 documentation 7.1.2 Executing Tests.
There is no return value for the execute() method, but you can easily use a listener to aggregate the final results in an object of your own. For an example see the SummaryGeneratingListener.
You can for instance produce a TestExecutionSummary which may indeed provide you with the information you want to collect:
final LauncherDiscoveryRequest request =
LauncherDiscoveryRequestBuilder.request()
.selectors(selectClass(TestClass.class))
.build();
final Launcher launcher = LauncherFactory.create();
final SummaryGeneratingListener listener = new SummaryGeneratingListener();
launcher.registerTestExecutionListeners(listener);
launcher.execute(request);
TestExecutionSummary summary = listener.getSummary();
long testFoundCount = summary.getTestsFoundCount();
List<Failure> failures = summary.getFailures();
...

Java Unit testing path matching

I am trying to understand Unit testing and how the correct classes are being fetched on test time,
I'm having a hard time understanding exactly what is going on behind the scenes and how safe/correct my usages for this are.
This is a very simple example of what i am trying to do, i wrote it here inline so it probably contains some errors, please try to ignore them, as stupid as they may be.
A very simple project directory:
ba/src/main/java/utils/BaUtils.java
ba/test/main/java/utils/BaUtilsTest.java
notBa/src/main/java/im/BaObj.java
BaUtils.java code:
package com.ba.utils;
import notBa.im.BaObj;
public class BaUtils{
public String doSomething(BaObj obj){
obj.doSomething();
}
}
I would like to test BaUtils wihtout actually calling doSomething, and i can't change anything on BaObj class or notBa package. I know that i can (by 'can' i mean it will work) add a new java file to 'ba' project (ba/test/java/notBa/main/java/im/BaObj.java) that will have the same package as the original BaObj, and at runtime the test will import this one instead of the real one, so BaUtils code is tested but BaObj code is not excecuted.
that should look something like like :
package notBa.im.Baobj
public class BaObj{
public void doSomething(){
System.out.println("Did something");
}
}
My questions are (And thank you for reaching this far):
How does this work (Reading references would be great).
Is this kind of test building considered 'good' or 'safe' ?
Thanks!

The solution is to use a mocking framework (I for myself like Mockito).
The test would look like this:
class BlaUtilTes{
#Rule
public MockitoRule mockitoRule = MockitoJUnit.rule();
#Mock
Blaobj blaobj;
#Test
public void doSomething_WithMockedBlaobj_callsDosomethingOnBlaobj(){
// arrange
BlaUtil blaUtil= new BlaUtil();
// act
blaUtil.doSomething(blaobj);
// assert
Mockito.verify(blaobj).doSomething();
}
}
find more information here http://www.vogella.com/tutorials/Mockito/article.html#testing-with-mock-objects

Your BaUtilsTest class should look like this.. I have used mockito for for mocking external dependencies. Also I changed the method return type to String for easy understanding.
#RunWith(MockitoJUnitRunner.class)
class BaUtilsTest {
BaUtils util;
#Mock
BaObj mockBaObj;
#Before
public void setup() {
util = new BaUtils();
}
#Test
public void testDoSomething() {
Mockito.when(mockBaObj.doSomething()).thenReturn("did the work using mock");
String result = util.doSomething(mockBaObj);
Assert.assertEquals("did the work using mock", result);
}
}

How to use JUnit4TestAdapter with objects

I am trying to write a test suite using JUnit4 by relying on JUnit4TestAdapter. Having a look at the code of this class I saw that it only works with a Class as input. I would like to build a test class and set a parameter on it before running it with my TestSuite. Unfortunately, Junit4TestAdapter is building the test by using reflection (not 100% sure about the mechanism behind it), which means that I cannot change my test class on runtime.
Has anybody done anything similar before? Is there any possible workaround to this issue? Thanks for your help!
public class SimpleTest {
#Test
public void testBasic() {
TemplateTester tester = new TemplateTester();
ActionIconsTest test = new ActionIconsTest();
test.setParameter("New Param Value");
tester.addTests(test);
tester.run();
}
}
/////
public class TemplateTester {
private TestSuite suite;
public TemplateTester() {
suite = new TestSuite();
}
public void addTests(TemplateTest... tests) {
for (TemplateTest test : tests) {
suite.addTest(new JUnit4TestAdapter(test.getClass()));
}
}
public void run() {
suite.run(new TestResult());
}
}
/////
public interface TemplateTest {
}
/////
public class ActionIconsTest extends BaseTestStrategy implements TemplateTest {
#Test
public void icons() {
//Test logic here
}
public void navigateToTestPage() {
//Here I need the parameter
}
}
/////
public abstract class BaseTestStrategy {
protected String parameter;
#Before
public void init() {
navigateToTestPage();
}
public abstract void navigateToTestPage();
public void setParameter(String parameter) {
this.parameter = parameter;
}
}
I am trying to test a web application with Selenium. The way I want to test is by splitting the functionality, e.g., I want to test the available icons (ActionIconsTest), then I'd like to test other parts like buttons, etc.
The idea behind this is to have a better categorization of the functionality available in certain screen. This is quite coupled with the way we are currently developing our web app.
With this in mind, TemplateTest is just an interface implemented by the different kind of tests (ActionIconTest, ButtonTest, etc) available in my system.
TemplateTester is a Junit suite test with all the different tests that implement the interface TemplateTest.
The reason for this question is because I was trying to implement a Strategy pattern and then realized of the inconvenient of passing a class to Junit4TestAdapter in runtime.

Well, taking in account that JUNIT needs your tester's Class object as an object factory (so he can create several instances of your tester), I can only suggest you pass parameters to your tester through System Properties.
Moreover, it's the recommended way of passing parameters: http://junit.org/faq.html#running_7

Extending Junit ParentRunner Class to filter test methods

I have a requirement of reading a text file which contains list of all the testmethods in yes/no value and to pick the "yes" marked testmethods only for a TestCase Class,and to execute in Junit.
So I have written a script to read the file and to group it in a map< TestCaseName,ArrayList_ofEnabledTestMethods > . To run that I found one option is to use Assume.assumeTrue().
But I wanted to try some otherway... instead of writting extra lines before each test methods , So I tried to write a custom runner (ABCSuite which extends ParentRunner) and planned to use it in my TestSuite file like below :
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
#RunWith(ABCSuite.class)
#Suite.SuiteClasses({TestCalc.class})
public class BatTest{
}
Here TestCalc.class contains all the test methods some of which is marked "yes" in the earlier mentioned text file .
Please let me know how I can use of extending the ParentRunner class/Junit Libraries to achieve this . If any good tutorial is there or any link which addressed this before please.. share

You can do this by extending BlockJUnit4ClassRunner:
public class FilterRunner extends BlockJUnit4ClassRunner {
private List<String> testsToRun = Arrays.asList(new String[] { "test1" });
public FilterRunner(Class<?> klass) throws InitializationError {
super(klass);
}
#Override
protected void runChild(final FrameworkMethod method, RunNotifier notifier) {
Description description= describeChild(method);
if (method.getAnnotation(Ignore.class) != null || !testsToRun.contains(method.getName())) {
notifier.fireTestIgnored(description);
} else {
runLeaf(methodBlock(method), description, notifier);
}
}
}
You can fill in testsToRun as you like. The above will mark the other tests as Ignored. You use this like:
#RunWith(Suite.class)
#SuiteClasses({Class1Test.class})
public class TestSuite {
}
#RunWith(FilterRunner.class)
public class Class1Test {
#Test
public void test1() {
System.out.println("test1");
}
#Test
public void test2() {
System.out.println("test2");
}
}
This produces the following output:
test1
If you don't want to add the #FilterRunner to each test class, look at my answer to How to define JUnit method rule in a suite?.

The JUnit way of implementing this would be an implementation of a Filter. It must be instantiated by the Runner that implements Filterable. Filters are applied recursively through the tree of tests. So you only need to apply that filter once in your base suite.
You need to extend a runner and in the constructor apply the filter. To make things more flexible, you could configure the filters that should be applied with annotations.
I had the same requirement and that worked out well.