We Keep Coding

Check that JUnit Extension throws specific Exception

Suppose I develop an extension which disallows test method names to start with an uppercase character.
public class DisallowUppercaseLetterAtBeginning implements BeforeEachCallback {
#Override
public void beforeEach(ExtensionContext context) {
char c = context.getRequiredTestMethod().getName().charAt(0);
if (Character.isUpperCase(c)) {
throw new RuntimeException("test method names should start with lowercase.");
}
}
}
Now I want to test that my extension works as expected.
#ExtendWith(DisallowUppercaseLetterAtBeginning.class)
class MyTest {
#Test
void validTest() {
}
#Test
void TestShouldNotBeCalled() {
fail("test should have failed before");
}
}
How can I write a test to verify that the attempt to execute the second method throws a RuntimeException with a specific message?

Another approach could be to use the facilities provided by the new JUnit 5 - Jupiter framework.
I put below the code which I tested with Java 1.8 on Eclipse Oxygen. The code suffers from a lack of elegance and conciseness but could hopefully serve as a basis to build a robust solution for your meta-testing use case.
Note that this is actually how JUnit 5 is tested, I refer you to the unit tests of the Jupiter engine on Github.
public final class DisallowUppercaseLetterAtBeginningTest {
#Test
void testIt() {
// Warning here: I checked the test container created below will
// execute on the same thread as used for this test. We should remain
// careful though, as the map used here is not thread-safe.
final Map<String, TestExecutionResult> events = new HashMap<>();
EngineExecutionListener listener = new EngineExecutionListener() {
#Override
public void executionFinished(TestDescriptor descriptor, TestExecutionResult result) {
if (descriptor.isTest()) {
events.put(descriptor.getDisplayName(), result);
}
// skip class and container reports
}
#Override
public void reportingEntryPublished(TestDescriptor testDescriptor, ReportEntry entry) {}
#Override
public void executionStarted(TestDescriptor testDescriptor) {}
#Override
public void executionSkipped(TestDescriptor testDescriptor, String reason) {}
#Override
public void dynamicTestRegistered(TestDescriptor testDescriptor) {}
};
// Build our test container and use Jupiter fluent API to launch our test. The following static imports are assumed:
//
// import static org.junit.platform.engine.discovery.DiscoverySelectors.selectClass
// import static org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder.request
JupiterTestEngine engine = new JupiterTestEngine();
LauncherDiscoveryRequest request = request().selectors(selectClass(MyTest.class)).build();
TestDescriptor td = engine.discover(request, UniqueId.forEngine(engine.getId()));
engine.execute(new ExecutionRequest(td, listener, request.getConfigurationParameters()));
// Bunch of verbose assertions, should be refactored and simplified in real code.
assertEquals(new HashSet<>(asList("validTest()", "TestShouldNotBeCalled()")), events.keySet());
assertEquals(Status.SUCCESSFUL, events.get("validTest()").getStatus());
assertEquals(Status.FAILED, events.get("TestShouldNotBeCalled()").getStatus());
Throwable t = events.get("TestShouldNotBeCalled()").getThrowable().get();
assertEquals(RuntimeException.class, t.getClass());
assertEquals("test method names should start with lowercase.", t.getMessage());
}
Though a little verbose, one advantage of this approach is it doesn't require mocking and execute the tests in the same JUnit container as will be used later for real unit tests.
With a bit of clean-up, a much more readable code is achievable. Again, JUnit-Jupiter sources can be a great source of inspiration.

If the extension throws an exception then there's not much a #Test method can do since the test runner will never reach the #Test method. In this case, I think, you have to test the extension outside of its use in the normal test flow i.e. let the extension be the SUT.
For the extension provided in your question, the test might be something like this:
#Test
public void willRejectATestMethodHavingANameStartingWithAnUpperCaseLetter() throws NoSuchMethodException {
ExtensionContext extensionContext = Mockito.mock(ExtensionContext.class);
Method method = Testable.class.getMethod("MethodNameStartingWithUpperCase");
Mockito.when(extensionContext.getRequiredTestMethod()).thenReturn(method);
DisallowUppercaseLetterAtBeginning sut = new DisallowUppercaseLetterAtBeginning();
RuntimeException actual =
assertThrows(RuntimeException.class, () -> sut.beforeEach(extensionContext));
assertThat(actual.getMessage(), is("test method names should start with lowercase."));
}
#Test
public void willAllowTestMethodHavingANameStartingWithAnLowerCaseLetter() throws NoSuchMethodException {
ExtensionContext extensionContext = Mockito.mock(ExtensionContext.class);
Method method = Testable.class.getMethod("methodNameStartingWithLowerCase");
Mockito.when(extensionContext.getRequiredTestMethod()).thenReturn(method);
DisallowUppercaseLetterAtBeginning sut = new DisallowUppercaseLetterAtBeginning();
sut.beforeEach(extensionContext);
// no exception - good enough
}
public class Testable {
public void MethodNameStartingWithUpperCase() {
}
public void methodNameStartingWithLowerCase() {
}
}
However, your question suggests that the above extension is only an example so, more generally; if your extension has a side effect (e.g. sets something in an addressable context, populates a System property etc) then your #Test method could assert that this side effect is present. For example:
public class SystemPropertyExtension implements BeforeEachCallback {
#Override
public void beforeEach(ExtensionContext context) {
System.setProperty("foo", "bar");
}
}
#ExtendWith(SystemPropertyExtension.class)
public class SystemPropertyExtensionTest {
#Test
public void willSetTheSystemProperty() {
assertThat(System.getProperty("foo"), is("bar"));
}
}
This approach has the benefit of side stepping the potentially awkward setup steps of: creating the ExtensionContext and populating it with the state required by your test but it may come at the cost of limiting the test coverage since you can really only test one outcome. And, of course, it is only feasible if the extension has a side effect which can be evaulated in a test case which uses the extension.
So, in practice, I suspect you might need a combination of these approaches; for some extensions the extension can be the SUT and for others the extension can be tested by asserting against its side effect(s).

After trying the solutions in the answers and the question linked in the comments, I ended up with a solution using the JUnit Platform Launcher.
class DisallowUppercaseLetterAtBeginningTest {
#Test
void should_succeed_if_method_name_starts_with_lower_case() {
TestExecutionSummary summary = runTestMethod(MyTest.class, "validTest");
assertThat(summary.getTestsSucceededCount()).isEqualTo(1);
}
#Test
void should_fail_if_method_name_starts_with_upper_case() {
TestExecutionSummary summary = runTestMethod(MyTest.class, "InvalidTest");
assertThat(summary.getTestsFailedCount()).isEqualTo(1);
assertThat(summary.getFailures().get(0).getException())
.isInstanceOf(RuntimeException.class)
.hasMessage("test method names should start with lowercase.");
}
private TestExecutionSummary runTestMethod(Class<?> testClass, String methodName) {
SummaryGeneratingListener listener = new SummaryGeneratingListener();
LauncherDiscoveryRequest request = request().selectors(selectMethod(testClass, methodName)).build();
LauncherFactory.create().execute(request, listener);
return listener.getSummary();
}
#ExtendWith(DisallowUppercaseLetterAtBeginning.class)
static class MyTest {
#Test
void validTest() {
}
#Test
void InvalidTest() {
fail("test should have failed before");
}
}
}
JUnit itself will not run MyTest because it is an inner class without #Nested. So there are no failing tests during the build process.
Update
JUnit itself will not run MyTest because it is an inner class without #Nested. So there are no failing tests during the build process.
This is not completly correct. JUnit itself would also run MyTest, e.g. if "Run All Tests" is started within the IDE or within a Gradle build.
The reason why MyTest was not executed is because I used Maven and I tested it with mvn test. Maven uses the Maven Surefire Plugin to execute tests. This plugin has a default configuration which excludes all nested classes like MyTest.
See also this answer about "Run tests from inner classes via Maven" and the linked issues in the comments.

JUnit 5.4 introduced the JUnit Platform Test Kit which allows you to execute a test plan and inspect the results.
To take a dependency on it from Gradle, it might look something like this:
testImplementation("org.junit.platform:junit-platform-testkit:1.4.0")
And using your example, your extension test could look something like this:
import org.junit.jupiter.api.extension.ExtendWith
import org.junit.jupiter.api.fail
import org.junit.platform.engine.discovery.DiscoverySelectors
import org.junit.platform.testkit.engine.EngineTestKit
import org.junit.platform.testkit.engine.EventConditions
import org.junit.platform.testkit.engine.TestExecutionResultConditions
internal class DisallowUpperCaseExtensionTest {
#Test
internal fun `succeed if starts with lower case`() {
val results = EngineTestKit
.engine("junit-jupiter")
.selectors(
DiscoverySelectors.selectMethod(ExampleTest::class.java, "validTest")
)
.execute()
results.tests().assertStatistics { stats ->
stats.finished(1)
}
}
#Test
internal fun `fail if starts with upper case`() {
val results = EngineTestKit
.engine("junit-jupiter")
.selectors(
DiscoverySelectors.selectMethod(ExampleTest::class.java, "TestShouldNotBeCalled")
)
.execute()
results.tests().assertThatEvents()
.haveExactly(
1,
EventConditions.finishedWithFailure(
TestExecutionResultConditions.instanceOf(java.lang.RuntimeException::class.java),
TestExecutionResultConditions.message("test method names should start with lowercase.")
)
)
}
#ExtendWith(DisallowUppercaseLetterAtBeginning::class)
internal class ExampleTest {
#Test
fun validTest() {
}
#Test
fun TestShouldNotBeCalled() {
fail("test should have failed before")
}
}
}

How to test a java test class?

I have created a JUnit conformance java test class. It basically runs as a presubmit test whenever a developer submitting a code.
#RunWith(JUnit4.class)
public class ConformanceTest {
#Before
public void setUp() {
// implementation, e.g. reading from a flag
}
#Test
public void testStuff() {
// chekcing implementation
}
}
I would like to create a test for that class (in this case ConformanceTestTest.java).
Do I have more elegant way rather than doing something like this
#RunWith(JUnit4.class)
public class ConformanceTestTest {
#Test
public void testTestStuff() {
ConformanceTest test = new ConformanceTest();
// Setting up the environment, e.g. setting the flag for conformance
// test class
/* Need to understand the ConformanceTest lifecycle
instead of testing 'testStuff()' only */
test.setUp();
assertException(AssertionError.class, test.testStuff());
/* I might have to do test.tearDown() if there is any */
}
}

Testing ClassNotFound Exception

I'm trying to test that a class is not found with UnitTest on Android.
What's going on:
1. I'm writing an android library with transitive dependencies which are resolved in the host application
2. The developer may remove some dependencies for example remove all com.example.package
3. I have a Factory that will try to instantiate (using reflection) an Object and catch the ClassNotFoundException. If the developer remove the dependencies, the exception should be thrown.
4. I want to test this case, but all I found is issue with dependencies, not how to test for it.
Example code I want to test
try {
sNetworkResponseBuilderClass = OkHttpNetworkResponse.Builder.class;
} catch (Exception e){
// <<<< I want to test this case
new ClassNotFoundException("Unable to find OkHttpNetworkResponse.Builder.class").printStackTrace();
return null;
}
library used: hamcrast, mockito, JUnit 4.
Do you know how to do it?

So for me the first thing you need to do is to extract the part of the code that can throw a ClassNotFoundException in order to be able to easily mock it, something like:
public Class<? extends NetworkResponseBuilder> getNetworkResponseBuilderClass()
throws ClassNotFoundException {
// Your logic here
}
Then you can test a real factory instance using Mockito.spy to be able to redefine the behavior of the method getNetworkResponseBuilderClass() as next:
public void testFactoryIfNetworkResponseBuilderNotFound() {
Factory factory = spy(new Factory());
when(factory.getNetworkResponseBuilderClass()).thenThrow(
new ClassNotFoundException()
);
// The rest of your test here
}
public void testFactoryIfNetworkResponseBuilderFound() {
Factory factory = spy(new Factory());
when(factory.getNetworkResponseBuilderClass()).thenReturn(
OkHttpNetworkResponse.Builder.class
);
// The rest of your test here
}
More details about Mockito.spy.

Not quite sure if I understood your question correctly, but you can check with JUnit if an exception gets thrown:
#Test(expected=ClassNotFoundException.class)
public void testClassNotFoundException() {
// a case where the exception gets thrown
}

OkHttpNetworkResponse.Builder might be as follows:
package com.example.model;
public class OkHttpNetworkResponse {
public static class Builder {
}
}
I have a Factory that will try to instantiate (using reflection) an Object and catch the ClassNotFoundException. If the developer remove
the dependencies, the exception should be thrown.
Factory Class: which will create any object might be as follows:
package com.example.factory;
public class Factory {
public static Object getInstance(String className)
throws ClassNotFoundException, InstantiationException,
IllegalAccessException {
Class clazz = Class.forName(className);
return clazz.newInstance();
}
}
The developer may remove some dependencies for example remove all com.example.package
I want to test this case, but all I found is issue with dependencies, not how to test for it.
FactoryTest Class: which will test whether ClassNotFoundException is thrown or not might be as follows: N.B: please Check the comments carefully.
package com.example.factory;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
import java.io.File;
import java.io.IOException;
import org.junit.Test;
public class FactoryTest {
Factory factory;
#Test(expected=ClassNotFoundException.class)
public void test() throws IOException, ClassNotFoundException, InstantiationException, IllegalAccessException {
ClassLoader loader = FactoryTest.class.getClassLoader();
String directory = loader.getResource(".").getPath() + "/com/example/model";
File dir = new File(directory);
//Checking directory already existed or not..
assertTrue("Directory:"+dir.getPath()+" not exist",dir.exists());
//Deleting directory
deleteDirectoryProgramatically(directory);
//Checking directory already deleted or not..
assertFalse("Directory:"+dir.getPath()+" still exist",dir.exists());
//Now getInstance Method will throw ClassNotFoundException because OkHttpNetworkResponse.Builder.class has been deleted programatically.
Factory.getInstance("OkHttpNetworkResponse.Builder.class");
}
private void deleteDirectoryProgramatically(String directory) {
File dir = new File(directory);
System.out.println(dir.getAbsolutePath());
String[] files = dir.list();
for (String f : files) {
File fl = new File(directory,f);
System.out.println(f+ " deleted?"+fl.delete());
}
System.out.println(dir+ " deleted?"+dir.delete());
}
}

It is very simple issue. JUnit4 exception unit testing is given below with an example. Hope it will clarify you.
MyNumber.java
public class MyNumber {
int number;
public MyNumber div(MyNumber rhs) {
if (rhs.number == 0) throw new IllegalArgumentException("Cannot divide by 0!");
this.number /= rhs.number;
return this;
}
}
MyNumberTest.java
public class MyNumberTest {
private MyNumber number1, number2; // Test fixtures
#Test(expected = IllegalArgumentException.class)
public void testDivByZero() {
System.out.println("Run #Test testDivByZero"); // for illustration
number2.setNumber(0);
number1.div(number2);
}
}
JUnit - Exceptions Test
To test if the code throws a desired exception, use annotation #Test(expected = exception.class), as illustrated in the previous example. For your case it will be
/**
* Check for class not found exception
**/
#Test(expected=ClassNotFoundException.class)
public void testClassNotFoundException() {
.....
}
For better understanding, you can go through this tutorial: Java Unit
Testing - JUnit & TestNG. It contains full running code example
step by step with explanation.

inside catch you can check the object with the instanceof operator as :
try {
sNetworkResponseBuilderClass = OkHttpNetworkResponse.Builder.class;
} catch (Exception e){
if(e instanceof ClassNotFoundException){
// here you can do the code you want in case of ClassNotFoundException thrown
}
}

it is your dictionary problem. in your dictionary in test class will not have . change your dictionary.

Use Class.forName("com.example.ClassName")
try {
Class.forName("com.example.OkHttpNetworkResponse.Builder");
} catch (ClassNotFoundException e) {
// This class was not found
}
See Class.forName(String className)

Functional test in java, explanation

This is a sample program:
public class FunctionalTest {
public int f(int r) {
int result = r * 5;
return result;
}
public static void main(String[] args) {
FunctionalTest funct = new FunctionalTest();
System.out.println(funct.f(5));
}
}
I'm a beginner.
How to write a functional test for this code?
How to write functional tests? Do I need to TestNG?
Is it enough to write the examination method?
Could someone explain to me and write a sample functional test for this program?

Well, if you're specifically asking for functional testing, there's not much you can do with that code snippet. You can do a unit test from the f method using JUnit like this:
#Test
public void testF(){
FunctionalTest t1 = new FunctionalTest();
assertEquals((t1.f(1) % 5), 0); //checks that is getting multiplied by 5.
}
However, you want functional testing, so by running your compiled app and assessing the result you're testing your app functionality by multiple units (AKA integration): your f method and your main method.
Regards!

First of all, you need to have a clear definition of contract you want to verify. From the code, I assume it is something like "the method should return the number equal to argument multiplied by 5".
TestNG, JUnit or other test frameworks is not mandatory for your case. The test may look like:
public void testF() {
int arg = 5;
int result = new FunctionalTest().f(arg);
assert result == arg * 5;
}
Also please keep in mind that to use assert you need JVM started with -ea flag.

Beware the terms you used:
the functional testing provide values to your user/customer
That implies:
black box testing: you have to test your whole system (hard+soft)
the test should target your user/customer needs (explicit report or test)
You can use whatever you want to test the feature (from unit test to jbehave).
In your case (using JUnit 4 and AssertJ):
import org.assertj.core.api.Assertions;
import org.junit.Test;
import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;
/*
As an user
I want have 25 whatever I sent
*/
public class NumberGenerationTest {
private static final String PATH = "directory of your class file";
private InputStream stdout;
/* Nominal case */
#Test
public void shall_return_number_25_when_called_with_5() throws Exception {
when_I_call_FunctionalTest_with("5");
then_it_returns("25");
}
/* Nominal case or potential error case */
#Test
public void shall_return_number_25_when_called_with_10() throws Exception {
when_I_call_FunctionalTest_with("10");
then_it_returns("25");
}
/* Nominal case or potential error case */
#Test
public void shall_return_number_25_when_called_with_ABC() throws Exception {
when_I_call_FunctionalTest_with("ABC");
then_it_returns("25");
}
private void when_I_call_FunctionalTest_with(String parameter) throws Exception {
ProcessBuilder builder = new ProcessBuilder("java" ,"-classpath", PATH,"FunctionalTest" , parameter);
builder.redirectErrorStream(true);
Process process = builder.start();
stdout = process.getInputStream ();
}
private void then_it_returns(String expectedResult) throws Exception {
BufferedReader reader = new BufferedReader (new InputStreamReader(stdout));
String line = reader.readLine ();
Assertions.assertThat(line).isNotNull();
Assertions.assertThat(line).isEqualTo(expectedResult);
}
}
It seems you have an error in your main() ... or not.

Is there a way to make Eclipse run a JUnit test multiple times until failure?

We occasionally have bugs that appear once in every X runs. Before people check in stuff (where it is automatically JUnit'd), our devs need to pass JUnit locally via Eclipse.
Is there some convenient way (built in or high-quality Plugin) to make Eclipse run the same test X times and stop if there's a failure? An alternative to just clicking Run X times?
Note that I'm looking for something in the UI (e.g., right click and say "Run X times" instead of just "Run").

If the for loop works, then I agree with nos.
If you need to repeat the entire setup-test-teardown, then you can use a TestSuite:
Right-click on the package containing the test to repeat
Go to New and choose to create a JUnit test SUITE
Make sure that only the test you want to repeat is selected and click through to finish.
Edit the file to run it multiple times.
In the file you just find the
addTestSuite(YourTestClass.class)
line, and wrap that in a for loop.
I'm pretty sure that you can use addTest instead of addTestSuite to get it to only run one test from that class if you just want to repeat a single test method.

If you really want to run a test class until failure, you need your own runner.
#RunWith(RunUntilFailure.class)
public class YourClass {
// ....
}
which could be implemented as follows...
package com.example;
import org.junit.internal.runners.*;
import org.junit.runner.notification.*;
import org.junit.runner.*;
public class RunUntilFailure extends Runner {
private TestClassRunner runner;
public RunUntilFailure(Class<?> klass) throws InitializationError {
this.runner = new TestClassRunner(klass);
}
#Override
public Description getDescription() {
Description description = Description.createSuiteDescription("Run until failure");
description.addChild(runner.getDescription());
return description;
}
#Override
public void run(RunNotifier notifier) {
class L extends RunListener {
boolean fail = false;
public void testFailure(Failure failure) throws Exception { fail = true; }
}
L listener = new L();
notifier.addListener(listener);
while (!listener.fail) runner.run(notifier);
}
}
...releasing untested code, feeling TDD guilt :)

Based on #akuhn's answer, here is what I came up with - rather than running forever, this will run 50 times or until failure, whichever comes first.
package com.foo
import org.junit.runner.Description;
import org.junit.runner.Runner;
import org.junit.runner.notification.Failure;
import org.junit.runner.notification.RunListener;
import org.junit.runner.notification.RunNotifier;
import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runners.model.InitializationError;
public class RunManyTimesUntilFailure extends Runner {
private static final int MAX_RUN_COUNT = 50;
private BlockJUnit4ClassRunner runner;
#SuppressWarnings("unchecked")
public RunManyTimesUntilFailure(final Class testClass) throws InitializationError {
runner = new BlockJUnit4ClassRunner(testClass);
}
#Override
public Description getDescription() {
final Description description = Description.createSuiteDescription("Run many times until failure");
description.addChild(runner.getDescription());
return description;
}
#Override
public void run(final RunNotifier notifier) {
class L extends RunListener {
boolean shouldContinue = true;
int runCount = 0;
#Override
public void testFailure(#SuppressWarnings("unused") final Failure failure) throws Exception {
shouldContinue = false;
}
#Override
public void testFinished(#SuppressWarnings("unused") Description description) throws Exception {
runCount++;
shouldContinue = (shouldContinue && runCount < MAX_RUN_COUNT);
}
}
final L listener = new L();
notifier.addListener(listener);
while (listener.shouldContinue) {
runner.run(notifier);
}
}
}

I know it doesn't answer the question directly but if a test isn't passing every time it is run it is a test smell known as Erratic Test. There are several possible causes for this (from xUnit Test Patterns):
Interacting Tests
Interacting Test Suites
Lonely Test
Resource Leakage
Resource Optimism
Unrepeatable Test
Test Run War
Nondeterministic Test
The details of each of these is documented in Chapter 16 of xUnit Test Patterns.

Here is a post I wrote that shows several ways of running the tests repeatedly with code examples:
http://codehowtos.blogspot.com/2011/04/run-junit-test-repeatedly.html
You can use the #Parametrized runner, or use the special runner included in the post
There is also a reference to a #Retry implementation

I don't believe there's a built in way for junit to do exactly what you're asking for.
If multiple runs produce different result, you should have a unit test testing that case. Wich might be as simple as running a for loop in the relevant test cases.

It is possible to implement such an loop with TestRules (since JUnit 4.9)
A very simple implementation that runs every Test 10 times:
import org.junit.rules.TestRule;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;
public class SimpleRepeatRule implements TestRule {
private static class SimpleRepeatStatement extends Statement {
private final Statement statement;
private SimpleRepeatStatement(Statement statement) {
this.statement = statement;
}
#Override
public void evaluate() throws Throwable {
for (int i = 0; i < 10; i++) {
statement.evaluate();
}
}
}
#Override
public Statement apply(Statement statement, Description description) {
return new SimpleRepeatStatement(statement);
}
}
usage:
public class Run10TimesTest {
#Rule
public SimpleRepeatRule repeatRule = new SimpleRepeatRule();
#Test
public void myTest(){...}
}
For a more useful implementation based on an annotation that define which test method has to been executed how often have a look at this blog:
http://www.codeaffine.com/2013/04/10/running-junit-tests-repeatedly-without-loops/