In my project mostly all imports relay on io.vertx.reactivex.core.Vertx package import, effectively makes whole project use reactivex (not core/vanilla) version of Vertx and it's verticles. I started to unit test a bit our application and to do so and to make it play nicely with JUint, according to this documentation following setup is needed to use JUnit and to run test cases in correct thread and verticle context:
#RunWith(VertxUnitRunner.class) /* Set the runner */
public class DaoTest {
#Rule /* Define rule to get vertx instance in test */
public RunTestOnContext rule = new RunTestOnContext();
#Test
public void exampleTest(TestContext context) {
TestClass t = new TestClass(rule.vertx());
}
}
the definition of TestClass is following:
import io.vertx.reactivex.core.Vertx; /* Mind here */
public class TestClass {
public TestClass(Vertx vertx) {
/* Something */
}
I'm unable to provide correct instance of Vertx because only one definition of RunTestOnContext exist in package io.vertx.ext.unit.junit and produces io.vertx.core.Vertx instance, which is incompatible with io.vertx.reactivex.core.Vertx that TestClass is using. Some other test utilities, like TestContext have their equivalents in reactivex packages io.vertx.reactivex.ext.unit.TestContext, but this seems not be a case for RunTestOnContext.
The question would be how to obtain correctly io.vertx.reactivex.core.Vertx instance in test context to still ensure thread and context consistency?
The vertx-unit project has only depdendency to vertx-core. And has no dependency to vertx-rx-java. And that is understandable. Therefore the RunTestOnContext is built using the io.vertx.core.Vertx as you see.
You can downcast with the vertx.getDelegate() from io.vertx.reactivex.core.Vertx to io.vertx.core.Vertx. Bu that doesnt work in the opposite direction.
Therefore your best option is to copy the code of the RunTestOnContext and create your reactivex version of it. (The fastest way is to just change the import to io.vertx.reactivex.core.Vertx and use the vertx.getDelegate() where it is accessed.)
Get RxVertx instance by rxVertx = Vertx.vertx(); & vertx instance by rxVertx.getDelegate();. Here is a full code snippet:
#RunWith(VertxUnitRunner.class)
public class MyVerticleTest {
protected Vertx vertx;
#Before
public void setUp(TestContext context) {
rxVertx = Vertx.vertx();
vertx = rxVertx.getDelegate();
Async async = context.async(1);
RxHelper.deployVerticle(rxVertx, new MyVerticle())
.subscribe(res -> async.countDown(), err -> {
throw new RuntimeException(err);
});
}
#Test
public void my_test(TestContext context) {
Async async = context.async();
rxVertx.eventBus().rxSend("address", dataToSend)
// .flatMap()
.subscribe(receivedData -> {
// assert what you want;
async.complete();
}, context::fail);
}
#After
public void tearDown(TestContext context) {
client.close();
rxVertx.close(context.asyncAssertSuccess());
}
}
Related
Suppose I develop an extension which disallows test method names to start with an uppercase character.
public class DisallowUppercaseLetterAtBeginning implements BeforeEachCallback {
#Override
public void beforeEach(ExtensionContext context) {
char c = context.getRequiredTestMethod().getName().charAt(0);
if (Character.isUpperCase(c)) {
throw new RuntimeException("test method names should start with lowercase.");
}
}
}
Now I want to test that my extension works as expected.
#ExtendWith(DisallowUppercaseLetterAtBeginning.class)
class MyTest {
#Test
void validTest() {
}
#Test
void TestShouldNotBeCalled() {
fail("test should have failed before");
}
}
How can I write a test to verify that the attempt to execute the second method throws a RuntimeException with a specific message?
Another approach could be to use the facilities provided by the new JUnit 5 - Jupiter framework.
I put below the code which I tested with Java 1.8 on Eclipse Oxygen. The code suffers from a lack of elegance and conciseness but could hopefully serve as a basis to build a robust solution for your meta-testing use case.
Note that this is actually how JUnit 5 is tested, I refer you to the unit tests of the Jupiter engine on Github.
public final class DisallowUppercaseLetterAtBeginningTest {
#Test
void testIt() {
// Warning here: I checked the test container created below will
// execute on the same thread as used for this test. We should remain
// careful though, as the map used here is not thread-safe.
final Map<String, TestExecutionResult> events = new HashMap<>();
EngineExecutionListener listener = new EngineExecutionListener() {
#Override
public void executionFinished(TestDescriptor descriptor, TestExecutionResult result) {
if (descriptor.isTest()) {
events.put(descriptor.getDisplayName(), result);
}
// skip class and container reports
}
#Override
public void reportingEntryPublished(TestDescriptor testDescriptor, ReportEntry entry) {}
#Override
public void executionStarted(TestDescriptor testDescriptor) {}
#Override
public void executionSkipped(TestDescriptor testDescriptor, String reason) {}
#Override
public void dynamicTestRegistered(TestDescriptor testDescriptor) {}
};
// Build our test container and use Jupiter fluent API to launch our test. The following static imports are assumed:
//
// import static org.junit.platform.engine.discovery.DiscoverySelectors.selectClass
// import static org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder.request
JupiterTestEngine engine = new JupiterTestEngine();
LauncherDiscoveryRequest request = request().selectors(selectClass(MyTest.class)).build();
TestDescriptor td = engine.discover(request, UniqueId.forEngine(engine.getId()));
engine.execute(new ExecutionRequest(td, listener, request.getConfigurationParameters()));
// Bunch of verbose assertions, should be refactored and simplified in real code.
assertEquals(new HashSet<>(asList("validTest()", "TestShouldNotBeCalled()")), events.keySet());
assertEquals(Status.SUCCESSFUL, events.get("validTest()").getStatus());
assertEquals(Status.FAILED, events.get("TestShouldNotBeCalled()").getStatus());
Throwable t = events.get("TestShouldNotBeCalled()").getThrowable().get();
assertEquals(RuntimeException.class, t.getClass());
assertEquals("test method names should start with lowercase.", t.getMessage());
}
Though a little verbose, one advantage of this approach is it doesn't require mocking and execute the tests in the same JUnit container as will be used later for real unit tests.
With a bit of clean-up, a much more readable code is achievable. Again, JUnit-Jupiter sources can be a great source of inspiration.
If the extension throws an exception then there's not much a #Test method can do since the test runner will never reach the #Test method. In this case, I think, you have to test the extension outside of its use in the normal test flow i.e. let the extension be the SUT.
For the extension provided in your question, the test might be something like this:
#Test
public void willRejectATestMethodHavingANameStartingWithAnUpperCaseLetter() throws NoSuchMethodException {
ExtensionContext extensionContext = Mockito.mock(ExtensionContext.class);
Method method = Testable.class.getMethod("MethodNameStartingWithUpperCase");
Mockito.when(extensionContext.getRequiredTestMethod()).thenReturn(method);
DisallowUppercaseLetterAtBeginning sut = new DisallowUppercaseLetterAtBeginning();
RuntimeException actual =
assertThrows(RuntimeException.class, () -> sut.beforeEach(extensionContext));
assertThat(actual.getMessage(), is("test method names should start with lowercase."));
}
#Test
public void willAllowTestMethodHavingANameStartingWithAnLowerCaseLetter() throws NoSuchMethodException {
ExtensionContext extensionContext = Mockito.mock(ExtensionContext.class);
Method method = Testable.class.getMethod("methodNameStartingWithLowerCase");
Mockito.when(extensionContext.getRequiredTestMethod()).thenReturn(method);
DisallowUppercaseLetterAtBeginning sut = new DisallowUppercaseLetterAtBeginning();
sut.beforeEach(extensionContext);
// no exception - good enough
}
public class Testable {
public void MethodNameStartingWithUpperCase() {
}
public void methodNameStartingWithLowerCase() {
}
}
However, your question suggests that the above extension is only an example so, more generally; if your extension has a side effect (e.g. sets something in an addressable context, populates a System property etc) then your #Test method could assert that this side effect is present. For example:
public class SystemPropertyExtension implements BeforeEachCallback {
#Override
public void beforeEach(ExtensionContext context) {
System.setProperty("foo", "bar");
}
}
#ExtendWith(SystemPropertyExtension.class)
public class SystemPropertyExtensionTest {
#Test
public void willSetTheSystemProperty() {
assertThat(System.getProperty("foo"), is("bar"));
}
}
This approach has the benefit of side stepping the potentially awkward setup steps of: creating the ExtensionContext and populating it with the state required by your test but it may come at the cost of limiting the test coverage since you can really only test one outcome. And, of course, it is only feasible if the extension has a side effect which can be evaulated in a test case which uses the extension.
So, in practice, I suspect you might need a combination of these approaches; for some extensions the extension can be the SUT and for others the extension can be tested by asserting against its side effect(s).
After trying the solutions in the answers and the question linked in the comments, I ended up with a solution using the JUnit Platform Launcher.
class DisallowUppercaseLetterAtBeginningTest {
#Test
void should_succeed_if_method_name_starts_with_lower_case() {
TestExecutionSummary summary = runTestMethod(MyTest.class, "validTest");
assertThat(summary.getTestsSucceededCount()).isEqualTo(1);
}
#Test
void should_fail_if_method_name_starts_with_upper_case() {
TestExecutionSummary summary = runTestMethod(MyTest.class, "InvalidTest");
assertThat(summary.getTestsFailedCount()).isEqualTo(1);
assertThat(summary.getFailures().get(0).getException())
.isInstanceOf(RuntimeException.class)
.hasMessage("test method names should start with lowercase.");
}
private TestExecutionSummary runTestMethod(Class<?> testClass, String methodName) {
SummaryGeneratingListener listener = new SummaryGeneratingListener();
LauncherDiscoveryRequest request = request().selectors(selectMethod(testClass, methodName)).build();
LauncherFactory.create().execute(request, listener);
return listener.getSummary();
}
#ExtendWith(DisallowUppercaseLetterAtBeginning.class)
static class MyTest {
#Test
void validTest() {
}
#Test
void InvalidTest() {
fail("test should have failed before");
}
}
}
JUnit itself will not run MyTest because it is an inner class without #Nested. So there are no failing tests during the build process.
Update
JUnit itself will not run MyTest because it is an inner class without #Nested. So there are no failing tests during the build process.
This is not completly correct. JUnit itself would also run MyTest, e.g. if "Run All Tests" is started within the IDE or within a Gradle build.
The reason why MyTest was not executed is because I used Maven and I tested it with mvn test. Maven uses the Maven Surefire Plugin to execute tests. This plugin has a default configuration which excludes all nested classes like MyTest.
See also this answer about "Run tests from inner classes via Maven" and the linked issues in the comments.
JUnit 5.4 introduced the JUnit Platform Test Kit which allows you to execute a test plan and inspect the results.
To take a dependency on it from Gradle, it might look something like this:
testImplementation("org.junit.platform:junit-platform-testkit:1.4.0")
And using your example, your extension test could look something like this:
import org.junit.jupiter.api.extension.ExtendWith
import org.junit.jupiter.api.fail
import org.junit.platform.engine.discovery.DiscoverySelectors
import org.junit.platform.testkit.engine.EngineTestKit
import org.junit.platform.testkit.engine.EventConditions
import org.junit.platform.testkit.engine.TestExecutionResultConditions
internal class DisallowUpperCaseExtensionTest {
#Test
internal fun `succeed if starts with lower case`() {
val results = EngineTestKit
.engine("junit-jupiter")
.selectors(
DiscoverySelectors.selectMethod(ExampleTest::class.java, "validTest")
)
.execute()
results.tests().assertStatistics { stats ->
stats.finished(1)
}
}
#Test
internal fun `fail if starts with upper case`() {
val results = EngineTestKit
.engine("junit-jupiter")
.selectors(
DiscoverySelectors.selectMethod(ExampleTest::class.java, "TestShouldNotBeCalled")
)
.execute()
results.tests().assertThatEvents()
.haveExactly(
1,
EventConditions.finishedWithFailure(
TestExecutionResultConditions.instanceOf(java.lang.RuntimeException::class.java),
TestExecutionResultConditions.message("test method names should start with lowercase.")
)
)
}
#ExtendWith(DisallowUppercaseLetterAtBeginning::class)
internal class ExampleTest {
#Test
fun validTest() {
}
#Test
fun TestShouldNotBeCalled() {
fail("test should have failed before")
}
}
}
I'm unit testing a ParDo function with Apache Beam that has 1 main output and 1 sideoutput:
public class GetPubsubMessageDoFn extends DoFn<PubsubMessage, PubsubPayload.PubsubPayloadDTO> {
#ProcessElement
public void processContext(ProcessContext processContext) {
PubsubPayload pubsubPayload = new PubsubPayload(processContext.element());
processContext.output(pubsubPayload.getPayload()); //main output
processContext.output(ORIGIN_PATH_TUPLE_TAG, GCSUtils.toGSURL(pubsubPayload.getPayload().bucket, pubsubPayload.getPayload().name)); //side output
}
}
I set up a unit test class for testing the main - and side outputs:
public class GetPubsubMessageDoFnTest {
private DoFnTester<PubsubMessage, PubsubPayloadDTO> getPubsubMessageDoFn;
private Injector injector;
private final TupleTagList tags = TupleTagList.of(PUBSUB_PAYLOAD_DTO_TUPLE_TAG).and(ORIGIN_PATH_TUPLE_TAG);
#Before
public void setup() {
injector = Guice.createInjector(new GetPubsubMessageTestModule());
this.getPubsubMessageDoFn = DoFnTester.of(injector.getInstance(GetPubsubMessageDoFn.class));
this.getPubsubMessageDoFn.setOutputTags(tags); //Does not compile
}
//Tests
According to the documentation I should be able to set the side output using setOutputTags(tags) only that function does not exist on the DoFnTester class. I'm using the Google Cloud Dataflow dependency version 2.1.0, which does use a subset of Apache Beam's features, but even looking at the Apache Beam reference documentation for DoFnTester setOutputTags isn't listed (even though it's mentioned again in the intro).
These methods are not available in 2.1.0. In fact, DoFnTester is being deprecated, see https://issues.apache.org/jira/browse/BEAM-3159.
The advice is to use TestPipeline with the DirectRunner to test a ParDo on their DoFn. You can carefully control the flow of input with TestStream. See a nice blog on this topic.
I am working in Eclipse Oxygen.1a (4.7.1a) with the JUnit 5 library, and it seems like none of my annotated methods are running correctly when I run a test class using JUnitCore.
For example, if I call the following class using JUnitCore.run(TestClass.class) :
public class TestClass {
#BeforeAll
public static void beforeAll() {
System.out.println("In TestClass.beforeAll");
}
#Test
public void testMethod() {
System.out.println("In TestClass.testMethod");
}
#AfterAll
public static void afterAll() {
System.out.println("In TestClass.afterAll");
}
}
There is no output to System.out, and the Result object says that 1 test failed. Implying that none of these methods ran. I can use a JUnit Launcher to run the test class like so:
final LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request()
.selectors(selectClass(TestClass.class)).build();
final Launcher launcher = LauncherFactory.create();
launcher.execute(request);
However, this does not give me any feedback about how many tests passed/failed, it seems to just run them. I can't find much documentation on using JUnitCore with JUnit5, is there something newer that I should be working with?
You should not use JUnitCore with JUnit 5, but instead use the platform launcher API as you did in the second part of your analysis.
launcher#execute(.) does not return any value, hence you need to use a listener to aggregate the results from the execution of your tests, as per the JUnit 5 documentation 7.1.2 Executing Tests.
There is no return value for the execute() method, but you can easily use a listener to aggregate the final results in an object of your own. For an example see the SummaryGeneratingListener.
You can for instance produce a TestExecutionSummary which may indeed provide you with the information you want to collect:
final LauncherDiscoveryRequest request =
LauncherDiscoveryRequestBuilder.request()
.selectors(selectClass(TestClass.class))
.build();
final Launcher launcher = LauncherFactory.create();
final SummaryGeneratingListener listener = new SummaryGeneratingListener();
launcher.registerTestExecutionListeners(listener);
launcher.execute(request);
TestExecutionSummary summary = listener.getSummary();
long testFoundCount = summary.getTestsFoundCount();
List<Failure> failures = summary.getFailures();
...
I write JUnit5 Extension. But I cannot find way how to obtain test result.
Extension looks like this:
import org.junit.jupiter.api.extension.AfterTestExecutionCallback;
import org.junit.jupiter.api.extension.TestExtensionContext;
public class TestResultExtension implements AfterTestExecutionCallback {
#Override
public void afterTestExecution(TestExtensionContext context) throws Exception {
//How to get test result? SUCCESS/FAILED
}
}
Any hints how to obtain test result?
This work for me:
public class RunnerExtension implements AfterTestExecutionCallback {
#Override
public void afterTestExecution(ExtensionContext context) throws Exception {
Boolean testResult = context.getExecutionException().isPresent();
System.out.println(testResult); //false - SUCCESS, true - FAILED
}
}
#ExtendWith(RunnerExtension.class)
public abstract class Tests {
}
As other answers point out, JUnit communicates failed tests with exceptions, so an AfterTestExecutionCallback can be used to gleam what happened. Note that this is error prone as extension running later might still fail the test.
Another way to do that is to register a custom TestExecutionListener. Both of these approaches are a little roundabout, though. There is an issue that tracks a specific extension point for reacting to test results, which would likely be the most straight-forward answer to your question. If you can provide a specific use case, it would be great if you could head over to #542 and leave a comment describing it.
You can use SummaryGeneratingListener from org.junit.platform.launcher.listeners
It contains MutableTestExecutionSummary field, which implements TestExecutionSummary interface, and this way you can obtain info about containers, tests, time, failures etc.
You can create custom listener, for example:
Create class that extends SummaryGeneratingListener
public class ResultAnalyzer extends SummaryGeneratingListener {
#Override
public void testPlanExecutionFinished(TestPlan testPlan) {
//This method is invoked after all tests in all containers is finished
super.testPlanExecutionFinished(testPlan);
analyzeResult();
}
private void analyzeResult() {
var summary = getSummary();
var failures = summary.getFailures();
//Do something
}
}
Register listener by creating file
src\main\resources\META-INF\services\org.junit.platform.launcher.TestExecutionListener
and specify your implementation in it
path.to.class.ResultAnalyzer
Enable auto-detection of extensions, set parameter
-Djunit.jupiter.extensions.autodetection.enabled=true
And that's it!
Docs
https://junit.org/junit5/docs/5.0.0/api/org/junit/platform/launcher/listeners/SummaryGeneratingListener.html
https://junit.org/junit5/docs/5.0.0/api/org/junit/platform/launcher/listeners/TestExecutionSummary.html
https://junit.org/junit5/docs/current/user-guide/#extensions-registration-automatic
I have only this solution:
String testResult = context.getTestException().isPresent() ? "FAILED" : "OK";
It seems that it works well. But I am not sure if it will work correctly in all situations.
Fails in JUnit are propagated with exceptions. There are several exceptions, which indicate various types of errors.
So an exception in TestExtensionContext#getTestException() indicates an error. The method can't manipulate actual test results, so depending on your use case you might want to implement TestExecutionExceptionHandler, which allows you to swallow exceptions, thus changing whether a test succeeded or not.
You're almost there.
To implement a test execution callback and get the test result for logging (or generating a report) you can do the following:
import org.junit.jupiter.api.extension.AfterTestExecutionCallback;
import org.junit.jupiter.api.extension.ExtensionContext;
public class TestResultExtension implements AfterTestExecutionCallback
{
#Override
public void afterTestExecution(ExtensionContext context) throws Exception
{
// check the context for an exception
Boolean passed = context.getExecutionException().isEmpty();
// if there isn't, the test passed
String result = passed ? "PASSED" : "FAILED";
// now that you have the result, you can do whatever you want
System.out.println("Test Result: " + context.getDisplayName() + " " + result);
}
}
And then you just add the TestResultExtension using the #ExtendWith() annotation for your test cases:
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import static org.junit.jupiter.api.Assertions.assertTrue;
#ExtendWith(TestResultExtension.class)
public class SanityTest
{
#Test
public void testSanity()
{
assertTrue(true);
}
#Test
public void testInsanity()
{
assertTrue(false);
}
}
It's a good idea to extend a base test that includes the extension
import org.junit.jupiter.api.extension.ExtendWith;
#ExtendWith(TestResultExtension.class)
public class BaseTest
{}
And then you don't need to include the annotation in every test:
public class SanityTest extends BaseTest
{ //... }
I have 2 test methods, and i need to run them with different configurations
myTest() {
.....
.....
}
#Test
myTest_c1() {
setConf1();
myTest();
}
#Test
myTest_c2() {
setConf2();
myTest();
}
//------------------
nextTest() {
.....
.....
}
#Test
nextTest_c1() {
setConf1();
nextTest();
}
#Test
nextTest_c2() {
setConf2();
nextTest();
}
I cannot run them both from one config (as in code below) because i need separate methods for tosca execution.
#Test
tests_c1() {
setConf1();
myTest()
nextTest();
}
I don't want to write those 2 methods to run each test, how can i solve this?
First i thought to write custom annotation
#Test
#RunWithBothConf
myTest() {
....
}
But maybe there are any other solutions for this?
What about using Theories?
#RunWith(Theories.class)
public class MyTest{
private static enum Configs{
C1, C2, C3;
}
#DataPoints
public static Configs[] configValues = Configs.values();
private void doConfig(Configs config){
swich(config){...}
}
#Theory
public void test1(Config config){
doConfig(config);
// rest of test
}
#Theory
public void test2(Config config){
doConfig(config);
// rest of test
}
Not sure why formatting if off.
I have a similar issue in a bunch of test cases I have, where certain tests need to be run with different configurations. Now, 'configuration' in your case might be more like settings, in which case maybe this isn't the best option, but for me it's more like a deployment model, so it fits.
Create a base class containing the tests.
Extend the base class with one that represents the different configuration.
As you execute each of the derived classes, the tests in the base class will be run with the configuration setup in its own class.
To add new tests, you just need to add them to the base class.
Here is how I would approach it:
Create two test classes
The first class configures to conf1 but uses the #Before attribute trigger the setup
The second class extends the first but overrides the configure method
In the example below I have a single member variable conf. If no configuration is run it stays at its default value 0. setConf1 is now setConf in the Conf1Test class which sets this variable to 1. setConf2 is now setConf in the Conf2Test class.
Here is the main test class:
public class Conf1Test
{
protected int conf = 0;
#Before
public void setConf()
{
conf = 1;
}
#Test
public void myTest()
{
System.out.println("starting myTest; conf=" + conf);
}
#Test
public void nextTest()
{
System.out.println("starting nextTest; conf=" + conf);
}
}
And the second test class
public class Conf2Test extends Conf1Test
{
// override setConf to do "setConf2" function
public void setConf()
{
conf = 2;
}
}
When I configure my IDE to run all tests in the package I get the following output:
starting myTest; conf=1
starting nextTest; conf=1
starting myTest; conf=2
starting nextTest; conf=2
I think this gives you what. Each test only has to be written once. Each test gets run twice, once with conf1 and once with conf2
The way you have it right now seems fine to me. You aren't duplicating any code, and each test is clear and easy to understand.