I am configuring my own Spring test runner and I try to define test execution order. But I want to change it in the process depending on failure of some test cases. Each test case configuration has it's own id and onFailId that indicates id of the next test to be executed in case of failure. If the test pass, simply the test with the next id is executed. There might be some scenarios when I want to rerun or execute earlier test again.
How can I force this logic, e.g. in test failure listener?
I figured out and implemented a solution. Defining a sorting method before running test suite is not enough. All methods mentioned e.g. here How to run test methods in specific order in JUnit4? in junit4 allow setting order before execution only.
I had to overload childrenInvoker method in my runner - that is SpringJUnit4ClassRunner - by default, it calls runChlidren (private method) which runs all children tests in a loop.
If you replace this loop by your own code, you can specify additional logic after each test execution - such as choosing the next test to be run.
Related
I am running some simple performance tests with JUnit 4 and using Jenkin's perfReport so that I can generate a performance report.
While running these tests, I noticed that the test method execution includes execution time of methods annotated with JUnit 4's #before and #after.
I came across a similar post: Exclude #Before method duration from JUnit test time, however I require my output format in a JUnit-style report since the Jenkin's perfReport parses JUnit-style format only.
As such, is there a way to exclude the execution time of these annotated methods?
I solved it by performing the following:
Extending the BlockJUnit4Runner
Overriding the runChild() method, as this is where test notifiers are received
Writing a custom runLeaf() method, as this is where the test notifiers are fired to notify the test has started or stopped.
Overriding the methodInvoker() method, as this is where the test method is invoked
Creating a new Statement class that functionally, performs the same set of actions as a invokeMethod() statement object created in methodInvoker. This class however, receives the test notifier, thus allow you to control how and when the test is considered to have started.
One issue of the above approach is that you will need to extract code sections that help to run JUnit rules, in order to preserve rule execution as these methods are strangely private.
Here is comment about setting default timezone for test code in #Before method of JUnit test. But TimeZone.setDefault is static method. Can it affect other test which are run after test with #Before and TimeZone.setDefault completes successful?
There are many things to check here, it depends on how do you run the tests.
The following factors may come into considerations:
Since you've tagged "maven" in a question: Maven's surefire/failsafe plugins responsible for running the tests can run multiple tests simultaneously in 1 or many JVMs, it all depends on their configurations.
So tests may start failing sporadically during the build even if they pass locally.
#Before and #After are called before and after each test in the test case respectively. #After is called even if the test fails. So probably memorizing the default timezone and setting it back after the test should be ok, but not "re-setting" the state in an "#After" block may lead to incorrect definitions in subsequent tests.
The better approach IMHO is using java.time.Clock abstraction. See this question for examples
Another possible option is refactoring a code to use some "factory" for providing current date / time. Then in Unit Test you could instantiate this factory and "inject" it as a dependency into the code-under-test. A kind of hand-crafted Clock
It will effect the other tests (as you assumed), as it won't be reset after running a single test.
Either reset it to "normal" by an #After method, or maybe change the code to take/inject the timestamp for "now" and make the code do it's calculation from there. From my experience this will give you alot more flexibility.
I can't find any ressources explaining what exactly the difference between BeforeEachCallback and BeforeTestExecutionCallback in the JUnit Jupiter extension model is. (I am of course also interested in the "After"-variants)
To my understanding, the following timeline describes what is happening:
BeforeEach - BeforeTestExecution - Actual execution of the test - AfterTestExecution - AfterEach
I suppose that BeforeTestExecution exists so you can execute code after all the BeforeEach callbacks have been worked on but before the actual test execution. However this is still unclear to me, because everyone could just use BeforeTestExecution instead of BeforeEach and the order of execution of these callbacks is random again.
So what is BeforeTestExecution exactly for and what happens if you use this callback in multiple extensions at the same time?
The Javadocs (here and here) don't make a clear distinction between them but the JUnit5 docs include the following:
BeforeTestExecutionCallback and AfterTestExecutionCallback define the APIs for Extensions that wish to add behavior that will be executed immediately before and immediately after a test method is executed, respectively. As such, these callbacks are well suited for timing, tracing, and similar use cases. If you need to implement callbacks that are invoked around #BeforeEach and #AfterEach methods, implement BeforeEachCallback and AfterEachCallback instead.
So, if you want to wrap just the test execution without any of the setup then use BeforeTestExecutionCallback. The docs go on to suggest timing and logging test execution as possible use cases for BeforeTestExecutionCallback.
I want to test javaFX application by using junit. I checked several approaches as follow:
user TestFX api: these api works sometime on a test method, in other word, for some execution of a special test, it's work !!!
List use JFXRunner : I used test runner which is defined in this question, but it for a few of the methods to be run and when number of test methods be large(e.g. >20) , test methods waiting forever !!!
define a test runner class such as JavaFXThreadingRule which has been intruduced in this question. these approach works only for a single test and when run some test, test method go into waiting forever same as JFXRunner.
what is the problem really? by debugging test method, I found that by using lath for initializing javaFX, after running several method, remaining method will be waiting for ever. When I set a special time for awaiting method of lath, it work's correctly by it is not logically to set a constant time to test application. How can solve this problem.
for approach 2,3; I used Platform.exit() command for exiting from GUI Application; so test runner which use latch to initialize JavaFX platform wait forever. So, it was my code fault.So, calling Platform.exit() method prevents to execute remaining test methods.
I have 100 test methods. After each test, I need to perform some actions (data cleanup). Each of these 100 tests have different actions. These 100 test are not in one package or class. They are distributed.
How can I achieve this?
Right now, if a test passes, the cleanup happens, since it is part of the test. However, if the test fails, the cleanup doesn't happen. How can I make this work?
Any pointers would help.
If the tests do not have any common cleanup, you can ensure the test gets cleaned up from within the test method using a try/finally block, something like:
try {
// do test
}
finally {
// do cleanup
}
If there is any common cleanup between the test methods you could use #AfterMethod to do the cleaup.
In your case, it doesn't sound like there is much common cleanup, so the first may work better for you. It might also be worth considering if you need 100 different cleanup methods or if there can be any common setup/cleanup.
#AfterMethod would mean that you would need that every class gets this method. So you would need to go and edit each class/method. Same for #AfterGroups.
What I would suggest is to implement the IInvokedMethodListener. This would give you beforeInvocation and afterInvocation methods. In the afterInvocation method, implement your cleanup code.
Create a suite file with all of your tests which need this cleanup and specify this listener.
Hope it helps.
It sounds like you may already be using #AfterMethod to cleanup after the tests. To make #AfterMethod work after a failure, you need to use:
#AfterMethod(alwaysRun=true)
You can use groups and run a #AfterGroups somewhere. There's a #BeforeGroups as well. Setting it up with build tooling is a bit tedious and there are some interactions with IDEs as well. There's a BeforeSuite and AfterSuite as well, I believe.
An alternative could be using Spring and using the same spring context in all your tests (spring context gets reused that way). You can then do some things when the context is destroyed after your tests.