Is there a way to make tests for specific classes run in sequential order [JUnit5]?. I am using the sure-fire plugin when building the project and my tests on some classes fail, while when I run them in IntelliJ class by class they all pass. I have tried using the Order annotation, but the tests still fail which tells me that this annotation doesn't make tests run in sequential order. I want to make two of my test classes to execute their tests sequentially, while others do it in parallel, but I couldn't find proper solution on how to do that?
P.S. I have added a configuration in the maven sure-fire plugin that runs one JVM process, but this is not the desired behavior that I want to have.
Any help would be appreciated. Thanks in advance!
As described in the JUnit user guide, you can use the #TestMedhodOrder annotation to control the ordering of tests during runtime. For example, to use #Order annotations on the test methods, you would apply #TestMethodOrder(OrderAnnotation.class) at the test class level.
Having said that, you really, really, really should consider that as a last resort. Unless you explicitly and knowingly (fully understanding why) wrote the tests to require a specific ordering, you will just be masking some underlying problem in either your tests or the code they are testing.
Do yourself a favor and do your best to figure out why they fail, instead of putting the band-aid of forced ordering on the problem.
Is it possible to run a non-main method without running the main method (and thus the whole class)?
It seems like this would be useful to test your code while writing an incomplete program.
It’s called unit testing and for Java JUnit would be the most common tool for the job. It will allow you to run single classes or methods. To further help you out, a mocking tool (for example Mockito) will allow you to substitute classes with mocked implementations so you can construct a class without having to construct all the dependencies for real.
Since you mentioned your IDE, IntelliJ (and all IDE’s worth mentioning) have extensive support for JUnit. Usually there is also support for code coverage so you can even see how much of your code is being tested by your tests.
We are using JUnit to execute integration tests and also the system integration tests which rely on external test systems (not necessarily maintained by our own company).
I wonder where to put the code that checks if the system is available prior to running the test cases? So I can determine if there is a network or other issue and not one with the test itself.
JUnit allows to setup some parts of the test in JUnit-Rules. Is it a good idea to setup the service that communicates with the external system within the rule and do some basic checks ("ping") to the external system within the rule? Or to store the state and within the Test use a JUnit assume(rule.isAvailable) to avoid having the test executed?
Or would it be smarter to put this verification code in a custom JUnit Runner?
Or is there even another way to do this? (simply create some utils?)
The goal is to skip the tests if some conditions are not met since it is obvious the tests will fail. I know this indicates a bad exception handling but there is a lot of legacy code I can't change altogether.
I tried to find some articles myself but it seems the search terms ("test", "external system" and so on) are a little thankless.
thanks!
Consider using org.junit.Assume.* as described here. When assumptions fail, your tests are ignored by default. You can write a custom runner to do something else when your assumptions fail.
The key thing though is that the tests don't fail when assumptions like the availability of your external services fail.
If the ping applies to every single test in the class, I would put the ping call in the #Before method. #Before will get executed before every single test method (i.e., #JUnit -annotated methods).
If the ping does not apply to all tests in the class, then you would have to hit ping explicitly from those methods.
Is there a program out there that can allow me to find all ignored junits?
By this I mean, I have seen unit tests that use the #Ignore and tests with method name like ignore_testFoo() or xtestBar() or xxtestBar1(), which all get ignored and they are very hard to find sometimes.
I could grep for those cases, but I was wondering if there was an application that would find any of those situations automatically.
I tried using cobertura to obtain coverage on junits, to see which methods were being executed and which were not being executed, and picking apart the bad unit tests that was.
I was just wondering if there was a program or another method to obtain this information without hacking something up.
A static analysis tool would serve you well here. Checkstyle is a decent choice amongst them, it has a long list of modules, and worst case you can easily write your own module to validate any coding convention you need.
You would locate or create a module for it then execute to find any non-conforming code.
Edit
PMD looks to be an excellent choice to handle this task. It actually comes with a set of JUnit rules already built in and its very easy to combine rules or create new ones.
It should be easy to detect ignored tests using junit3 by a grep on your java test files. Find all lines matching test and parenthesis but with a method name that doesn't start by test.
For junit4, you could
* implement your own test runner by extending the default one, print out ignored tests
* build a small app that loads test classes, get all declared methods through introspect, print out those markedas ignored.
There may be a tool to do that, maybe even some runners already do, but actually it could take a few hours to have those tools from scratch if you really need them.
I'm currently re-using JUnit 4 tests from another project against my code. I obtain them directly from the other project's repository as part of my automated Ant build. This is great, as it ensures I keep my code green against the very latest version of the tests.
However, there is a subset of tests that I never expect to pass on my code. But if I start adding #Ignore annotations to those tests, I will have to maintain my own separate copy of the test implementation, which I really don't want to do.
Is there a way of excluding individual tests without modifying the Test source? Here's what I have looked at so far:
As far as I can see, the Ant JUnit task only allows you to exclude entire Test classes, not individual test methods - so that's no good for me, I need method granularity.
I considered putting together a TestSuite that uses reflection to dynamically find and add all of the original tests, then add code to explicitly remove the tests I don't want to run. But I ditched that idea when I noticed that the TestSuite API doesn't provide a method for removing tests.
I can create my own Test classes that extend the original Test classes, override the specific tests I don't want to run, and annotate them with #Ignore. I then run JUnit on my subclasses. The downside here is that if new Test classes are added to the original project, I won't pick them up automatically. I'll have to monitor for new Test classes as they are added to the original project. This is my best option so far, but doesn't feel ideal.
The only other option I can think of is to run the bad tests anyway and ignore the failures. However, these tests take a while to run (and fail!) so I'd prefer to not run them at all. Additionally, I can't see a way of telling the Ant task to ignore failures on specific test methods (again - I see how you can do it for individual Test classes, but not methods).
If you can't touch the original test at all you are going to have some serious limitations. Your overriding sounds like the best bet, but with a couple of changes:
Build the Ant tests specifically excluding the super classes, so that additional classes that you don't know about get run.
You can use the #Rule annotation (new to JUnit 4.7) to know what test is being run and abort it (by returning an empty Statement implementation) rather than overriding specific methods, giving you more flexibility in knowing whether or not to avoid the test. The only problem with this method is that you can't stop the #Before methods from running using this method, which may be slow. If that is a problem (and you really can't touch the tests) then #Ignore in the overridden method is the only thing I can think of.
If, however, you can touch those tests, some additional options open up:
You could run them with a custom runner by specifying the #RunWith tag on the class. This runner would just pass over execution to the standard runner (JUnit4.class) in that project, but in your project (via a system property or some other mechanism) would inspect the test name and not run a test. This has the advantage of being the least intrusive, but the most difficult to implement (runners are hairy beasts, one of the stated goals of #Rule was to eliminate most of the need to make them).
Another is to make an assumeThat statement on the test that would check some configuration setting that would be true if that test should run. That would actually involve injecting right into the test, which is most likely a deal breaker in anything remotely labeled a "separate project."
It doesn't help you now, but TestNG supports this sort of ability.
OK, this is a rather heavyweight solution, but don't throw things at me if it sounds ridiculous.
The core of Junit4 is the org.junit.runner.Runner class, and its various subclasses, most importantly org.junit.runners.Suite. These runners determine what the tests are for a given test class, using things like #Test and #Ignore.
It's quite easy to create custom implementations of a runner, and normally you would hook them up by using the #RunWith annotation on your test classes, but obviously that's not an option for you.
However, in theory you could write your own Ant task, perhaps based upon the standard Ant Junit task, which takes your custom test runner and uses it directly, passing each test class to it in turn. Your runner implementation could use an external config file which specifies which test methods to ignore.
It'd be quite a lot of work, and you'd have to spend time digging around in the prehistoric Ant Junit codebase to find out how it works. The investment in time may be worth it, however.
It's just a shame that the Junit Ant task doesn't provide a mechanism to specify the test Runner, that would be ideal.
A possibility I can think of to achieve what you want with the stated constraints is to use bytecode modification. You could keep a list of classes and methods to ignore in a separate file, and patch the bytecode of the test classes as you load them to remove this methods altogether.
If I am not mistaken, JUnit uses reflection to find the test methods to execute. A method rename operation would then allow you to remove these methods before JUnit finds them. Or the method can be modified to return immediately, without performing any operation.
A library like BCEL can be used to modify the classes when loaded.
If you want to run only a subset of the tests it sounds like that class has more than one responsibility and should be refactored down. Alternately the test class could be broken apart so that the original project had all the tests but on one or more classes(I'm guessing some of the tests are really integration tests and touch the database or network) and you could exclude the class(es) you didn't want.
If you can't do any of that, your option of overriding is probably best. Take the process of whenever you need to ignore some methods you extend that class and add it to your Ant exclude list. That way you can exclude what you can't pass and will still pull in all new tests (methods you didn't override and new test classes) without modifying your build.
If the unwanted tests are in specific classes/packages, you could use a fileset exclude in Ant to exclude them during import.
Two options
Work with the owner of the borrowed tests to extract your ones into a separate class you both can share.
Create your own test class which proxies the test class you want to use. For each method you want to include have a method in your class. You'll need to construct an instance of the test class you are calling and do before and after methods too if they're in the original.
Create a custom Junit runner based on blockjunitrunner and use it to filter out or in the tests you want.