I'm currently re-using JUnit 4 tests from another project against my code. I obtain them directly from the other project's repository as part of my automated Ant build. This is great, as it ensures I keep my code green against the very latest version of the tests.
However, there is a subset of tests that I never expect to pass on my code. But if I start adding #Ignore annotations to those tests, I will have to maintain my own separate copy of the test implementation, which I really don't want to do.
Is there a way of excluding individual tests without modifying the Test source? Here's what I have looked at so far:
As far as I can see, the Ant JUnit task only allows you to exclude entire Test classes, not individual test methods - so that's no good for me, I need method granularity.
I considered putting together a TestSuite that uses reflection to dynamically find and add all of the original tests, then add code to explicitly remove the tests I don't want to run. But I ditched that idea when I noticed that the TestSuite API doesn't provide a method for removing tests.
I can create my own Test classes that extend the original Test classes, override the specific tests I don't want to run, and annotate them with #Ignore. I then run JUnit on my subclasses. The downside here is that if new Test classes are added to the original project, I won't pick them up automatically. I'll have to monitor for new Test classes as they are added to the original project. This is my best option so far, but doesn't feel ideal.
The only other option I can think of is to run the bad tests anyway and ignore the failures. However, these tests take a while to run (and fail!) so I'd prefer to not run them at all. Additionally, I can't see a way of telling the Ant task to ignore failures on specific test methods (again - I see how you can do it for individual Test classes, but not methods).
If you can't touch the original test at all you are going to have some serious limitations. Your overriding sounds like the best bet, but with a couple of changes:
Build the Ant tests specifically excluding the super classes, so that additional classes that you don't know about get run.
You can use the #Rule annotation (new to JUnit 4.7) to know what test is being run and abort it (by returning an empty Statement implementation) rather than overriding specific methods, giving you more flexibility in knowing whether or not to avoid the test. The only problem with this method is that you can't stop the #Before methods from running using this method, which may be slow. If that is a problem (and you really can't touch the tests) then #Ignore in the overridden method is the only thing I can think of.
If, however, you can touch those tests, some additional options open up:
You could run them with a custom runner by specifying the #RunWith tag on the class. This runner would just pass over execution to the standard runner (JUnit4.class) in that project, but in your project (via a system property or some other mechanism) would inspect the test name and not run a test. This has the advantage of being the least intrusive, but the most difficult to implement (runners are hairy beasts, one of the stated goals of #Rule was to eliminate most of the need to make them).
Another is to make an assumeThat statement on the test that would check some configuration setting that would be true if that test should run. That would actually involve injecting right into the test, which is most likely a deal breaker in anything remotely labeled a "separate project."
It doesn't help you now, but TestNG supports this sort of ability.
OK, this is a rather heavyweight solution, but don't throw things at me if it sounds ridiculous.
The core of Junit4 is the org.junit.runner.Runner class, and its various subclasses, most importantly org.junit.runners.Suite. These runners determine what the tests are for a given test class, using things like #Test and #Ignore.
It's quite easy to create custom implementations of a runner, and normally you would hook them up by using the #RunWith annotation on your test classes, but obviously that's not an option for you.
However, in theory you could write your own Ant task, perhaps based upon the standard Ant Junit task, which takes your custom test runner and uses it directly, passing each test class to it in turn. Your runner implementation could use an external config file which specifies which test methods to ignore.
It'd be quite a lot of work, and you'd have to spend time digging around in the prehistoric Ant Junit codebase to find out how it works. The investment in time may be worth it, however.
It's just a shame that the Junit Ant task doesn't provide a mechanism to specify the test Runner, that would be ideal.
A possibility I can think of to achieve what you want with the stated constraints is to use bytecode modification. You could keep a list of classes and methods to ignore in a separate file, and patch the bytecode of the test classes as you load them to remove this methods altogether.
If I am not mistaken, JUnit uses reflection to find the test methods to execute. A method rename operation would then allow you to remove these methods before JUnit finds them. Or the method can be modified to return immediately, without performing any operation.
A library like BCEL can be used to modify the classes when loaded.
If you want to run only a subset of the tests it sounds like that class has more than one responsibility and should be refactored down. Alternately the test class could be broken apart so that the original project had all the tests but on one or more classes(I'm guessing some of the tests are really integration tests and touch the database or network) and you could exclude the class(es) you didn't want.
If you can't do any of that, your option of overriding is probably best. Take the process of whenever you need to ignore some methods you extend that class and add it to your Ant exclude list. That way you can exclude what you can't pass and will still pull in all new tests (methods you didn't override and new test classes) without modifying your build.
If the unwanted tests are in specific classes/packages, you could use a fileset exclude in Ant to exclude them during import.
Two options
Work with the owner of the borrowed tests to extract your ones into a separate class you both can share.
Create your own test class which proxies the test class you want to use. For each method you want to include have a method in your class. You'll need to construct an instance of the test class you are calling and do before and after methods too if they're in the original.
Create a custom Junit runner based on blockjunitrunner and use it to filter out or in the tests you want.
Related
Is there a way to make tests for specific classes run in sequential order [JUnit5]?. I am using the sure-fire plugin when building the project and my tests on some classes fail, while when I run them in IntelliJ class by class they all pass. I have tried using the Order annotation, but the tests still fail which tells me that this annotation doesn't make tests run in sequential order. I want to make two of my test classes to execute their tests sequentially, while others do it in parallel, but I couldn't find proper solution on how to do that?
P.S. I have added a configuration in the maven sure-fire plugin that runs one JVM process, but this is not the desired behavior that I want to have.
Any help would be appreciated. Thanks in advance!
As described in the JUnit user guide, you can use the #TestMedhodOrder annotation to control the ordering of tests during runtime. For example, to use #Order annotations on the test methods, you would apply #TestMethodOrder(OrderAnnotation.class) at the test class level.
Having said that, you really, really, really should consider that as a last resort. Unless you explicitly and knowingly (fully understanding why) wrote the tests to require a specific ordering, you will just be masking some underlying problem in either your tests or the code they are testing.
Do yourself a favor and do your best to figure out why they fail, instead of putting the band-aid of forced ordering on the problem.
I am looking for a way to change the order in which the test methods are executed inside of a test class using commandline. Three important remarks:
Yes, I am fully aware of issues about test order execution, and that running tests in a fixed order may cause dependency between tests, I am aware of mocking techniques, tear down methods etc. However, I am doing a research where doing this is necessary, I am not developing software.
I am looking for a way to change the order of test methods NOT classes.
I am looking for a way to dynamically induce this, meaning something like #TestMethodOrder(MethodOrderer.Alphanumeric.class) would not be ideal. Being able to change the order through commandline arguments would be ideal.
Hi recently we have done the unit testing for the entire project using mockito framework. My project is on Java spring rest project. But the coverage is below 35%. Need to improve the unit testing coverage.
1. Want to remove the unneccesay package from the code coverage, like test packages and beans class
2. Do we need to write the unit test case for the controller class and generated class from the tools.
I will be very grateful, if you can help me.
Test classes and packages are not counted in test coverage, if they were, how do you test the code that tests the code etc..
When you run coverage it should only run over src/main/Java etc.
Controller classes should be tested, when you call a method, is the correct delegated class and method called?
Generated classes, if from xml using jaxb etc do not need to be explicitly tested if they are just plain old Java objects with getter, setters and fields. It's likely they will be tested via another class that uses these objects and calls their methods. These classes will be generated/compiled before your tests run so will be available- make sure you aren't committing generated classes to your code repository.
You may want to consider testing the behaviour of third party libraries you depend on. This way you can instantly see if any updates to libraries may cause issues, but this should be from a high level.
You need to not start ignoring classes, but instead run a code coverage tool and see what's uncovered, and get those unit tests up to par. Test your failure cases, too!
The test packages should NOT be part of the 35%, so removing them is not going to help. None of the coverage tools I know of consider test packages.
As for generated classes - most likely not, but again, run a code coverage tool and you'll quickly see what needs to be covered.
If you use IntelliJ then coverage tools are built in:
https://www.jetbrains.com/help/idea/2016.2/code-coverage.html
Eclipse uses plugins, one is:
http://www.eclemma.org/
There are more if you google.
I prefer Sonar:
http://www.sonarqube.org/
But whatever tool you use, that's the way to go.
I'd like to implement a better system to filter tests to run under JUnit in Eclipse. I'd like the #Categories, but not have to specify the list of classes in a #SuiteClasses({}) annotation , since I think that reduces the value, increases the amount of manual maintenance we have to do.
Is there some way to hook into the process where Eclipse runs JUnit test, to be able to do custom filtering on all the classes with a #Test in it? E.g. a class like this:
#CustomFilteredTesting
public class TheCustomFilteredTestRun {
public boolean includeThisTestClass(Class<?> klass) {
// e.g. test whether klass is in a package or subsystem
// or a subtype of some interface.
}
}
Any help appreciated, -j
Eclipse uses the default JUnit code to run the tests, so you only need to teach JUnit your new trick and Eclipse will be able to do it.
That out of the way: How can you split tests into suites without #SuiteClasses?
Tough. From my experience, there are several solutions:
Write a test that reads all suites and tries to find all tests and then makes sure that each test is in one suite. This test will let you know when you forgot to add a new test to the suites.
Nice: No JUnit magic involved, will keep you safe if you forget something (like to add a #Category to a new test).
Drawback: Needs quite some ugly File reading/code parsing/reflection mojo.
You can write your own test runner by extending BlockJUnit4ClassRunner. That would allow you to add your own, custom annotations to classify tests and run them.
Add assumptions. org.junit.Assume is a nice little beast to silently fail a test. The test won't run (or it will run until the first assumption fails) but it also won't log an error. It's a bit like a smart #Ignore
Write setup code that determines which tests to run and put assumeThat() in your code.
I actually prefer to use suites because:
The "do I have all tests" test tells when I'm wrong
I have a single place where I can see which test goes where
I can easily write more tests to make sure tests aren't repeated in suites
Consider using ClassPathSuite: http://johanneslink.net/projects/cpsuite.jsp. We use it and define a Suite as follows:
#RunWith(ClasspathSuite.class)
#ExcludeBaseTypeFilter({ NotUnitTestable.class })
public class AllTests {
}
Exclusion is not possible using Annotations so we simply defined a marker interface (NotUnitTestable). We start our tests this way both in Eclipse and in our command line build using the JUnit ANT integration.
Use TestNG. It has test groups, you can see an example on front page. It has better support in parametrized tests. The integration with Spring through spring-test is better.
Is there a program out there that can allow me to find all ignored junits?
By this I mean, I have seen unit tests that use the #Ignore and tests with method name like ignore_testFoo() or xtestBar() or xxtestBar1(), which all get ignored and they are very hard to find sometimes.
I could grep for those cases, but I was wondering if there was an application that would find any of those situations automatically.
I tried using cobertura to obtain coverage on junits, to see which methods were being executed and which were not being executed, and picking apart the bad unit tests that was.
I was just wondering if there was a program or another method to obtain this information without hacking something up.
A static analysis tool would serve you well here. Checkstyle is a decent choice amongst them, it has a long list of modules, and worst case you can easily write your own module to validate any coding convention you need.
You would locate or create a module for it then execute to find any non-conforming code.
Edit
PMD looks to be an excellent choice to handle this task. It actually comes with a set of JUnit rules already built in and its very easy to combine rules or create new ones.
It should be easy to detect ignored tests using junit3 by a grep on your java test files. Find all lines matching test and parenthesis but with a method name that doesn't start by test.
For junit4, you could
* implement your own test runner by extending the default one, print out ignored tests
* build a small app that loads test classes, get all declared methods through introspect, print out those markedas ignored.
There may be a tool to do that, maybe even some runners already do, but actually it could take a few hours to have those tools from scratch if you really need them.