How to build up test cases in junit? - java

I'm coming from a Perl background where I used Test::More to handle unit testing. Using that framework, I knew the order in which the tests took place and could rely on that, which I understand is not encouraged with the JUnit framework. I've seen several ways to get around this, but I want to understand the proper/intended way of doing things.
In my Perl unit testing I would build up tests, knowing that if test #3 passed, I could make some assumptions in further tests. I don't quite see how to structure that in the JUnit world so that I can make every test completely independent.
For example, suppose I have a class that parses a date from a string. Methods include:
parse a simple date (YYYY-MM-DD)
parse a simple date with alternate separator (YYYY_MM_DD or YYYY/MM/DD)
parse a date with a string for a month name (YYYY-MON-DD)
parse a date with a string month name in a different language
and so on
I usually write my code to focus as many of the externally-accessible methods into as few core methods as possible, re-using as much code as possible (which is what most of us would do, I'm sure). So, let's say I have 18 different tests for the first method, 9 that are expected to pass and 9 that throw an exception. For the second method, I only have 3 tests, one each with the separators that work ('_' & '/') and one with a separator that doesn't work ('*') which is expected to fail. I can limit myself to the new code being introduced because I already know that the code properly handles the standard boundary conditions and common errors, because the first 18 tests already passed.
In the Perl world, if test #20 fails, I know that it's probably something to do with the specific separator, and is not a general date parsing error because all of those tests have already passed. In the JUnit world, where tests run in a random order, if test #20 fails, I don't know if it's because of a general date parsing issue or because of the separator. I'd have to go and see which other ones failed and then make some assumptions there. That's not too hard to do, of course, but maybe in a bigger, more complex class, it would be more difficult to do.
How do other people deal with building up a set of tests? Should I put each and every test in a separate class and use a test suite? That seems tedious. (And before someone suggests that I put the first18 in one class and the second 3 in another, and use a test suite for just those groupings, let's pretend that all 18 of the early tests build on each other, too).
And, again, I know there are ways around this (FixedMethodOrder in JUnit 4.11+ or JUnit-HierarchicalContextRunner) but I want to understand the paradigm as its intended to be used.

In the JUnit world, where tests run in a random order, if test #20 fails, I don't know if it's because of a general date parsing issue or because of the separator. I'd have to go and see which other ones failed and then make some assumptions there.
Yes that is correct. If something in your code is broken then multiple tests may fail. That is a good thing. Use intent revealing test method names and possibly use the optional String message parameter in the JUnit assertions to explain what exactly failed the test.
How do other people deal with building up a set of tests? Should I put each and every test in a separate class and use a test suite?
The general convention is one test class per source class. Depending on what build tool you are using, you may or may not need to use test suites. If you are using Ant, you probably need to collect the tests into test suites, but if you are using Maven, the test plugins for maven will find all your test classes for you so you don't need suites.
I also want to point out that you should be coding to Java interfaces as much as possible. If you are testing class C that depends on an implementation of interface I, then you should mock your I implementation in your C test class so that C is tested in isolation. Your mock I should follow what the interface is supposed to do. This also keeps the number of failing tests down. If there is a bug in your real I implementation, then only your I tests should fail, the C tests should still all pass (since you are testing it against a fake but working I implementation)

Don't worry about suites yet. You'll know when you need them. I've only had to use them a handful of times, and I'm not entirely sold on their usefulness...but I leave that decision up to you.
To the meat of your question - the conventional way with JUnit tests is to neither know nor depend on the order of execution of your tests; this ensures that your tests are not run-order dependent, and if they are, something is wrong with your tests* and validation.
The main core concept behind unit tests is that they test a unit of code - as simple as a single function. If you're attempting to test five different things at once, your test is far too large, and should be broken out. If the method you're testing is monolithic in nature, and difficult to test, it should be refactored and broken out into different slices of responsibility.
Tests that exercise a larger flow are better suited for integration-style tests, which tend to be written as unit tests, but aren't actually unit tests.
I've not run into a scenario in which, if I knew that if a certain test failed, I could expect different behavior in the other tests. I've never thought that such a thing was necessary to be noted, since the only thing I care about in my unit test is how that unit of code behaves given a certain input.
Keep your tests small and simple to understand; the test should only make one assertion about the result (or a general assertion of the state of your result).
*: That's not to say that it's completely broken, but those sorts of tests should be fixed sooner rather than later.

Related

Using application classes as utilities in unit tests

This is a question about best practices for unit tests. Let's say I have a ClassA that does some object transforms (from CSVs to Java beans, using external library). Then I have ClassB that needs these transformed Java beans to execute a calculation.
When writing a unit test for ClassB, is it acceptable to use ClassA's transform method to get these transformed bean objects? In other words, use an application class as utility in unit test.
The other option would be to write a util test method which would have same code as ClassA (the transformation is done using an external library, so it's easy to use it in both application class and test utility class)
You are asking about best practices - thus I will not answer for your particular example (for which others have already given you some hints how to handle it), but I will look at the general question: Is it acceptable to use application classes as utility classes within test code.
There can be situations where this is the design alternative which is the best tradeoff between all goals, but it depends on the specific situation. The competing goals are the following:
Avoiding code duplication, test code readability: Creating complex objects from test code can make the test code complex and maintenance intensive. Re-using other application code to create these objects can simplify the test code, which otherwise would somehow duplicate the code for object creation.
Trustworthiness of the test code: If your test code depends on other components, the test results become less trustworthy - failing tests may be due to the other components, passing tests can be due to the created objects having different properties than intended in the test. Note that the risk of this happening can be reduced if the functions used as utilities are themselves thoroughly tested.
Completeness of the test suite: When using utility functions to create input objects there is a risk that only those input objects are used during tests that the utility functions are able to create. That is, if the utility functions will never create malformed objects, then you might limit your testing such that you never test your SUT with malformed objects. If you are aware of this risk, you can certainly still have some additional tests where malformed objects are used - in these tests you can, however, not use the utility functions.
Complexity of the test setup, execution time etc.: If the utility functions themselves have further dependencies, or they compile, link or execute slowly, they bring all these disadvantages into the test code. Thus, only certain application functions will be suitable to be used in test code.
An example that comes to my mind is testing some function that receives a parse tree of an algebraic expression as input and shall evaluate that expression. Assume that the same application also has a parse function that creates parse trees from strings.
Using the parse function within unit-tests of the evaluation function would IMO be a good option here: The expression strings would be nicely readable in the unit-tests, the parse function will likely only have dependencies that the test code would otherwise need by itself anyway (like, classes for nodes and edges of the parse tree). The parsing can be expected to have no significant impact on test performance. Plus, if the parse function is well tested its use within the test code will not impact the trustworthiness in a significant way. It may nevertheless still be necessary to have some extra tests where malformed parse trees are intentionally created to test the behaviour of the evaluation function for such cases.
when you unit test a method
you prepare the data ( mocks, hardcoded etc ),
you call the method under test
you evaluate the result
what you do not do is call other things, utility methods etc.
Why?
Because that means that your code has dependencies on other things and it does not run in isolation.
It also means that your test has too much knowledge about the code and that's a bad things especially when you start doing refactoring. Too much knowledge about the code means that as soon as you change the code you will break most tests which defeats one of of their purposes which is to make refactoring ( and developers life ) easier.

Should I follow One Test Class per Class Pattern Or usecase Pattern to write test cases

I am learner in writing Junit Test Cases. I have seen writing Junit cases Pattern that We usually make test class for each class indivisually by their name and write test cases for each method of that class in its respective class so that maximum code coverage can occur.
What I was thinking If I make test cases for my feature that would be better choice because In future any number of methods Signature changes I don't have to change or create again unnecessary test cases for those modified methods or newly created. Because that moment I would have certain test cases for my developed feature. So my test cases are running fine for particular feature then I can be sure in minimum number of test cases code that everything is fine.
By keeping this I don't have to write test cases for each and every methods of each class. Is it a good way?
Well, test cases are written for a reason. Each and every methods have to be working properly as expected. If you only do test cases for the feature level, how do you find exactly where the error occurred and how confidently you can ship your code to next level?
The better approach will be to do unit test cases for each class and do an integration test to make sure everything works good.
We found success in utilizing both. By default we use one per class. But when particular use-cases come up, e.g. use-cases that involve multiple classes or use cases where the existing boiler plate testing code prevents the use case from being properly tested, then we would create a test class for that scenario.
by keeping this I don't have to write test cases for each and every
methods of each class. Is it a good way?
In this case you write only integration tests and no unit tests.
Writing tests for use cases is really nice but it is not enough because it is hard to cover all cases of all methods invoked in an integration test because there may have a very important number of branches while it is much easier in an unit test.
Besides, a use case test may be successful for bad reasons : thanks to side effects between multiple methods invoked.
By writing an unit test you protect yourself against this kind of issue.
Definitively, unit and integration tests are are not opposed but complementary. So you have to write both to get a robust application.

Writing junits in Test Driven Development before writing actual code?

Folks it is always said in TDD that
we should write junits even before we write the actual code.
Somehow I am not able to understand this in right spirit. I hope what it means is that you just write empty methods wih right signatures and your test case is expected to fail initially
Say in TDD approach i need to get the list of customers.
As per my understanding i will write the empty method like below
public List<CustomerData> getCustomers(int custId){
return null;
}
Now i will write junit test case where i will check the size as 10(that i am eactually expecting). Is this Right?
Basically my question is in TDD, How we can write junit test case before writing actual code?
I hope what it means is that you just write empty methods wih right signatures
Yes. And with most modern IDEs, if you write a method name which does not exist in your test, they will create a stub for you.
Say in TDD approach i need to get the list of customers. whats the right way to proceed?
Your example is not quite there. You want to test for a 0-length array, but you already return it: you should first return null, the test will obviously fail.
Then modify the method so that the test succeeds.
Then create a test method for customer add. Test fails. Fix it. Rinse. Repeat.
So, basically: with TDD, you start and write test that you KNOW will fail, and then fix your code so that they work.
Recommended read.
Often you'll write the test alongside the skeleton of the code. Initially you can write a non-functional implementation (e.g. throw an UnsupportedOperationException) and that will trigger a test failure. Then you'd flesh out the implementation until finally your test passes.
You need to be pragmatic about this. Obviously you can't compile your test until at least your unit under test compiles, and so you have to do a minimal amount of implementation work alongside your test.
Check out this recent Dr Dobbs editoral, which discusses exactly this point and the role of pragmatism around this, especially by the mavens of this practise (Kent Beck et al)
A key principle of TDD is that you write no code without first writing
a failing unit test. But in fact, if you talk to the principal
advocates of TDD (such as Kent Beck, who popularized the technique,
and Bob Martin, who has taught it to thousands of developers), you
find that both of them write some code without writing tests first.
They do not — I should emphasize this — view these moments as lapses
of faith, but rather as the necessary pragmatism of the intelligent
developer.
That's partly right.
Using an IDE (Eclipse, IntelliJ) you can create a test. In that test invoke a method (that does not exist) and using a refactoring tool create a method with the proper signature.
That's a trick that makes working with TDD easier and more fun.
According to Now i will write junit test case where i will check the size as 0. Is this Right? you should write a test that fails, and the provide proper implementation.
I think go write the test first, Think about the signature of the function while writing the test.
It's the same as writing the signature and then writing the test, But when inventing the signature of the function while writing the test will be helpful since you will have all information about the responsibility of the function and you will be able to come up with the proper signature.

When to use Mockito.verify()?

I write jUnit test cases for 3 purposes:
To ensure that my code satisfies all of the required functionality, under all (or most of) the input combinations/values.
To ensure that I can change the implementation, and rely on JUnit test cases to tell me that all my functionality is still satisfied.
As a documentation of all the use cases my code handles, and act as a spec for refactoring - should the code ever need to be rewritten. (Refactor the code, and if my jUnit tests fail - you probably missed some use case).
I do not understand why or when Mockito.verify() should be used. When I see verify() being called, it is telling me that my jUnit is becoming aware of the implementation. (Thus changing my implementation would break my jUnits, even though my functionality was unaffected).
I'm looking for:
What should be the guidelines for appropriate usage of Mockito.verify()?
Is it fundamentally correct for jUnits to be aware of, or tightly coupled to, the implementation of the class under test?
If the contract of class A includes the fact that it calls method B of an object of type C, then you should test this by making a mock of type C, and verifying that method B has been called.
This implies that the contract of class A has sufficient detail that it talks about type C (which might be an interface or a class). So yes, we're talking about a level of specification that goes beyond just "system requirements", and goes some way to describing implementation.
This is normal for unit tests. When you are unit testing, you want to ensure that each unit is doing the "right thing", and that will usually include its interactions with other units. "Units" here might mean classes, or larger subsets of your application.
Update:
I feel that this doesn't apply just to verification, but to stubbing as well. As soon as you stub a method of a collaborator class, your unit test has become, in some sense, dependent on implementation. It's kind of in the nature of unit tests to be so. Since Mockito is as much about stubbing as it is about verification, the fact that you're using Mockito at all implies that you're going to run across this kind of dependency.
In my experience, if I change the implementation of a class, I often have to change the implementation of its unit tests to match. Typically, though, I won't have to change the inventory of what unit tests there are for the class; unless of course, the reason for the change was the existence of a condition that I failed to test earlier.
So this is what unit tests are about. A test that doesn't suffer from this kind of dependency on the way collaborator classes are used is really a sub-system test or an integration test. Of course, these are frequently written with JUnit too, and frequently involve the use of mocking. In my opinion, "JUnit" is a terrible name, for a product that lets us produce all different types of test.
David's answer is of course correct but doesn't quite explain why you would want this.
Basically, when unit testing you are testing a unit of functionality in isolation. You test whether the input produces the expected output. Sometimes, you have to test side effects as well. In a nutshell, verify allows you to do that.
For example you have bit of business logic that is supposed to store things using a DAO. You could do this using an integration test that instantiates the DAO, hooks it up to the business logic and then pokes around in the database to see if the expected stuff got stored. That's not a unit test any more.
Or, you could mock the DAO and verify that it gets called in the way you expect. With mockito you can verify that something is called, how often it is called, and even use matchers on the parameters to ensure it gets called in a particular way.
The flip side of unit testing like this is indeed that you are tying the tests to the implementation which makes refactoring a bit harder. On the other hand, a good design smell is the amount of code it takes to exercise it properly. If your tests need to be very long, probably something is wrong with the design. So code with a lot of side effects/complex interactions that need to be tested is probably not a good thing to have.
This is great question!
I think the root cause of it is the following, we are using JUnit not only for unit testing. So the question should be splited up:
Should I use Mockito.verify() in my integration (or any other higher-than-unit testing) testing?
Should I use Mockito.verify() in my black-box unit-testing?
Should I use Mockito.verify() in my white-box unit-testing?
so if we will ignore higher-than-unit testing, the question can be rephrased "Using white-box unit-testing with Mockito.verify() creates great couple between unit test and my could implementation, can I make some "grey-box" unit-testing and what rules of thumb I should use for this".
Now, let's go through all of this step-by-step.
*- Should I use Mockito.verify() in my integration (or any other higher-than-unit testing) testing?*
I think the answer is clearly no, moreover you shouldn't use mocks for this. Your test should be as close to real application as possible. You are testing complete use case, not isolated part of the application.
*black-box vs white-box unit-testing*
If you are using black-box approach what is you really doing, you supply (all equivalence classes) input, a state, and tests that you will receive expected output. In this approach using of mocks in general is justifies (you just mimic that they are doing the right thing; you don't want to test them), but calling Mockito.verify() is superfluous.
If you are using white-box approach what is you really doing, you're testing the behaviour of your unit. In this approach calling to Mockito.verify() is essential, you should verify that your unit behaves as you're expecting to.
rules of thumbs for grey-box-testing
The problem with white-box testing is it creates a high coupling. One possible solution is to do grey-box-testing, not white-box-testing. This is sort of combination of black&white box testing. You are really testing the behaviour of your unit like in white-box testing, but in general you make it implementation-agnostic when possible. When it is possible, you will just make a check like in black-box case, just asserts that output is what is your expected to be. So, the essence of your question is when it is possible.
This is really hard. I don't have a good example, but I can give you to examples. In the case that was mentioned above with equals() vs equalsIgnoreCase() you shouldn't call Mockito.verify(), just assert the output. If you couldn't do it, break down your code to the smaller unit, until you can do it. On the other hand, suppose you have some #Service and you are writting #Web-Service that is essentially wrapper upon your #Service - it delegates all calls to the #Service (and making some extra error handling). In this case calling to Mockito.verify() is essential, you shouldn't duplicate all of your checks that you did for the #Serive, verifying that you're calling to #Service with correct parammeter list is sufficient.
I must say, that you are absolutely right from a classical approach's point of view:
If you first create (or change) business logic of your application and then cover it with (adopt) tests (Test-Last approach), then it will be very painful and dangerous to let tests know anything about how your software works, other than checking inputs and outputs.
If you are practicing a Test-Driven approach, then your tests are the first to be written, to be changed and to reflect the use cases of your software's functionality. The implementation depends on tests. That sometimes mean, that you want your software to be implemented in some particular way, e.g. rely on some other component's method or even call it a particular amount of times. That is where Mockito.verify() comes in handy!
It is important to remember, that there are no universal tools. The type of software, it's size, company goals and market situation, team skills and many other things influence the decision on which approach to use at your particular case.
In most cases when people don't like using Mockito.verify, it is because it is used to verify everything that the tested unit is doing and that means you will need to adapt your test if anything changes in it.
But, I don't think that is a problem. If you want to be able to change what a method does without the need to change it's test, that basically means you want to write tests which don't test everything your method is doing, because you don't want it to test your changes. And that is the wrong way of thinking.
What really is a problem, is if you can modify what your method does and a unit test which is supposed to cover the functionality entirely doesn't fail. That would mean that whatever the intention of your change is, the result of your change isn't covered by the test.
Because of that, I prefer to mock as much as possible: also mock your data objects. When doing that you can not only use verify to check that the correct methods of other classes are called, but also that the data being passed is collected via the correct methods of those data objects. And to make it complete, you should test the order in which calls occur.
Example: if you modify a db entity object and then save it using a repository, it is not enough to verify that the setters of the object are called with the correct data and that the save method of the repository is called. If they are called in the wrong order, your method still doesn't do what it should do.
So, I don't use Mockito.verify but I create an inOrder object with all mocks and use inOrder.verify instead. And if you want to make it complete, you should also call Mockito.verifyNoMoreInteractions at the end and pass it all the mocks. Otherwise someone can add new functionality/behavior without testing it, which would mean after while your coverage statistics can be 100% and still you are piling up code which isn't asserted or verified.
As some people said
Sometimes you don't have a direct output on which you can assert
Sometimes you just need to confirm that your tested method is sending the correct indirect outputs to its collaborators (which you are mocking).
Regarding your concern about breaking your tests when refactoring, that is somewhat expected when using mocks/stubs/spies. I mean that by definition and not regarding a specific implementation such as Mockito.
But you could think in this way - if you need to do a refactoring that would create major changes on the way your method works, it is a good idea to do it on a TDD approach, meaning you can change your test first to define the new behavior (that will fail the test), and then do the changes and get the test passed again.

Is it possible to programmatically generate JUnit test cases and suites?

I have to write a very large test suite for a complex set of business rules that are currently captured in several tabular forms (e.g., if parameters X Y Z are such and such, the value should be between V1 and V2). Each rule has a name and its own semantics.
My end goal is to have a test suite, organized into sub test suites, with a test case to each rule.
One option is to actually hard code all these rules as tests. That is ugly, time consuming, and inflexible.
Another is to write a Python script that would read the rule files and generate Java classes with the unit tests. I'd rather avoid this if I can. Another variation would be to use Jython.
Ideally, however I would like to have a test suite that would read the files, and would then define sub-suites and tests within them. Each of these tests might be initialized with certain values taken from the table files, run fixed entry points in our system, and then call some validator function on the results based on the expected value.
Is there a reasonable way to pull this off using only Java?
Update: I may have somewhat simplified our kind of rules. Some of them are indeed tabular (excel style), others are more fuzzy. The general question though remains as I'm probably not the first person to have this problem.
Within JUnit 4 you will want to look at the Parameterized runner. It was created for the purpose you describe (data driven tests). It won't organize them into suites however.
In Junit 3 you can create TestSuites and Tests programatically. The answer is in Junit Recipes, which I can expand if you need it (remember that JUnit 4 can run Junit 3 tests).
Have you considered using FIT for that?
You seem to have the tables already ready, and "business rules" sounds like "business people write them using excel".
FIT is a system for checking tests based on tables with input->expected output mappings, and a open source java library for running those tests is available.
We tried FIT and decided to go with Concordion. The main advantages of this library are:
the tests can be checked in alongside the code base (into a Subversion repository, for example)
they are executed by a standard JUnit runner
I wrote something very similar using JUnit. I had a large number of test cases (30 pages) in an XML file. Instead of trying to generate different tests, I did it all in a single test, which worked just fine.
My test looked something like this:
void setup() {
cases = read in the xml file
}
void test_fn_works() {
for case in cases {
assert(case.expected_result, fn(case.inputs),
'Case ' + case.inputs + ' should yield ' + case.expected_result);
}
}
With Ruby, I did exactly what you are saying-- generating tests on the fly. Doing this in Java, though, is complex, and I don't think it is worth it since there is another, quite reasonable approach.
Hope this helps.

Categories