I have a class used for handling a connection to an external system.
The class has some a few public methods, let's say:
close()
configure()
send()
connect()
And a handful of private methods.
The class is intended to hide most of the re-establishing, fail-checking and connection-handling in the internal works.
Now, I get a code coverage error on this, since there is no unit tests for this class, besides the configure-method.
Is there another way of writing unit tests for such classes, except for heavy mocking?
If so, isn't that a good proof the class should be tested at an intergration-test or system-test level rather than unit-test? Do communication-classes belong to unit-tests or system tests?
As long as you cannot instantiate the external system locally in an unit test, this task is a classical integration or system test. So the communication classes, typically do not belong to unit tests.
The coverage will be low for such classes, showing that there is
need to test them otherwise.
Further the coverage report, e.g in SonarCube can (hopefully) be filtered.
One could define exception filters, together with the responsible person (e.g Software Architect.)
It may help to move all such communication classes to its own package.
Or even to its own project or jar file. For that project the coverage would then not be executed.
Sometimes it makes sense to build a dummy external system, to work with.
If the external system is yet under development and when it changes its interfaces daily, you can lose much time in using the external system.
In such situations a dummy system can be used, which is maybe instantiable from the unit test, too.
That depends on the amount of code in the class. If it just configures another service (like a socket or a database driver), unit testing doesn't make sense since someone probably already tested the actual service.
If the code is very complex (error handling, data transformation), you should write unit tests for this part of the code and mock the service out.
Related
We are using JUnit to execute integration tests and also the system integration tests which rely on external test systems (not necessarily maintained by our own company).
I wonder where to put the code that checks if the system is available prior to running the test cases? So I can determine if there is a network or other issue and not one with the test itself.
JUnit allows to setup some parts of the test in JUnit-Rules. Is it a good idea to setup the service that communicates with the external system within the rule and do some basic checks ("ping") to the external system within the rule? Or to store the state and within the Test use a JUnit assume(rule.isAvailable) to avoid having the test executed?
Or would it be smarter to put this verification code in a custom JUnit Runner?
Or is there even another way to do this? (simply create some utils?)
The goal is to skip the tests if some conditions are not met since it is obvious the tests will fail. I know this indicates a bad exception handling but there is a lot of legacy code I can't change altogether.
I tried to find some articles myself but it seems the search terms ("test", "external system" and so on) are a little thankless.
thanks!
Consider using org.junit.Assume.* as described here. When assumptions fail, your tests are ignored by default. You can write a custom runner to do something else when your assumptions fail.
The key thing though is that the tests don't fail when assumptions like the availability of your external services fail.
If the ping applies to every single test in the class, I would put the ping call in the #Before method. #Before will get executed before every single test method (i.e., #JUnit -annotated methods).
If the ping does not apply to all tests in the class, then you would have to hit ping explicitly from those methods.
I have a service, which works as a mediator between two other services. It basically validates the inputs, then passes them to those two service sequentially (by trying to keep transactional integrity), and then, if everything goes well, saves a result to the database.
My problem now is to test this service in isolation. Of course, I can provide stubs to satisfy the dependencies. I can also test the validation of inputs, whether appropriate data is saved in the DB in a normal case, as well as whether transactional integrity is kept if any of the dependencies throws an exception.
Yet, this is only half of what the service really does. My dilemma is if I should try to prove whether the other two dependency services actually processed the data appropriately as well? The scope of my service is quite broad, so I guess it is better to also know if the dependency services also did their job well. Yet, this gets out of the scope unit testing, and moves into integration testing, right?
I am kind of confused here.
If you're asking about unit-testing, then the way to do it is to test the class in isolation using mocks or stubs.
BUT, if you feel that just doing that is not enough, you can write some component tests, where you use the all the real classes you want to test, and use a stub (or inmemory) database and mock some of the dependencies that you consider not important for what you are trying to test.
In the past, I've tested small clusters of classes that had a high interaction between them in this way (and sometimes skipping unit-tests for those classes, as the component tests covered all the scenarios). Obviously, the problem with doing this is that the number of scenarios grows almost exponentially the more classes you're testing. Maybe you can test the bridge and the 2 real classes that use that bridge.
You should do both.
For unit testing, definitely use mock-objects for dependencies, preferrably using a tool like EasyMock. As a sidenote, if you feel that the functionality of your mediator service is too broad for unit testing, you may want to consider breaking it down into smaller pieces.
Of course, you additionally should do integration testing as well, using real dependencies, to make sure your services work together as intended.
Can I create a junit environment without file system and network access? I want to enforce stricter rules for our test cases.
Ideally, this would be configurable with maven and work for the default test phase.
Based on this answer https://stackoverflow.com/a/309427/116509, I think you could set a SecurityManager on test setup and restore the original on teardown.
However, IMHO, some unit tests should be allowed to touch the file system, if for example the class under test actually creates files as part of its contract. (You can use dependency injection to make sure the files are created in a temp directory). Likewise, a good unit test of a class that uses HTTP should test it against an HTTP endpoint. Otherwise you just end up mocking everything and your test becomes worthless. I suppose the default should be to deny access, and then a developer would need to specifically override the permissions for this kind of test.
The typical way to handle these dependancies on a file system/network access, is to mock them out in a test context. This way, the real code can go through the normal channels, but your tests don't have to rely on a file system or a network.
Look into mocking frameworks to help you do a lot of this. Enabling this kind of testing will also make your code cleaner, too. :)
You can use Ashcroft to prohibit access to file system and other resources from your tests. It uses Java Security manager to restrict access to certain resources.
Another approach would be to use AspectJ and implement several advices prohibiting calling certain APIs or packages.
I'm not sure what you mean by JUnit environment, but you should not need a file system, or network access to run unit tests. On the other hand, if you are testing code that uses network and filesystem APIs, you may have an issue. In that case, you may need to abstract your code into smaller testable chunks. You should not be testing weather the network and filesystem APIs are working in a unit test, these are integration tests. You should only be testing your code in unit tests.
I have a class that is responsible for network communication. I would like to have a unit test for it.
Here is where I'm stuck in order to write a test for it i have to implement a server to communicate with but that in turn will require its own test.
How do you write tests for a class such as this?
Unless you are writing low level drivers, your class undoubtedly depends on other classes to do the actual communications. In that event, I would use dependency injection to supply those classes or wrappers around them, if they aren't easily mocked. In you tests, you'd supply a mock version of the classes you are depending on (or the wrappers if you can't mock the actual classes). Verify that the correct methods with the proper parameters are invoked by your methods on the mocks that you supply. Make sure you have enough unit tests to satisfy yourself that you've covered the full range of behavior from the real dependencies. This will suffice for your unit tests.
You will also need some integrations tests. Unfortunately, the easiest way to do this is probably to develop a full-up mock server to communicate with. Note that your mock server need only implement the interface -- not the actual code on the other end. Supply some extra methods to allow you to set the server up for your integration tests so that the expected behavior with the mock server occurs.
If you're unit-testing a class that's responsible for network communication, then you only test that you're doing the communication correctly. So supply a mock (as in, a dummy) server for it to communicate with. There are many mock libraries that you can use such as EasyMock and jMockit. If you want to test an actual communication with an actual server, then that's integration-testing. Of course, the exact definition of what unit-testing encompasses varies from developer to developer.
Right my junit tests look like a long story:
I create 4 users
I delete 1 user
I try to login with the deleted user and make sure it fails
I login with one of the 3 remaining user and verify I can login
I send a message from one user to the other and verify that it appears in the outbox of the sender and in the inbox of the receiver.
I delete the message
...
...
Advantages:
The tests are quite effective (are very good at detecting bugs) and are very stable, becuase they only use the API, if I refactor the code then the tests are refactored too. As I don't use "dirty tricks" such as saving and reloading the db in a given state, my tests are oblivious to schema changes and implementation changes.
Disadvantages:
The tests are getting difficult to maintain, any change in a test affects other tests. The tests run 8-9 min which is great for continuous integration but is a bit frustrating for developers. Tests cannot be run isolated, the best you can do is to stop after the test you are interested in has run - but you absolutely must run all the tests that come before.
How would you go about improving my tests?
First, understand the tests you have are integration tests (probably access external systems and hit a wide range of classes). Unit tests should be a lot more specific, which is a challenge on an already built system. The main issue achieving that is usually the way the code is structured:
i.e. class tightly coupled to external systems (or to other classes that are). To be able to do so you need to build the classes in such a way that you can actually avoid hitting external systems during the unit tests.
Update 1: Read the following, and consider that the resulting design will allow you to actually test the encryption logic without hitting files/databases - http://www.lostechies.com/blogs/gabrielschenker/archive/2009/01/30/the-dependency-inversion-principle.aspx (not in java, but ilustrates the issue very well) ... also note that you can do a really focused integration tests for the readers/writers, instead of having to test it all together.
I suggest:
Gradually include real unit tests on your system. You can do this when doing changes and developing new features, refactoring appropriately.
When doing the previous, include focused integration tests where appropriate. Make sure you are able to run the unit tests separated from the integration tests.
Consider your tests are close to testing the system as a whole, thus are different from automated acceptance tests only in that they operate on the border of the API. Given this think about factors related to the importance of the API for the product (like if it will be used externally), and whether you have good coverage with automated acceptance tests. This can help you understand what is the value of having these on your system, and also why they naturally take so long. Take a decision on whether you will be testing the system as a whole on the interface level, or both the interface+api level.
Update 2: Based on other answers, I want to clear something regarding doing TDD. Lets say you have to check whether some given logic sends an email, logs the info on a file, saves data on the database, and calls a web service (not all at once I know, but you start adding tests for each of those). On each test you don't want to hit the external systems, what you really want to test is if the logic will make the calls to those systems that you are expecting it to do. So when you write a test that checks that an email is sent when you create an user, what you test is if the logic calls the dependency that does that. Notice that you can write these tests and the related logic, without actually having to implement the code that sends the email (and then having to access the external system to know what was sent ...). This will help you focus on the task at hand and help you get a decoupled system. It will also make it simple to test what is being sent to those systems.
unit tests should - ideally - be independent, and able to run in any order. So, I would suggest that you:
break up your tests to be independent
consider using an in-memory database as the backend for your tests
consider wrapping each test or suite in a transaction that is rolled back at the end
profile the unit tests to see where the time is going, and concentrate on that
if it takes 8 minutes to create a few users and send a few messages, the performance problem may not be in the tests, rather this may be a symptom of performance problems with the system itself - only your profiler knows for sure!
[caveat: i do NOT consider these kinds of tests to be 'integration tests', though i may be in the minority; i consider these kinds of tests to be unit tests of features, a la TDD]
Now you are testing many things in one method (a violation of One Assertion Per Test). This is a bad thing, because when any of those things changes, the whole test fails. This leads it to not being immediately obvious why a test failed and what needs to be fixed. Also when you intentionally change the behaviour of the system, you need to change more tests to correspond the changed behaviour (i.e. the tests are fragile).
To know what kind of tests are good, it helps to read more on BDD: http://dannorth.net/introducing-bdd http://techblog.daveastels.com/2005/07/05/a-new-look-at-test-driven-development/ http://jonkruger.com/blog/2008/07/25/why-behavior-driven-development-is-good/
To improve the test that you mentioned, I would split it into the following three test classes with these context and test method names:
Creating user accounts
Before a user is created
the user does not exist
When a user is created
the user exists
When a user is deleted
the user does not exist anymore
Logging in
When a user exists
the user can login with the right password
the user can not login with a wrong password
When a user does not exist
the user can not login
Sending messages
When a user sends a message
the message appears in the sender's outbox
the message appears in the reciever's inbox
the message does not appear in any other message boxes
When a message is deleted
the message does not anymore exist
You also need to improve the speed of the tests. You should have a unit test suite with good coverage, which can run in a couple of seconds. If it takes longer than 10-20 seconds to run the tests, then you will hesitate to run them after every change, and you lose some of quick feedback that running the tests gives you. (If it talks to the database, it's not a unit test, but a system or integration test, which have their uses, but are not fast enough to be executed continually.) You need to break the dependencies of the classes under test by mocking or stubbing them. Also from your description it appears that your tests are not isolated, but instead the tests depend on the side-effects caused by previous tests - this is a no-no. Good tests are FIRST.
Reduce dependencies between tests. This can be done by using Mocks. Martin Fowler speaks about it in Mocks aren't stubs, especially why mocking reduces dependencies between tests.
You can use JExample, an extension of JUnit that allows test methods to have return values that are reused by other tests. JExample tests run with the normal JUnit plugin in Eclipse, and also work side by side with normal JUnit tests. Thus migration should be no problem. JExample is used as follows
#RunWith(JExample.class)
public class MyTest {
#Test
public Object a() {
return new Object();
}
#Test
#Given("#a")
public Object b(Object object) {
// do something with object
return object;
}
#Test
#Given("#b")
public void c(Object object) {
// do some more things with object
}
}
Disclaimer, I am among the JExample developers.
If you use TestNG you can annotate tests in a variety of ways. For example, you can annotate your tests above as long-running. Then you can configure your automated-build/continuous integration server to run these, but the standard "interactive" developer build would not (unless they explicitly choose to).
This approach depends on developers checking into your continuous build on a regular basis, so that the tests do get run!
Some tests will inevitably take a long time to run. The comments in this thread re. performance are all valid. However if your tests do take a long time, the pragmatic solution is to run them but not let their time-consuming nature impact the developers to the point that they avoid running them.
Note: you can do something similar with JUnit by (say) naming tests in different fashions and getting your continuous build to run a particular subset of test classes.
By testing stories like you describe, you have very brittle tests. If only one tiny bit of functionality is changing, your whole test might be messed up. Then you will likely to change all tests, which are affected by that change.
In fact the tests you are describing are more like functional tests or component tests than unit tests. So you are using a unit testing framework (junit) for non-unit tests. In my point of view there is nothing wrong to use a unit testing framework to do non-unit tests, if (and only if) you are aware of it.
So there are following options:
Choose another testing framework which supports a "story telling"-style of testing much better, like other user already have suggested. You have to evaluate and find a suitable testing framework.
Make your tests more “unit test”-like. Therefore you will need to break up your tests and maybe change your current production code. Why? Because unit testing aims on testing small units of code (unit testing purists suggest only one class at once). By doing this your unit tests become more independent. If you change the behavior of one class, you just need to change a relatively small amount of unit test code. This makes your unit test more robust. During that process you might see that your current code does not support unit testing very well -- mostly because of dependencies between classes. This is the reason that you will also need to modify your production code.
If you are in a project and running out of time, both options might not help you any further. Then you will have to live with those tests, but you can try to ease your pain:
Remove code duplication in your tests: Like in production code eliminate code duplication and put the code into helper methods or helper classes. If something changes, you might only need to change the helper method or class. This way you will converge to the next suggestion.
Add another layer of indirection to your tests: Produce helper methods and helper classes which operate on a higher level of abstraction. They should act as API for your tests. These helpers are calling you production code. Your story tests should only call those helpers. If something changes, you need to change only one place in your API and don't need to touch all your tests.
Example signatures for your API:
createUserAndDelete(string[] usersForCreation, string[] userForDeletion);
logonWithUser(string user);
sendAndCheckMessageBoxes(string fromUser, string toUser);
For general unit testing I suggest to have a look into XUnit Test Patterns from Gerard Meszaros.
For breaking dependencies in your production tests have a look into Working Effectively with Legacy Code from Michael Feathers
In addition to the above, pick up a good book on TDD (I can recommend "TDD and Acceptance TDD for Java Developers"). Even though it will approach from a TDD point of view there is alot of helpful information about writing the right kind of unit tests.
Find someone who has alot of knowledge in the area and use them to figure out how you can improve your tests.
Join a mailing list to ask questions and just read the traffic coming through. The JUnit list at yahoo (something like groups.yahoo.com/junit). Some of the movers and shakers in the JUnit world are on that list and actively participate.
Get a list of the golden rules of unit tests and stick them on your (and others) cubicle wall, something like:
Thou shalt never access an external system
Thou shalt only test the code under test
Thou shalt only test one thing at once
etc.
Since everyone else is talking about structure I'll pick different points. This sounds like a good opportunity to profile the code to find bottleknecks and to run it through code coverage to see if you are missing anything (given the time it takes to run it the results could be interesting).
I personally use the Netbeans profiler, but there are ones in other IDEs and stand alone ones as well.
For code coverage I use Cobertura, but EMMA works too (EMMA had an annoyance that Cobertura didn't have... I forget what it was and it may not be an issue anymore). Those two are free, there are paid ones as well that are nice.