In-container testing is often opposed to testing with mock objects. However, as mock objects simply mimic the behavior of the real objects, isn't the in-container testing the only way to really test the system in its' real environment?
As an partial alternative to in-container testing and mock objects, Spring provides the TestContext framework that initializes Spring nicely without needing to start up the actual application container (web application server, in my case). However, this is somewhat limited approach as it only initializes Spring-specific features while the application server -specific features are not supported. So you cannot test everything. Also, as it is not 100% the same as the default WebApplicationContext that is used in real web execution, isn't this approach a bit hackyish? Is it bad?
For in-container testing, there are at least Cactus (outdated), Jeeunit (a very little project) and JBoss Arquillian (still alpha, but looks promising). I don't see any of these projects too widely used, so is there something bad with in-container testing? The main drawback often mentioned with in-container testing is slow execution speed. However, when run in a continuous integration environment and in a relatively small project, this shouldn't be a problem.
To summarize: should we do either in-container or out-container testing and why? Would you feel bad using mock objects or an alternative initialization mechanism (as in Spring TestContext) for your integration tests?
A subnote: I recently asked about categorization of integration test, which might be relevant.
However, as mock objects simply mimic the behavior of the real objects, isn't the in-container
testing the only way to really test the system in its' real environment?
I think the short answer is yes, but... I think your "categorization of integration test" question is very relevant. Unit and integration tests are both important although the serve different functions.
Unit tests are closely associated with the code, should startup and run very fast, should be run often by developers iterating on the code, and usually use mocks to a high degree. The idea is to test the code in question, not its dependencies or integration points. The problem with making the unit tests all be in-container is that they will be run less often or they will waste too much developer time.
We have isolated our in-container/integration tests elsewhere in another project with dependencies on the code projects. They are designed to closely mimic production configurations -- as much as possible. These tests take longer to setup, run longer, and are more useful to run by something like cruisecontrol. They are run by hand especially when we are closing in on a release or after development has stabilized. Often I fire up the integration tests when I'm going to lunch or a meeting. When an integration test discovers a bug, we try to write a unit test that also demonstrates the bug with the mocks -- this is often hard and can be impossible.
We usually have a couple small in-container tests with the unit tests just to make sure that the spring wiring works or to test some basic functionality but the rest of the integration testing is done in another project.
All this said, there is not a firm difference between the two. Sometimes we move unit tests that just deal with a lot of data and take a long time out to integration tests and sometimes integration tests run fast enough and are valuable enough to be included along with the code.
I'd say we should do both in-container and out-container testing. The main problem I've found with testing in-container is it's much more work to automate everything. You can get some cheaper wins with something like Spring's integration testing support, but we shouldn't fool ourselves that this covers everything that testing in deployment container would.
On the topic of mocks, I find that they can play a part in integration testing with Spring, but I'm more likely to use fake/stub services with canned results for integration and in-container testing (which I'd call system testing).
If you're not sure on what the difference is between mocking and stubbing, check out this Martin Fowler article.
Related
I am basically practicing with Java socket programming by building client and server (not necessarily HTTP server). In brief, the clients are sending request through sockets to server and server adds requests to task queue. The thread pool initially has certain number of threads and each free one is assigned to one runnable task in the task queue. My web server also has a simple storage that stores and retrieves data from a file from disk. In this project, I have to take care of several concurrency issues.
Basically, I have to build client, server, thread pool, handler, storage. However, I want to test thoroughly in a good systematic way (unit test, integration test, etc.). I don't have much experience in testing so I am looking for pointers, methodologies, frameworks, or tutorials. (I use Ant to automate building, and initially consider JUnit and EasyMock for testing)
Before testing, I'd start by coding some rough and ready prototpye code. Just to see it working and to get a feel for the APIs I will be using.
Then introduce some unit tests with JUnit (there are other frameworks but JUnit is ubiquitous, and you'll find plenty of tutorials to get you started).
If your object needs to interact with some other Objects to complete it's tasks, then use mocks (EasyMock or whatever) to provide the interaction - this will probably lead to a bit of re-factoring.
Once you are happy, you can start to look at testing how your Objects interact, you can write new (integration) tests that replace the Mocks with the real thing. Greater interaction results in greater complexity.
Some things to remember
trivial methods aren't worth testing (e.g. simple accessors)
100% coverage is a waste of time
any test is better than none
Unit test is easier to achieve than integration test
Not all tests are functional
Testing multi-threaded applications is hard
There is a book on how Google does testing. Basically they don't write tests until something looks viable. They have engineers who advise on how to structure code for testing. The point is:
Runnable code is the goal
Tests add to that goal, but do not replace it
Writing code that can be tested is a learnt skill
The basic question is "How should one start with writing unit and integration testing for a untested project? Especially considering the fact that the person is not familiar with the code and has not done integration testing before."
Consider the scenario where unit tests and integration tests have to be written for a project. The project uses Java/J2EE technology does not have any tests written at all.
The dilemma that I face is since I have not written the code, I don't want to refactor the code immediately to write tests. I also have to select a testing framework. I am thinking of using Mockito and Powermock.
I also have to estimate code coverage for the tests. And then perform integration testing. I will have to research on integration testing tools and select one. I have not done any integration testing or estimated acceptable level of code coverage for a project before.
Since I am working independently, if there are some strategies, tips, suggestions on what should I start with and tools that one can recommend, I will appreciate it.
First comes first:
Understand the architecture, what are the main components?
If you have no good overview of the features and functions the program offers, make a list of them and create a hierarchy of them
Get familiar with the code, I recommend the following approach:
after you understand where the code of its different components are started, try to figure out the method invocation hierarchy (in Eclipse you can easily jump the source code definitions by pressing F3)
later you can do the same, while debugging the code, this way it will jump automatically to the definitions, plus you can observe how the state of the program changes
For Unit Testing itself, I can recommend Clean Code Chapter 9 (circa 12 pages) for starters. It uses JUnit for the example and gives a very good introduction how good testing is done.
There you will learn things like the F.I.R.S.T. principle, that Unit Tests should be:
Fast, Independent, Repeatable, Self-Validating and Timly
Some clarifications, JUnit is the most used and accepted test framework itself. Mockito and Powermock are mocking frameworks, they are used together with JUnit when you want to do integration tests.
For code coverage I can only recommend Cobertura, but there are many more.
Start with unit tests before you dive into integration tests (bottom-up), you can also do it the other way around (top-bottom), but since you say you are not so much experienced I would stay stick to the first.
Finally, just go for it and get started. You will learn the most and fastest while actually writing the test code.
Stop. "..not familiar with the code..". First get familiar with the code and most importantly its expected functionality. You can't refactor or unit test a code that you are not comfortable with.
Since you have not done unit-tests before, I would suggest learning and getting convenient with unit-tests.
Important: Bad/Wrong unit-tests are worse than no unit-tests. This is because the next guy who will maintain your code will misinterpret
the functionality.
There are bunch of Code Coverage tools out there. You can use which ever seduces you better.
Adding tests to legacy code that has no tests is a difficult task. As #Suraj has mentioned, get familiar with the code base and the expected functionality. You can't test it if you don't know what it is supposed to do.
In terms of choosing which areas of the code to test. Start with the high business value areas. Which functionality is most important? You want to make sure you have a strong test set for that code.
Since you don't have any unit/integration tests, I would start with some high level end to end tests that at least ensure that given some inputs to the system you get some expected outputs. This doesn't ensure correctness but at least ensures consistency.
Then as you develop a test suite you can be confident that the refactorings you are doing are not changing the behavior of the code (unless you find bugs of course that are being fixed).
For testing frameworks, JUnit is the standard unit testing framework. Note that the frameworks Mockito and Powermock are not testing frameworks themselves, but they can be used within JUnit.
For acceptance tests, there are also a variety of frameworks to help. For web UI testing, Selenium is pretty standard. There are also tools like Fitnesse for more table driven testing.
There are also some common frameworks to help with code coverage - Cobertura, Emma, Clover come to mind.
I would also set up an automated build (Jenkins build server is pretty simple to set up). This will allow you to run your tests on every checkin. Even though your code coverage is going to be low to start, getting in this habit is a good one.
I have heard that Java EE unit testing is much harder than standard Java applications. The testing for my company's application stays at User Acceptance testing. We verify UI and functions behave as they are supposed to. Does it make sense to do unit testing on Java EE applications? If yes, what are some good starting points?
Of course it makes sense.
Let me guess: in the case of your company, when you make a minor change to a central component, I suppose that you'll repeat once and again your User Acceptance Tests. A real PITA, and overboring for a development team, in my opinion.
Unit testing allows you to develop in a consistent way the different layers of your Java EE app (i.e. view, business and model), AND, moreover, it allows you to do regression testing. The magic with regression testing is that in a long life application, this provides you a way to be sure that a new patch, code refactoring, or evolution doesn't break the current system behaviour. And it can be done automatically if you use a continuous integration tool.
So, you can write unit test for business and model layer in a direct way using, as an example, jUnit. And here comes the problem with Unit testing in Java EE apps: What do we do with the view layer?
Well, for the view layer you can use an automated test tool, say Apache jMeter, to check the expected behaviour of your Webapps, check your data validation, your system stability (programing long tests with a regular number of users) and scalability (with increasing and decreasing concurrent users), stress testing, etc. (I know it's not Unit testing, but it can be used in a similar way for regression testing).
I think it's a capital part of every software project, and I invite you to apply it to some of your project, and compare by yourself the final result.
Java EE 6's CDI resource injection was practically made for unit testing; you can swap out #Inject-ed FacesContexts, database resources, and beans for #Alternative beans that have known, predictable behaviour for testing.
It always makes sense to do unit testing on the server. Java EE is not necessarily harder to unit test. A good starting point is the code and the various layers of the application. i.e. start writing unit tests for persistence, then write unit tests for the services, etc....
Testing at the UI level alone is dangerous, because you can have bugs in the lower layer that are either hidden by, or compensated for in, the upper layers MOST of the time. It also means when you experience a problem, you have to first find out whether the problem is in the front end or the back end.
Yeah, Java EE is harder to unit-test. It's also much harder to fix it when you break something and don't notice it for two weeks because you never bothered to write unit tests.
The more complex the application is, the more you need to have a good testing regimen. And since we're talking about Java EE, it's safe to assume that your application is complex.
I'm working on an existing Java EE project with various maven modules that are developed in Eclipse, bundled together and deployed on JBoss using Java 1.6. I have the opportunity to prepare any framework and document how unit testing should be brought to the project.
Can you offer any advice on...
JUnit is where I expect to start, is this still the defacto choice for the Java dev?
Any mocking frameworks worth setting as standard? JMock?
Any rules that should be set - code coverage, or making sure it's unit rather than integration tests.
Any tools to generate fancy looking outputs for Project Managers to fawn over?
Anything else? Thanks in advance.
Any tools to generate fancy looking outputs for Project Managers to fawn over?
Be careful. A fancy tool for displaying metrics on unit test counts, coverage, code quality metrics, line counts, check-in counts and so on can be dangerous in the hands of some project managers. A project manager (who is not in touch with the realities of software development) can get obsessed with the metrics, and fail to realize that:
they don't give the real picture of the project's health and progress, and
they can give a completely false picture of the performance of individual team members.
You can get silly situations where a manager gives the developers the message that they should (for example) try to achieve maximal unit test coverage for code where this is simply not warranted. Time is spent on pointless work, the important work doesn't get done, and deadlines are missed.
Any rules that should be set - code coverage, or making sure it's unit rather than integration tests.
Code coverage is more important for parts of the code that are likely to be fragile / buggy. Acceptable coverage levels should reflect this.
Unit tests versus integration tests depends on the nature and complexity of the system you are building.
Adding lots of unit level tests after the fact is probably a waste of time. It should only be done for class identified as being problematic / needing maintenance work.
Adding integration level tests after the fact is useful, especially if the projects original developers are no longer around. A decent integration test suite helps to increase your confidence that some change does not break important system functionality. But this needs to be done judiciously. A test suite that tests the N-th degree of a website's look and feel can be a nightmare to maintain ... and impediment to progress.
Concerning the unit testing framework, there are mainly two of them : jUnit and TestNG. Both have theuir advantages, and both are equally performant. The main dvantage of jUnit is (to my mind) its default incoproration of an Eclipse plugin allowing easy tests calling.
Concerning the mocking framework, I don't find them to be a required part of your testing approach. Of course they're useful, but they solve a specific purpose : testing a behaviour (as opposite to testing an interface - what jUnit allows. With mocking frameworks, you're able to test how a specific class implements a specific interface. Will you need it ? Obviously. Will you need it first ? I don't know.
Concerning the rules, the only one I've found to be useful is simple (as always) : "always test code that broke at least once.". Consider your bug tracker. Each time a bug is encountered, there must be a unit test ensuring there is no regression. It's, to my mind, the faster way to have quality code.
Concerning the fancy- and efficient - output, I can recommend you enough to install a continous integration server (Hudson, obviously). It will run all your test suite each time code is commited, to ensure there are no side effects. it will generate graphs shoiwing the number of test run, and so on. it also can integrate code coverage tools and graphs. This continuous integration server will really become fast your testing buddy.
This is a complex question, so just a few notes about our practice at $work:
JUnit is indeed still the standard. Most documentation and literature treats JUnit.
Mockito seems to be the new star in Java mocking, although we still use JMock and think it's fine for our needs.
We use the EclEmma Eclipse plugin for checking our test coverage, and like it.
If you haven't done so already, read Working Effectively with Legacy Code by Michael Feathers.
I've been retrofitting unit tests to a C++ project and it is not pleasant.
First thing I did was to identify where most of the 'action' occurs. Then use that to start putting unit tests on the functions that can be test easily.
Then once you have the easier ones you can start looking at expanding the coverage virally - attack the functions that have fewer dependancies, run through them a few times in a debugger seeing what values are passed in and then write unit tests with those values to make sure you don't break anything.
Don't expect a quick fix - it's taken 3 weeks (6hr days, 5 days a week) to get 20% coverage but the code spends 80% of the time in that code so I think it has been time well spent and has uncovered quite a few bugs.
Regarding test coverage, I think that when you're bringing in unit testing to an existing project it's too early to start setting coverage expectations. You should start by ensuring that you actually can integrate the test framework and get reports from the coverage tools. Once you've done that you can start monitoring coverage, and then you can consider targets.
it's good practice to write Unittests to
be independed of the whole spring application context
automate the test you are doing for continuous integration
avoid dependency on the Servlet Container
I guess with JRebel there's a temptation to test everything in the running application and 'forget' to write Unittests. What's your experience?
I like to think that you're not writing unit tests because of the slow turnaround cycle but for other reasons. The moment the turnaround cycle goes to zero there is still benefits of unit tests.
I think that would be a big mistake. Sure it is faster, but you have no assurance that you aren't breaking things.
So it is faster to do manual unit testing, but the disadvantages aren't just about speed. You can't retest everything on every change without automated tests.
JRebel does make in-container unit tests more viable, though.
I tend to think that writing and running unit tests is a good way to compile the appropriate classes so that JRebel can reload them.