first time poster and TDD adopter. :-) I'll be a bit verbose so please bear with me.
I've recently started developing SOAP based web services using the Apache CXF framework, Spring and Commons Chain for implementing business flow. The problem I'm facing here is with testing the web services -- testing as in Unit testing and functional testing.
My first attempt at Unit testing was a complete failure. To keep the unit tests flexible, I used a Spring XML file to keep my test data in. Also, instead of creating instances of "components" to be tested, I retrieved them from my Spring Application context. The XML files which harbored data quickly got out of hand; creating object graphs in XML turned out to be a nightmare. Since the "components" to be tested were picked from the Spring Application Context, each test run loaded all the components involved in my application, the DAO objects used etc. Also, as opposed to the concept of unit test cases being centralized or concentrated on testing only the component, my unit tests started hitting databases, communicating with mail servers etc. Bad, really bad.
I knew what I had done wrong and started to think of ways to rectify it. Following an advice from one of the posts on this board, I looked up Mockito, the Java mocking framework so that I could do away with using real DAO classes and mail servers and just mock the functionality.
With unit tests a bit under control, this brings me to my second problem; the dependence on data. The web services which I have been developing have very little logic but heavy reliance on data. As an example, consider one of my components:
public class PaymentScheduleRetrievalComponent implements Command {
public boolean execute(Context ctx) {
Policy policy = (Policy)ctx.get("POLICY");
List<PaymentSchedule> list = billingDAO.getPaymentStatementForPolicy(policy);
ctx.put("PAYMENT_SCHEDULE_LIST", list);
return false;
}
}
A majority of my components follow the same route -- pick a domain object from the context, hit the DAO [we are using iBatis as the SQL mapper here] and retrieve the result.
So, now the questions:
- How are DAO classes tested esp when a single insertion or updation might leave the database in a "unstable" state [in cases where let's say 3 insertions into different tables actually form a single transaction]?
- What is the de-facto standard for functional testing web services which move around a lot of data i.e. mindless insertions/retrievals from the data store?
Your personal experiences/comments would be greatly appreciated. Please let me know in case I've missed out some details on my part in explaining the problem at hand.
-sasuke
I would stay well away from the "Context as global hashmap" 'pattern' if I were you.
Looks like you are testing your persistence mapping...
You might want to take a look at: testing persistent objects without spring
I would recommend an in-memory database for running your unit tests against, such as HSQL. You can use this to create your schema on the fly (for example if you are using Hibernate, you can use your XML mappings files), then insert/update/delete as required before destroying the database at the end of your unit test. At no time will your test interfere with your actual database.
For you second problem (end-to-end testing of web services), I have successfully unit tested CXF-based services in the past. The trick is to publish your web service using a light-weight web server at the beginning of your test (Jetty is ideal), then use CXF to point a client to your web service endpoint, run your calls, then finally shut down the Jetty instance hosting your web service once your unit test has completed.
To achive this, you can use the JaxWsServerFactoryBean (server-side) and JaxWsProxyFactoryBean (client-side) classes provided with CXF, see this page for sample code:
http://cwiki.apache.org/CXF20DOC/a-simple-jax-ws-service.html#AsimpleJAX-WSservice-Publishingyourservice
I would also give a big thumbs up to SOAP UI for doing functional testing of your web service. JMeter is also extremely useful for stress testing web services, which is particularity important for those services doing database lookups.
First of all: Is there a reason you have to retrieve the subject under test (SUT) from the Spring Application context? For efficient unit testing you should be able to create the SUT without the context. It sounds like you have some hidden dependencies somewhere. That might be the root of some of your headache.
How are DAO classes tested esp when a
single insertion or updation might
leave the database in a "unstable"
state [in cases where let's say 3
insertions into different tables
actually form a single transaction]?
It seems you are worried about the database's constistency after you have running the tests. If possible use a own database for testing, where you don't need care about it. If you have such a sandbox database you can delete data as you wish. In this case I would do the following:
Flag all your fake data with some common identifier, like putting a special prefix to a field.
Before running the test drop a delete statement, which deletes the flagged data. If there is none, then nothing bad happens.
Run your single DAO test. After that repeat step 2. for the next test.
What is the de-facto standard for
functional testing web services which
move around a lot of data i.e.
mindless insertions/retrievals from
the data store?
I am not aware of any. From the question your are asking I can infer that you have on one side the web service and on the other side the database. Split up the responsibilities. Have separate test suites for each side. One side just testing database access (as described above). On the other side just testing web service requests and responses. In this case it pays of the stub/fake/mock the layer talking to the network. Or consider https://wsunit.dev.java.net/.
If the program is only shoving data in and out I think that there is not much behavior. If this is the case, then the hardest work is to unit test the database side and the web service side. The point is you can do unit testing without the need for "realistic" data. For functional testing you will need handrolled data, which is close to reality. This might be cumbersome, but if you already unit tested the database and web service parts intensively, this should reduce the need for "realistic" test cases considerably.
First of all, make thing clear.
In an ideal world the lifecycle of the software your are building is something like this:
- sy makes a report with the customer, so you got an user story with examples about how the application should work
- you generalize the user story, so you got rules, which you call as use cases
- you start to write a piece of functional (end to end) test, and it fails...
- after that your build the ui and mock out the services, so you got a green functional test and a specification about how your services should work...
- your job is to keep the functional test green, and implement the services step by step writing integration tests, and mocking out dependencies with the same approach until you reach the level of unit tests
- after that you do the next iteration with the use cases, write the next piece of functional test, and so on until the end of the project
- after that you make acceptance tests with the customer who accepts the product and pays a lot
So what did we learn from this:
There are many types of tests (don't confuse them with each other)
functional tests - for testing the use cases (mock out nothing)
integration tests - for testing application, component, module, class interactions (mock out the irrelevant components)
unit tests - for testing a single class in isolation from its environment (mock out everything)
user acceptance tests - customer makes sure, that she accepts the product (manual functional tests, or presentation made from automatic functional tests in working)
You don't need to test everything by functional tests and integration tests, because it is impossible. Test only the relevant part by functional and integration tests and test everything by unit tests! Familiarize yourself with the testing pyramid.
Use TDD, it makes life easier!
How are DAO classes tested esp when a single insertion or updation might leave the database in a "unstable" state [in cases where let's
say 3 insertions into different tables actually form a single
transaction]?
You don't have to test your database transactions. Assume, that they are working well, because database developers have already tested them, and I am sure you don't want to write concurrency tests... Db is an external component, so you don't have to test it yourself. You can write a data access layer to adapt the data storage to your system, and write integration tests only for those adapters. In case of database migration these tests will work by the adapters of the new database as well, because your write them to implement a specific interface... By any other tests (except functional tests) you can mock out your data access layer. Do the same with every other external component as well, write adapters and mock them out. Put these kind of integration tests to a different test suite than the other tests, because they are slow because of database access, filesystem access, etc...
What is the de-facto standard for functional testing web services which move around a lot of data i.e. mindless insertions/retrievals
from the data store?
Your can mock out your data store with an in memory db which implements the same storage adapters until you implemented everything else except the database. After that your implement the data access layer for the database and test it with your functional tests as well. It will be slow, but it has to run only once, for example by every new release... If you need functional tests by developing, you can mock it out with an in memory solution again... An alternative approach to run only the affected functional tests by developing, or modify the settings of the test db to make things faster, and so on... I am sure there are many test optimization solutions...
I must say I don't really understand
your exact problem. Is the problem
that your database is left in an
altered state after you've run the
test?
Yes, there are actually two issues here. First one being the problem with the database left in an inconsistent state after running the test cases. The second one being that I'm looking for an elegant solution in terms of end-to-end testing of web services.
For efficient unit testing you should
be able to create the SUT without the
context. It sounds like you have some
hidden dependencies somewhere. That
might be the root of some of your
headache.
That indeed was the root cause of my headaches which I am now about to do away with with the help of a mocking framework.
It seems you are worried about the
database's constistency after you have
running the tests. If possible use a
own database for testing, where you
don't need care about it. If you have
such a sandbox database you can delete
data as you wish.
This is indeed one of the solutions to the problem I mentioned in my previous post but this might not work in all the cases esp when integrating with a legacy system in which the database/data isn't in your control and in cases when some DAO methods require a certain data to be already present in a given set of tables. Should I look into database unit testing frameworks like DBUnit?
In this case it pays of the
stub/fake/mock the layer talking to
the network. Or consider
https://wsunit.dev.java.net/.
Ah, looks interesting. I've also heard of tools like SOAPUI and the likes which can be used for functional testing. Has anyone here had any success with such tools?
Thanks for all the answers and apologies for the ambiguous explanation; English isn't my first language.
-sasuke
Related
So I got a task where I have to verify the REST response of a remote micro service.
For now, I built an app that calling the given REST endpoint and now I have to verify the result.
My question is, where should I write the tests? If I write into the test scope with Junit/TestNG it should be unit tests with mocked objects, right?. But in this case, I couldn't test the remote service correctly.
Is it a bad pattern to call real service from test scope using Junit?
Thanks!
It really depends on the trade-offs that you'll need to make and could generally be pretty difficult to move towards a reliable state. For instance, if you wanted to test in QA but other teams have different philosophies on what QA is and happens to use it like a local development environment, their services are probably failing more often than it probably would have otherwise. In that case, when you want to run integration tests with real services that other teams own, chances are your tests may not be too indicative whether your services ran correctly. It could be the service dependency failing that trips failures when testing your services.
In that scenario, I would mock out downstream services that your team does not own because its out of your control. This could be mocking it by spinning up a web service that responds with JSON or XML payloads or whatever metalanguage the service you're consuming responding with. It could also be mocking it at the request client level meaning the lowest level of the library that sends the request.
As an effect, that means the requests that go to your service will go through all the layers of logic in your service without any mocks to the lowest possible part where your team has control over. It's based on black box testing where the service dependencies are isolated with the assumption that the other team is properly versioning their services or providing backwards compatibility. This type of testing is the less costly to maintain if you have ever seen the testing pyramid. The gist of the pyramid is that the more integrative your tests, the costlier it is to maintain and build.
If you want even more guarantees, you can combine this with contract driven tests. Essentially, another test suite that tests downstream services and making sure that they still adhere to the payload schema that your service depends on. This way, you can decouple these two sets of tests without having to worry about all the logic that comes with it. It is also more transparent in terms of failures as you will know whether your service failed or the service dependency failed.
That would be the strategy and I would advocate for using an acceptance testing framework like Cucumber to actually do the job. There are many variants of Cucumber but essentially, you would be writing behavioral driven tests (that even the product owner can read) that is parsed by a language of your choice. See this link for details.
Today i begin a project that i must to spread a backend in two pieces, i dont know what exactly does the backend do, only that i have to move specific services into new spring, jersey, maven multi module project.
So, the task to move was really easy and now its time to write tests. Previously, the project does not have any test.
When i start to write a JUnits from my Business Objects, i saw, that most services only perform a basic operation with a DAO, like getAll, get, save, update and delete. The other services got bussines validation, but it is not comlpex.
So the questions are more theorical:
Should i write JUnits to test a simple DAO get that will be mocked (no integration test) which will be the benefits for do this?
Which will be the correct way to make a integration test for a simple DAO get, getAll, or create (create that does not have any validation to execute before)
Unit&Integration tests are for developers, so if you don't feel like some developer will benefit from the test - don't write it. Also consider that tests verify behavior, not code, so if you don't see any behavior that needs testing - don't spend the time.
In your case I would at most write integration tests for services, and maybe for DAOs (if there is no ORM).
Anyway correct answer to your question depends on the level of quality that you need to provide for your project, team size, potential of destructive code changes, etc.
Two examples:
1) Small site's admin-panel in CRUD style, only one developer introduces changes simultaneously, and almost no business logic is present. Bugs presence is non-critical.
In this case I would not spend the time on any tests - most likely you need to focus on other things (e.g. client-side).
2) You are starting a complex project, that is currently in CRUD style, but interactions between distant services/DAOs are present, business logic tends to become complex at some time. Team is rapidly growing/changing, more than one person involved, new developers can't understand how the system works easily. Bugs presence are bad for business.
In this case I would at least start with integration tests for services.
I have a service, which works as a mediator between two other services. It basically validates the inputs, then passes them to those two service sequentially (by trying to keep transactional integrity), and then, if everything goes well, saves a result to the database.
My problem now is to test this service in isolation. Of course, I can provide stubs to satisfy the dependencies. I can also test the validation of inputs, whether appropriate data is saved in the DB in a normal case, as well as whether transactional integrity is kept if any of the dependencies throws an exception.
Yet, this is only half of what the service really does. My dilemma is if I should try to prove whether the other two dependency services actually processed the data appropriately as well? The scope of my service is quite broad, so I guess it is better to also know if the dependency services also did their job well. Yet, this gets out of the scope unit testing, and moves into integration testing, right?
I am kind of confused here.
If you're asking about unit-testing, then the way to do it is to test the class in isolation using mocks or stubs.
BUT, if you feel that just doing that is not enough, you can write some component tests, where you use the all the real classes you want to test, and use a stub (or inmemory) database and mock some of the dependencies that you consider not important for what you are trying to test.
In the past, I've tested small clusters of classes that had a high interaction between them in this way (and sometimes skipping unit-tests for those classes, as the component tests covered all the scenarios). Obviously, the problem with doing this is that the number of scenarios grows almost exponentially the more classes you're testing. Maybe you can test the bridge and the 2 real classes that use that bridge.
You should do both.
For unit testing, definitely use mock-objects for dependencies, preferrably using a tool like EasyMock. As a sidenote, if you feel that the functionality of your mediator service is too broad for unit testing, you may want to consider breaking it down into smaller pieces.
Of course, you additionally should do integration testing as well, using real dependencies, to make sure your services work together as intended.
I am trying to figure out the best way(s) to test Service and DAO layers. So, a few sub questions...
When testing a service layer, is it best to test against a mock DAO layer or a "live" DAO layer pointed at a testing environment?
How should SQL in the DAO layer be tested when the only test database is in a shared environment (Oracle/DB2)
How do you solve the paradox of any DAO writes/updates need to be tested with DAO reads which is something that also has to be tested?
I am looking for any good documentation, articles, or references in this area along with any tools to help automate the process. I already know about JUint for unit testing and Hudson for CI.
Get Growing Object-Oriented Software, Guided by Tests. It has some great tips about how to test database access.
Personally, I usually break the DAO tests in 2, a unit test with a mocked database to test functionality on the DAO, and an integration test, to test the queries against the DB. If your DAO only has database access code, you won't need a unit test.
One of the suggestions from the book that I took, is that the (integration) test has to commit the changes to the DB. I've learn to do this, after using hibernate and figuring out that the test was marked for rollback and the DB never got the insert statement. If you use triggers or any kind of validation (even FKs) I think this is a must.
Another thing, stay away from dbunit, it's a great framwork to start working, but it becomes hellish when a project becomes something more than tiny. My preference here, is to have a set of Test Data Builder classes to create the data, and insert it in the setup of the test or in the test itself.
And check dbmigrate, it's not for testing, but it will help you to manage scripts to upgrade and downgrade your DB schema.
In the scenario where the DB server is shared, I've creates one schema/user per environment. Since each developer has his own "local" environment, he also owns one schema.
Here are my answers :
Use mock DAOs to test your services. Much easier, mush faster. Use EasyMock or Mockito or any other mock framework to test the service layer.
Give each developer its own database schema to execute his tests. Such schemas are typically empty : the unit tests populate the database with a small test data set before running a test, and empties it once the test is completed. Use DBUnit for this.
If the reads work against a well-defined, static, test data set (which you should unit-test), then you can rely on them to unit-test the writes. But you can also use ad-hoc queries or even DBUnit to test that the writes work as expected. The fact that the tests are not necessarily run in this order doesn't matter. If everything passes, then everything is OK.
Background:
I have a Flex web app that communicates with a Java back-end via BlazeDS. The Flex client is composed of a flex-client module, which holds the views and presentation models and a separate flex-service module which holds the models (value objects) and service objects.
I am in the process of writing asynchronous integration tests for the flex-service module's RemoteObjects using FlexUnit4. In some of the tests, I modify the test data and query it back to see if everything works (a technique shown here: http://saturnboy.com/2010/02/async-testing-with-flexunit4)
Question:
How do I go about resetting the database to a known state before each FlexUnit4 test method (or test method chain)? In my Java server integration tests, I did this via a combination of DBUnit and Spring Test's transactions -- which rollback after each test method. But these Flexunit integration span multiple requests and thus multiple transactions.
Short of implementing an integration testing service API on the backend, how can this be accomplished. Surely others have run into this as well? Similar questions have been asked before ( Rollback database after integration (Selenium) tests ), but with no satisfactory answers.
There are several options:
If you use sequences for primary keys: After the database has been loaded with the test data, delete the sequence generator and replace it with one that starts with -1 and counts down. After the test, you can delete objects with a primary key < 0. Breaks for tests which modify existing data.
A similar approach is to create a special user or, if you have created timestamp columns, then the initial data must be before some date in the past. That needs additional indexes, though.
Use a database on the server which can be quickly wiped (H2, for example). Add a service API which you can call from the client to reset the DB.
Add undo to your web app. That's quite an effort but a very cool feature.
Use a database which allows to move back in time with a single command, like Lotus Notes.
Don't use a database at all. Instead write a proxy server which will respond to the correct input with the correct output. Add some code to your real server to write the data exchanged into a file and create your tests from that.
Or write test cases which run against the real server and which create these files. That will allow you to track which files change when you modify code on the server or client.
On the server, write tests which make sure that it will do the correct DB modifications.
Similar to "no database at all", hide all code which accesses the DB in a DB layer and use interfaces to access it. This allows you to write a mock-up layer which behaves like the real database but which saves the data in memory. Sounds simple but is usually a whole lot of work.
Depending on the size of your test database, you could automate clean backups/restores that gives you the exact environment you had on each test run.
I've that approach in on of my current projects (different platform) and we also test data schema change scripts with the same approach.
I'm dehydrated (my fav excuse for short-comings). So sorry if this answer is too close to the "integration testing service API on the backend" response that you didn't want.
The team that set-up flexUnit 'ages ago' made choices and created solutions based on our architecture, some of which would only apply to our infrastructure. Things to consider:
1) all of our backend methods return the same remotely-mapped class. 2) most all of our methods have an abstracted method that telling the method to (or not to) run a "begin transaction" at the beginning of the method and a "commit transaction" at the end (not sure of your db chunk).
The latter isn't probably the most object oriented solution, but here's what an asynchronous unit-test call does: Every unit test calls the same method-wrapper, and we pass-in the method-name/package-locale, plus the [...]args. A beginTransaction is done. The method is called, passing a false to the method for FE unit tests (to ignore the beginTransaction and commitTransaction lines), everything is run and the main 'response' class is generated and returned to the unit test method. A db-rollback is run and the response is returned to the unit test.
All of our unit tests are based on rolling-back transactions. I couldn't tell you of the issues that they had when setting up that jive, but that's my general understanding of how schtuff works.
Hope that helps. Understandable if it doesn't.
Best of luck,
--jeremy