I need to build a datadriven framework using selenium webdriver with java binding. i am bit confused with regards to choosing the right tool for building framework
Could somebody please recommend which framework (i.e. Junit, TestNG, Cucumber) works better with large set of test data
Also, please suggest which data file format i.e. .xls or .tsv to use for executing automated tests from Jenkins(deployed on Linux box)
As fare as I know and what I have earlier used Selenium for is to implement automated test scenarios of a web-application. Would typically be accept-test og end-to-end test (e2e).
These kind of test are what you call Black-Box testing. You give a input and expect a output, but does not know what is going on inside the black-box (you application).
To implement and execute your tests scenarios you would use a test-framework as you also mentioned (JUnit, TestNG, Cucumber etc.).
All test are able to be executed on a build server running Jenkins whether it is unit/integration or e2e-test (which could be implemented with Selenium).
Selenium test are expensive to maintain and often you would run into timing issues which causes test to fail even though the code does not contain any errors.
Therefore it is important to implement the test correct, otherwise you would run into a maintenance hell.
I am not sure if you are new to testing?? Maybe you could benefit from reading a bit about software testing.
The following is a good link: https://www.softwaretestingmaterial.com/software-testing/
Let me know, if I have misunderstood you question totally :)
I am a Java developer. We want to use cucumber testing in our project. We are working mainly on creating APIs. I am good with unit testing and researching about cucumber.
I am thinking about testing persistence methods - CRUD operations as an starter. My questions is that what could be the scenerios in this testing.
Also should I mock the database by creating tables in the feature file. Should I use mockito with Cucumber to mock call to some other services which connects to database and server.
What should be the cucumber testing in these scenerios and whats the best way to create framework to use cucumber in our Java API's project.
Also, how to populate models if not using database
IMO Gherkin (the language you write Cucumber features in), is good for writing business readable, simple scenarios.
To answer quickly, I would say that Cucumber is not a good fit for testing methods, if it is what you want to do.
As you can see with the file naming convention, you write *.feature files, and I think these files must only contains feature-related descriptions.
However, if you do have features to test, you have to choose how to test them
disconnected, can be run quicky by your CI
you will have to mock everything that cannot start-up in the build lifecycle
and they are solutions to start almost anything using Docker, like Testcontainers
connected to a environment
you do not have to mock anything
your tests may be slower
your tests may break because of the environement (failed deployement, server down, etc.)
I have a doubt. Say I have a web application which is big and relies on Java/Java EE (JSP/Servlets).
Every time before a drop we test each and every functionality on GUI so that everything is working properly. Previously it was easy but now as the number of modules has increased exponentially, manually testing each and every GUI with required functionality is no more a feasible option.
I am on lookout for tools in which I can write my entire test case say about 1000 and then just run it once before the drop and it will list down all the test cases that have failed.
The tool preferably must be free to download and use it.
I dont know whether using
Arquilian
or
JUnit
in this regard will help or not but automating testing before the drop is really needed..
Please guide.
Use Junit together with a mock framework i.e Mockito to test units (service methods)
Use Arquillian to test on an integration level ( how different services, modules work together )
Use a database testing tool (i.e dbunit) to test your database / persistence layer)
Use Selenium to test your frontend
Test as much as possible.
Use Jenkins and Sonar to track your build process and your quality of tests and code
You should always test your application on different level. There is not just one solution.
Use unit testing to test small pieces of your application and to make refactoring as easy as possible.
Use integration test to check your modules still work together as expected.
Use GUI testing to check if your customers can work with your software.
If its relevant, think about performance testing (i.e. jmeter )
Definitively Selenium. Couple it with maven cause you will probably need to package your project specifically for testing purpose. Moreover maven allow you to launch a container during the integration-test phase and to close it automatically at the end. You can also configure this as a nightly build on jenkins / hudson so you will be quicly notified of any regression.
I want to create a test suit for my java web application. Its a JSP applications with JDBC connectivity . My requirements are as follows,
1 - I should be able to test my database logic (Queries etc) through my models.
2 - Its great if i could test my .jsp pages as well (if possible)
After doing some research I found that DBUnit is good for database backend system testing, but unfortunately i couldnt find any good resource as a starter
What are you all think about testing options I have and it would be great if you could post some links to resources/ examples as well
EDIT:
and I have come across with mock objects (like JMock..), wonder I could use it as a replacement for DBUnit ?
thanks in advance
cheers
sameera
It's not clear from your question if you want to run Integration tests (Front end + back end) or Unit Tests against you Database layer.
If you need a tool that allows you to write Integration tests, you should definitively look at Selenium.
With Selenium you can generate functional tests by simply navigating your web site (JSP pages) and asserting that stuff on the page exists or it's equal to some values.
Selenium comes with a Firefox plugin that will basically generate the code for you. You can replay the test in the browser or export them as Java code and make them part of your test suite. Selenium is an invaluable tool.
The drawback of using a tool like Selenium is that your application need to be deployed somewhere before you can run your test suite. This may be a limitation if you plan to run automated tests generated using Selenium.
If you are only interested in testing your database access code (DAO, Data Access Layer) DBUnit is the perfect tool.
Generally, DBUnit is used to initialize the database tables before testing and, less often, to run assertions on the database content. DBUnit uses an XML based format to represent the data that will be inserted into the database.
The XML files containing the data to pre-populate the db are normally triggered by a build script (Ant, Maven, etc.) or directly in your unit test code.
I have used both approaches, it really depends on how your code is structured and how you access the database (Hibernate, Spring+Hibernate, JDBC...).
If your database is not too big, I'd recommend you populate it just before running your test suite. Alternatively, you can populate only the tables that you are interested in testing prior to every test.
Here is a link to Unitils, that is an additional library that can be used on top of DBUnit to simplify the database testing strategy. I think it can be used as a reference to get you started:
http://www.unitils.org/tutorial.html#Database_testing
Here is anoter link (quite old, 2004) showing the basic mechanics of DBUnit:
http://onjava.com/pub/a/onjava/2004/01/21/dbunit.html
DBUnit's official getting-started article here worked for me. Re: database logic testing, you might also want to check this out.
As for JSP testing, I have used Cactus to test my servlets before with success. However, I'm don't know about its JSP-testing facilities.
For your 1st question have a look at this StackOverFlow thread...
For 2nd, I would go with Chry's suggestion of Cactus or Selenium.
Hope that helps.
first time poster and TDD adopter. :-) I'll be a bit verbose so please bear with me.
I've recently started developing SOAP based web services using the Apache CXF framework, Spring and Commons Chain for implementing business flow. The problem I'm facing here is with testing the web services -- testing as in Unit testing and functional testing.
My first attempt at Unit testing was a complete failure. To keep the unit tests flexible, I used a Spring XML file to keep my test data in. Also, instead of creating instances of "components" to be tested, I retrieved them from my Spring Application context. The XML files which harbored data quickly got out of hand; creating object graphs in XML turned out to be a nightmare. Since the "components" to be tested were picked from the Spring Application Context, each test run loaded all the components involved in my application, the DAO objects used etc. Also, as opposed to the concept of unit test cases being centralized or concentrated on testing only the component, my unit tests started hitting databases, communicating with mail servers etc. Bad, really bad.
I knew what I had done wrong and started to think of ways to rectify it. Following an advice from one of the posts on this board, I looked up Mockito, the Java mocking framework so that I could do away with using real DAO classes and mail servers and just mock the functionality.
With unit tests a bit under control, this brings me to my second problem; the dependence on data. The web services which I have been developing have very little logic but heavy reliance on data. As an example, consider one of my components:
public class PaymentScheduleRetrievalComponent implements Command {
public boolean execute(Context ctx) {
Policy policy = (Policy)ctx.get("POLICY");
List<PaymentSchedule> list = billingDAO.getPaymentStatementForPolicy(policy);
ctx.put("PAYMENT_SCHEDULE_LIST", list);
return false;
}
}
A majority of my components follow the same route -- pick a domain object from the context, hit the DAO [we are using iBatis as the SQL mapper here] and retrieve the result.
So, now the questions:
- How are DAO classes tested esp when a single insertion or updation might leave the database in a "unstable" state [in cases where let's say 3 insertions into different tables actually form a single transaction]?
- What is the de-facto standard for functional testing web services which move around a lot of data i.e. mindless insertions/retrievals from the data store?
Your personal experiences/comments would be greatly appreciated. Please let me know in case I've missed out some details on my part in explaining the problem at hand.
-sasuke
I would stay well away from the "Context as global hashmap" 'pattern' if I were you.
Looks like you are testing your persistence mapping...
You might want to take a look at: testing persistent objects without spring
I would recommend an in-memory database for running your unit tests against, such as HSQL. You can use this to create your schema on the fly (for example if you are using Hibernate, you can use your XML mappings files), then insert/update/delete as required before destroying the database at the end of your unit test. At no time will your test interfere with your actual database.
For you second problem (end-to-end testing of web services), I have successfully unit tested CXF-based services in the past. The trick is to publish your web service using a light-weight web server at the beginning of your test (Jetty is ideal), then use CXF to point a client to your web service endpoint, run your calls, then finally shut down the Jetty instance hosting your web service once your unit test has completed.
To achive this, you can use the JaxWsServerFactoryBean (server-side) and JaxWsProxyFactoryBean (client-side) classes provided with CXF, see this page for sample code:
http://cwiki.apache.org/CXF20DOC/a-simple-jax-ws-service.html#AsimpleJAX-WSservice-Publishingyourservice
I would also give a big thumbs up to SOAP UI for doing functional testing of your web service. JMeter is also extremely useful for stress testing web services, which is particularity important for those services doing database lookups.
First of all: Is there a reason you have to retrieve the subject under test (SUT) from the Spring Application context? For efficient unit testing you should be able to create the SUT without the context. It sounds like you have some hidden dependencies somewhere. That might be the root of some of your headache.
How are DAO classes tested esp when a
single insertion or updation might
leave the database in a "unstable"
state [in cases where let's say 3
insertions into different tables
actually form a single transaction]?
It seems you are worried about the database's constistency after you have running the tests. If possible use a own database for testing, where you don't need care about it. If you have such a sandbox database you can delete data as you wish. In this case I would do the following:
Flag all your fake data with some common identifier, like putting a special prefix to a field.
Before running the test drop a delete statement, which deletes the flagged data. If there is none, then nothing bad happens.
Run your single DAO test. After that repeat step 2. for the next test.
What is the de-facto standard for
functional testing web services which
move around a lot of data i.e.
mindless insertions/retrievals from
the data store?
I am not aware of any. From the question your are asking I can infer that you have on one side the web service and on the other side the database. Split up the responsibilities. Have separate test suites for each side. One side just testing database access (as described above). On the other side just testing web service requests and responses. In this case it pays of the stub/fake/mock the layer talking to the network. Or consider https://wsunit.dev.java.net/.
If the program is only shoving data in and out I think that there is not much behavior. If this is the case, then the hardest work is to unit test the database side and the web service side. The point is you can do unit testing without the need for "realistic" data. For functional testing you will need handrolled data, which is close to reality. This might be cumbersome, but if you already unit tested the database and web service parts intensively, this should reduce the need for "realistic" test cases considerably.
First of all, make thing clear.
In an ideal world the lifecycle of the software your are building is something like this:
- sy makes a report with the customer, so you got an user story with examples about how the application should work
- you generalize the user story, so you got rules, which you call as use cases
- you start to write a piece of functional (end to end) test, and it fails...
- after that your build the ui and mock out the services, so you got a green functional test and a specification about how your services should work...
- your job is to keep the functional test green, and implement the services step by step writing integration tests, and mocking out dependencies with the same approach until you reach the level of unit tests
- after that you do the next iteration with the use cases, write the next piece of functional test, and so on until the end of the project
- after that you make acceptance tests with the customer who accepts the product and pays a lot
So what did we learn from this:
There are many types of tests (don't confuse them with each other)
functional tests - for testing the use cases (mock out nothing)
integration tests - for testing application, component, module, class interactions (mock out the irrelevant components)
unit tests - for testing a single class in isolation from its environment (mock out everything)
user acceptance tests - customer makes sure, that she accepts the product (manual functional tests, or presentation made from automatic functional tests in working)
You don't need to test everything by functional tests and integration tests, because it is impossible. Test only the relevant part by functional and integration tests and test everything by unit tests! Familiarize yourself with the testing pyramid.
Use TDD, it makes life easier!
How are DAO classes tested esp when a single insertion or updation might leave the database in a "unstable" state [in cases where let's
say 3 insertions into different tables actually form a single
transaction]?
You don't have to test your database transactions. Assume, that they are working well, because database developers have already tested them, and I am sure you don't want to write concurrency tests... Db is an external component, so you don't have to test it yourself. You can write a data access layer to adapt the data storage to your system, and write integration tests only for those adapters. In case of database migration these tests will work by the adapters of the new database as well, because your write them to implement a specific interface... By any other tests (except functional tests) you can mock out your data access layer. Do the same with every other external component as well, write adapters and mock them out. Put these kind of integration tests to a different test suite than the other tests, because they are slow because of database access, filesystem access, etc...
What is the de-facto standard for functional testing web services which move around a lot of data i.e. mindless insertions/retrievals
from the data store?
Your can mock out your data store with an in memory db which implements the same storage adapters until you implemented everything else except the database. After that your implement the data access layer for the database and test it with your functional tests as well. It will be slow, but it has to run only once, for example by every new release... If you need functional tests by developing, you can mock it out with an in memory solution again... An alternative approach to run only the affected functional tests by developing, or modify the settings of the test db to make things faster, and so on... I am sure there are many test optimization solutions...
I must say I don't really understand
your exact problem. Is the problem
that your database is left in an
altered state after you've run the
test?
Yes, there are actually two issues here. First one being the problem with the database left in an inconsistent state after running the test cases. The second one being that I'm looking for an elegant solution in terms of end-to-end testing of web services.
For efficient unit testing you should
be able to create the SUT without the
context. It sounds like you have some
hidden dependencies somewhere. That
might be the root of some of your
headache.
That indeed was the root cause of my headaches which I am now about to do away with with the help of a mocking framework.
It seems you are worried about the
database's constistency after you have
running the tests. If possible use a
own database for testing, where you
don't need care about it. If you have
such a sandbox database you can delete
data as you wish.
This is indeed one of the solutions to the problem I mentioned in my previous post but this might not work in all the cases esp when integrating with a legacy system in which the database/data isn't in your control and in cases when some DAO methods require a certain data to be already present in a given set of tables. Should I look into database unit testing frameworks like DBUnit?
In this case it pays of the
stub/fake/mock the layer talking to
the network. Or consider
https://wsunit.dev.java.net/.
Ah, looks interesting. I've also heard of tools like SOAPUI and the likes which can be used for functional testing. Has anyone here had any success with such tools?
Thanks for all the answers and apologies for the ambiguous explanation; English isn't my first language.
-sasuke