How do you automate integration testing? I use JUnit for some of these tests. This is one of the solutions or is totally wrong? What do you suggest?
I've used JUnit for doing a lot of integration testing. Integration testing can, of course, mean many different things. For more system level integration tests, I prefer to let scripts drive my testing process from outside.
Here's an approach that works well for me for applications that use http and databases and I want to verify the whole stack:
Use Hypersonic or H2 in in-memory mode as a replacement for the database (this works best for ORMs)
Initialize the database in #BeforeSuite or equivalent (again: easiest with ORMs)
Use Jetty to start an in-process web server.
#Before each test, clear the database and initialize with the necessary data
Use JWebUnit to execute HTTP requests towards Jetty
This gives you integration tests that can run without any setup of database or application server and that exercises the stack from http down. Since it has no dependencies on external resources, this test runs fine on the build server.
Here some of the code I use:
#BeforeClass
public static void startServer() throws Exception {
System.setProperty("hibernate.hbm2ddl.auto", "create");
System.setProperty("hibernate.dialect", "...");
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setJdbcUrl("jdbc:hsqldb:mem:mytest");
new org.mortbay.jetty.plus.naming.Resource(
"jdbc/primaryDs", dataSource);
Server server = new Server(0);
WebAppContext webAppContext = new WebAppContext("src/main/webapp", "/");
server.addHandler(webAppContext);
server.start();
webServerPort = server.getConnectors()[0].getLocalPort();
}
// From JWebUnit
private WebTestCase tester = new WebTestCase();
#Before
public void createTestContext() {
tester.getTestContext().setBaseUrl("http://localhost:" + webServerPort + "/");
dao.deleteAll(dao.find(Product.class));
dao.flushChanges();
}
#Test
public void createNewProduct() throws Exception {
String productName = uniqueName("product");
int price = 54222;
tester.beginAt("/products/new.html");
tester.setTextField("productName", productName);
tester.setTextField("price", Integer.toString(price));
tester.submit("Create");
Collection<Product> products = dao.find(Product.class);
assertEquals(1, products.size());
Product product = products.iterator().next();
assertEquals(productName, product.getProductName());
assertEquals(price, product.getPrice());
}
For those who'd like to know more, I've written an article about Embedded Integration Tests with Jetty and JWebUnit on Java.net.
JUnit works. There are no limitations that restrict it to being unit tests only. We use JUnit, Maven and CruiseControl to do CI.
There may be tools that are specific for integration testing, but I would think their usefulness is dependent on what type of system components you are integrating. JUnit will work fine for non UI type testing.
When using Maven to build a project, I've had a little more luck with TestNG because it has #BeforeSuite and #AfterSuite operations. Which are useful because Maven will not execute the 'post-integration-test` if any of the integration tests fail. Not a problem with Ant, so I just use jUnit out of preference with it.
In either case, segmenting out the tests as both TestNG and jUnit do is helpful with integration tests too.
In our work here, our integration testing solution has three major parts:
CruiseControl is the foundation of our continuous integration methodology.
Our CruiseControl configuration kicks off a quick-test build within 3 minutes of anyone's checkin to Subversion. The tests that happen here are "does everything still compile?" and "do the unit tests all still pass?". JUnit is obviously the major facilitator in answering the second questions.
Every hour, it kicks off a larger build that constructs the online help and installers that we use on our various deployment platforms. This step verifies the bigger questions of "do we still have a deployable product for each of our target platforms?"
The end result is that most people here never worry about integration testing: it just happens. Unit testing, on the other hand, is everyone's priority. JUnit makes it easy to construct tests, though good tests will always require thought and development time.
Yes, you may use junit for integration tests, but it depends on the type of integration test you need.
Testing a servlet:
setup the servlet context and config
do the tests using mock servlet requests (Spring has support for this, but you may also use EasyMock or your own mocks)
Testing a spring application:
use AbstractDependencyInjectionSpringContextTests to setup the context
test the wired beans
there are also subclasses of AbstractDependencyInjectionSpringContextTests supporting transaction handling when testing with a database.
But pure Junit has its limit. Testing user interfaces is a typical case. You may use selenium for web applications, soapui for webservices or other appropriate tools.
But whatever you use, it should be possible to integrate it in your continious build (cruise control, team city or whatever).
Definitely!
We use a combination of JUnit, ANT tasks to run them, and Hudson for continues integration tests. Works like a charm.
The suggestion depends on your application and your objective.
I've written integration tests in JUnit, but I've also seen people use HtmlUnit (JUnit extension), Selenium, Watir, Fit/Fitness, and even commercial tools like WinRunner and Silk.
So tell us a bit more about your domain and the objectives of your tests and you can probably get a better answer.
There is a very good extension for JUnit called Jitr.
Jitr is a JUnit Integration Test Runner and it allows your web application integration tests to easily run against a lightweight web container in the same JVM as your tests.
See their site for details: http://www.jitr.org/
Update for 2012: Whilst JUnit can be used (and benefits from CI support) JWebUnit and Selenium appear to be eating up the mindshare for Integration Testing.
I think automation and integration tests do not play well together. The very basic problem is environment setup before every test. The more integration-type test bigger setup is needed.
My thoughts on test automation on integration layer: http://blog.aplikacja.info/2012/03/whats-wrong-with-automated-integration-tests/
Related
Often some testing framework for automated testing - like Selenium - is used to continiuosly verify the integrity of a deployed application. These tests often cover real user scenarios and may also utilize a range of deployed applications in combination.
We would like to achieve some what the same for a "backend only" application - that is, an application (or more really) without frontend. We are currently building a series of batchjobs where one job produces input to the next.
We have a great unit-test suite that tests the individual jobs however we would really like to test the series of jobs when deployed to some environment.
Do you have any suggestions for such testing framework? The framework must be able to leverage other Java SDKs such as AWS SDK (e.g. to instruct startup of batchjob, inject data to queues etc.). Whether the framework with tests needs to be deployed as an application as well or run directly from CI is secondary.
If you already have backedn tests that can be run on the production all you need is to schedule running on those tests. Jenkins is fine for that (https://wiki.jenkins.io/display/JENKINS/Schedule+Build+Plugin)
You could have emails (or other alerts) for failed jobs. Jenkins will also care for test reports- exactly as it does for unit tests.
Technologies for scheduling test runs
You could schedule running your tests using any other technology- for example Amazon AWS instances, AWS Elastic Beanstalk Worker Environments (https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html) etc.
I find Jenkins most reasonable, because you have out-of-the box support for test reports, notifications, etc.
For any other technologies you would have to write reportting, notifications on your own.
Technologies for writing tests
I could write tests in any technology that is capable of making HTTP REST calls. For performance tests Jmeter or Gatling are good choices.
For acceptance test you could use RestEasy, TestRestTemplate from Spring, Apache HTTP client, etc.
As test running framework you could use Junit4,Juni5, TestNG or Spock (if you are fine with Groovy language). Test structure could be similar to those of ordinary tests. Well named independend methods that test one thing well, meaningfull assertions, etc.
For writing assertions my personal preference is AssertJ, but JavaHamcrest would also do.
Those tests can (and should) be written in src/test directory, in separate repository (or in same repository or different module).
For that test module you may write test-releated services in src/main directory, so src/test directory would contain only test scenarios. Test services may everyting you need- manage files, inspect database, etc.
You may consider writting test scenarios in BDD-style and tools like JBehave or Cucumber. Personally I see value in BDD tests only if business is interested in test scenarios. If those tests are to be used only by technical people then I find easier to maintain such tests in non-BDD technologies (Junit, AssertJ).
I am trying to create the following setup :
A Selenium (Java) project that has a set of 10 automated test cases.
When this project is executed, it generates an HTML test execution report.
This project should be 'hosted' on an internal network.
Anyone who has access to the network should be able to 'invoke' this project, which in turn executes the test cases and passes the HTML report to the person who invoked it.
The project should be accessible ONLY for execution and the code should NOT be accessible.
My goal is that this implementation should be executable by any framework irrespective of the technology that the framework uses. I was thinking of creating the project as a WebService using Java (servlet).
My question is:
Can this implementation be accessed by any external automation framework ?
Are there any limitations to this implementation?
Is there a better way to implement this requirement?
Thanks in advance.
You can create a maven project and have your automated tests under maven test folder.Configure your tests to run through POM.xml(use maven surefire plugin).Configure a jenkins job to run the maven test.Anybody with access the jenkins can build/run this task!
Below link should give you a headstart
http://learn-automation.com/selenium-integration-with-jenkins/
As a matter of fact, it is something we did on one of our projects. As I cannot share specifics, I will give you overall architectural view of the project.
The core of all things was a service that could run JUnit tests on requests. It was a Soap web-service, but nothing stops you from making it REST. To implement this you need to implement your version of JUnit test runners (see for example: http://www.mscharhag.com/java/understanding-junits-runner-architecture or https://github.com/junit-team/junit4/wiki/Test-runners)
If you use JUnit as test framework for running your Selenuim tests this may be a great solution for you - JUnit will generate HTML reports for you if you configure it properly, it will hide actual test suite implementation from users and run test suite on demand. This solution is also great because it operates on JUnit level and does not care about what kind of tests it actually runs, so it can be also reused for any other kind of automated tests.
So to answer all your questions:
Can this implementation be accessed by any external automation
framework ? -> yes, it can be accessed by anybody who able send http
requests
Are there any limitations to this implementation? -> none that I am
aware of
Is there a better way to implement this requirement? -> well, I
didn't actually work with TestNG much so I don't know if it is
easier or more difficult to do it on Junit level. You can use
Jenkins or other CI tool as well to achieve same results - they can
run JUnit tests for you and almost always have API ready for this,
although those APIs may be not perfect.
So I'd say that if you need this only for one thing you can use CI tools for this purpose, if you don't have CI tools available then choice has been made for you. However, from our experience, having this kind of service was a great asset for a company and I really wonder why there's no such products available elsewhere yet.
I am an entry level tester, mainly been doing manual testing for a company in the UK following scripts on a spreadsheet which I have written in the BDD format, however, I have been learning some automation on the side as that's what I want to move into full time. I have some questions though which are as follows.
I've been using Selenium web driver + java bindings to make simple tests such as logging in or filling out a registration form, i've also set up log4j but only basic to record low level recording. I have now come across testNG. My main question is this framework used by testers? or developers? Is testNG only for unit tests? or UI tests?
From what i've learnt so far the developer does the unit and component tests and the tester does the services/ui tests is this correct?
Unfortunately I was put into a team of developers and not testers as this is my first job outside of university. So I haven't had the chance to learn from other testers. There was no plan for me when I started just that I was going to be the first tester in this development team without any prior testing knowledge.
Which is why I need a bit of guidance on these issues.
My main question is this framework used by testers? or developers? Is
testNG only for unit tests? or UI tests?
TestNG can be used for both, developers and automation testers, it is a tool that can operate over and together with Junit, basically in some cases is being used to create the concept of test suite, that allows to split all the test cases based on specific criteria (time, module, complexity). Also this framework can be used in unit testing and integration testing as well as ui-testing.
TestNG also in some cases replaced Junit entirely, whit this approach you will have a framework with some out of the box capabilities as DataProviders, Multi threading support and other, you could check this link, consider this as and powerful option for Junit.
From what i've learnt so far the developer does the unit and component
tests and the tester does the services/ui tests is this correct?
Unit testing which I consider very similar as "component test" is being done by the developers. If you have web services or a REST API, developers sometimes are in charge of create some test using integration testing, basically verify that services are working as we expected, returning JSON/XML with the correct format and other kind of validations.
Testers also could check services, using tools such as Jmeter, SOAP-UI, they check more things related to the business logic.
Finally I would said UI test is being done in most of the places by the manual and automation testing team, in places where is no QA department this tasks also belongs to the DEV team.
In order to run tests you need to have a test runner it could be anything, most common in java world is JUnit and TestNG, with those frameworks you can run the tests which annotated by #Test tag, also you can group the tests the way you want it and run them in parallel.
Testers use it to run Selenium tests and do assertions, even though for assertions it is good to have knowledge of hamcrest matchers. Also it providing you reports after tests been completed.
Developers would use same frameworks for unit testing purposes.
Check out guys from toolsqa.com they have pretty comprehensive tutorials on using Selenium with TestNG.
TestNG is basically used by developers for doing unit testing, I agree. But it is also widely used by system test automation using Selenium. This framework is inspired by JUnit framework, and most of the automation test developers use this framework because of its advantages and more added features to support reporting.
I can say following advantages I got by using this framework:
1.Support for parameters.
2.Supports dependent methods testing.
3.Test configuration flexible. Supports powerful execution model.
4.Embeds BeanShell for further flexibility.
5.TestNG has a more elegant way of handling parameterized tests with the data-provider concept.
6.For the same test class TestNG support for multiple instances.
7.Extendibility of using different Tools and plug-ins like Eclipse, Maven, IDEA etc.
8.Default JDK functions for runtime and logging (no dependencies).
9.Supported different Annotations like #BeforeSuite, #AfterSuite, #BeforeClass, #AfterClass, #BeforeTest, #AfterTest, #BeforeGroups, #AfterGroups, #BeforeMethod, #AfterMethod, #DataProvider, #Factory, #Listeners, #Parameters, #Test.
The most beautiful part I found in testNG is, using data provider, i can easily read test inputs and expected results from excel. And I can able to see the Results of Pass/Fail and skip test cases in an emailable format.
For testing a system, we don't need any training/extra classes. Just if we know the system requirements, and this as a end user what they want from the system and start testing. If any deviations found in the system behavior and are not as per the expectations of user. Then mark it as an issue and raise a defect and track it until it get resolved. Retest the same and confirm that the system is working as per the expectations. even at the Unit test level this principle holds the same. But only the difference is that we can do Structure based testing there.
To your questions ..
1.My main question is this framework used by testers? or developers? Is testNG only for unit tests? or UI tests?
Answer = Test NG can be used for unit testing as well as UI testing. the advantage of test NG over JUNIT is that you dont need to write code for test result reporting.
I am working on a Spring MVC application.
Unit tests are already writen (nice 85% code coverage and lots of assertions :)
Now I need to write integration tests. I already have a look at stackoverflow still I do have some questions
Right now I am using a standard maven structure with main and test directories, do you recommend to create another directory called integration-tests and write the unit tests there? If so, why?
Another alternative would be to write the integration tests in the "tests" directory, mixing unit and integration tests, and then using maven to run one or the other (maybe using different file suffixes depending on the test type)
In any case, the way I am planning to run the integration tests is essentially to (almost) reuse the unit tests of the controllers WITHOUT injecting mocks, that means that my current stack (Controller-Service-DAO) will contain no mocks (of course, in the unit tests, it does), by removing the mocks I will access to the real resources (Database and so), is this a good approach?
UPDATE: Just to clarify, the project has not JSP or any HTML-related views, output is XML which can be easily validated with XSDs
I am assuming that you have used JUNIT to create your unit tests to achieve the impressive 85% code coverage. Please notice that JUNIT is designed for unit testing only (thus the name JUNIT). Unit testing is done while the code is running in the development environment.
Integration testing can only be performed once the target code has been deployed in the target integration environment.
You have mentioned that you application isn't a web application. Is it SOAP/Rest Web Service? If so, you can use Soap UI [http://www.soapui.org/], to create and save automated regression/integration tests.
I recently managed to convince my mates in the project that we need testing (!). Due to the highly dynamic and flexible structure of our web application, with behavior depending of lots of parameters and permission relationships, they had rejected testing altogether, for the usual reasons (time consuming, test maintenance, etc.).
We will introduce testing at the service layer:
Web Browser -> GWT/RPC -> GWT Servlet -> RMI -> SessionEJB -> RMI -> Spring beans
Thus after the GWT Servlet.
Do people recommend to use junit? Or are there other test frameworks better suited? Any other general suggestions? Thanks
You can indeed use plain JUnit or TestNG with a mock framework to test your SessionEJB and individual Spring beans in isolation, i.e. proper Unit testing.
But since there is already a lot of code written, you'll probably find more bugs with less code using system testing or integration testing, i.e. test your complete SessionEJB and spring beans roundtrip in a test application context, with even a real database behind.
For integration and system testing, you can use DBUnit to have a fixture of test data in a database. And Spring also has a lot of test support utils. All of this things work with both JUnit and TestNG.
You should be able to JUnit your Servlets & EJBs. I suggest using some kind of mock framework (e.g. EasyMock) for your servlet context and if you are using any kind of JNDI resource or dependency injection.
As for a testing framework, I highly recommend TestNG (http://testng.org), with Mockito (code.google.com/p/mockito/). I love using both due to their ease of use. #DataProvider in TestNG helps me a lot, as well as other annotations for setting up a test before/after running. I was using JUnit before until I met TestNG at work and don't think I'll be going back anytime soon :)
Check them out, TestNG is definitely picking up some steam and gaining reputation.