In our project, we have a plenty of unit tests. They help to keep project rather well-tested.
Besides them, we have a set of tests which are unit tests but depends on some kind of external resource. We call them external tests. For example, they can sometimes access web-services.
While unit tests are easy to run, the integrational tests couldn't pass sometimes: for example, due to timeout error. Also, these tests can take too much time to run.
Currently, we keep integration/external unit tests just to run them when developing corresponding functionality.
For plain unit tests, we use TeamCity for continuous integration.
How do you run the integration unit tests and when do you run them?
In our project we have separate suite for regular/plain unit tests and separate suite for integration tests. The are two reasons for that:
performance: integration tests are much slower,
test fragility: integration tests fail more often due to environment-related conditions (give false positives).
We use TeamCity as our main Continuous Integration server and Maven as build system. We use the following algorithm to run the tests:
We run unit tests at within Eclipse IDE and before every commit.
We run unit tests automatically after each commit on TeamCity agents using Maven's mvn clean install
We run integration tests automatically on TeamCity agent after "main" build is completed.
The way we trigger integration tests execution is by configuring TeamCity's integration.tests task to be dependent on "main" continous.build task, see here for details: http://confluence.jetbrains.net/display/TCD4/Dependencies+Triggers
We run only integration tests (excluding unit tests) by:
using separate directory named
"src/it/java" to keep integration
tests,
excluding by default this source folder from maven-surefire-plugin configuration (configuration/excludes element),
using Maven profile called "integration" to exclude regular unit tests and include tests from "src/it/java" (this profile is configured by passing -Pintegration in integration.tests task).
We're using Maven2: maven-surefire-plugin to run unit tests (in the test phase) and maven-failsafe-plugin for integration tests (integration-test phase).
By default, all tests run when the project is built, however integration tests can be turned off using profiles.
In many cases integration tests are the part of the module, n some cases there are also dedicated modules which only do integration tests.
One of the teams also uses Fitnesse for acceptance testing. These tests are also in dedicated modules.
We're using Hudson for CI.
We run all the tests in one huge suite. It takes 7 minutes to run.
Our integration tests create mock servers. They never time out -- except when the test requires the server to time out.
So we have the following kinds of things. (The code sample is Python)
class SomeIntegrationTest( unittest.TestCase ):
def setUp( self ):
testclient.StartVendorMockServer( 18000 ) # port number
self.connection = applicationLibrary.connect( 'localhost', 18000 )
def test_should_do_this( self ):
self.connection.this()
self.assert...
def tearDown( self ):
testClient.KillVendorMockServer( 18000 )
This has some limitations -- it's always forking the client mock server for each test. Sometimes that's okay, and sometimes that's too much starting and stopping.
We also have the following kinds of things
class SomeIntegrationTest( unittest.TestCase ):
def setUp( self ):
self.connection = applicationLibrary.connect( 'localhost', 18000 )
def test_should_do_this( self ):
self.connection.this()
self.assert...
if __name__ == "__main__":
testclient.StartVendorMockServer( 18000 ) # port number
result= unittest.TextTestRunner().run()
testclient.KillVendorMockServer( 18000 )
system.exit( result.failures + result.errors )
To support this testing, we have a number of mocked-up servers for various kinds of integration tests.
We use Jenkins to run our tests automatically.
Be careful of differencing between Unit and Integration - Tests. It is confusing to talk about "Integration Unit Tests"
Maven offers good support to distinguish between Unit and Integration Tests-
Failsafe & Surefire Plugin.
From Apache Maven Project:
The Failsafe Plugin is designed to run integration tests while the Surefire Plugin is designed to run unit tests.
(see: http://maven.apache.org/surefire/maven-failsafe-plugin/)
You need to configure these Plugins in your pom.xml
You then only use mvn test - to run unit tests or mvn verify to run integration tests.
Unit test should run periodically f.i. every 15 min.
Integration Test, normally take long, and should run f.i. every 24 hours.
Hope that helps others.
Related
I'm using JenkinsFile to manage our pipeline.
Since, we integrated maven-failsafe-plugin to run the unit tests and integration tests separately, we get separate result reports from both plugins.
This is how i configured the same in JenkinsFile in the post section:
junit testResults: '**/target/surefire-reports/*.xml, **/target/failsafe-reports/*.xml'
I would have expected Jenkins to show unit tests and integration tests separately, but unfortunately, Jenkins merges the results of both XML and there is no clear distinction between unit test and integration tests.
We want our builds to have a separate view on Integration tests results.
Is there anyway to do that with Jenkins?
I dont know a way how to do that in Jenkins itself, but maybe a workaround. After you use mvn clean verify you can use mvn site in a jenkins pipeline and build a HTML representation. These testresults are splitted between failsafe and surefire and you can open it directly from your workspace and look into the diffrent results from failsafe and surefire plugin.
In Java, you can create a junit test suites and put all your junit test cases in it. This allows you to run all your test cases all at once and get the testing results immediately (e.g. how many tests passsed and failed, and which tests failed). Is there something equivalent in Scala within the ScalaTest?
Thanks
I don't know if there's anything equivalent for ScalaTest but in general this is a bad idea. It requires you to keep your test suite class up to date when you add new tests. Both your IDE and your build tool should let you be able to automatically discover and run all tests at once.
If you're using maven or gradle, just placing all your tests under the src/test/scala directory should be enough that running the test target will execute all test
Hibernate has its own test infrastructure, that can be used to test Hibernate and its various dialects.
The infrastructure is based on JUnit (the Tests) and Gradle (automation of the test process).
More on it you can find here:
https://github.com/hibernate/hibernate-matrix-testing
https://github.com/hibernate/hibernate-orm/wiki/Hibernate-JUnit-Infastructure
You can start all tests from the test suite using gradle:
gradle hibernate-core:matrix_mysql51
In this case all tests of the hibernate-core module will be started.
There are more than 4000 tests in the module.
I would like to start only some of them.
How do it do it?
Is it possible to use the same testing infrastructure, but start single tests from the testsuite?
gradle hibernate-core:matrix_mysql51 -Dmatrix_mysql51.single=annotations/EntityTest
I have like trillion test packages with bazillion tests and I want to run just some of packages. Now I must run whole project (some tests takes long to complete) or I need to run every single file manually. How is possible to run just some packages in NetBeans ? I can't find this option ...
It's probably not what you want, but the NetBeans help topic, Running a JUnit Test, says:
If you want to run a subset of the
project's tests or run the tests in a
specific order, you can create test
suites that specify the tests to run
as part of that suite. After creating
a test suite you run the suite in the
same way you run a single test class.
Creating a test suite is covered in the topic Creating a JUnit Test.
If you use JUnit 4 then try ClasspathSuite and its regex filters.
I have a project with nearly 500 individual tests in around 200 test classes. Some of these tests don't do a great job of tearing down their own state after they're finished, and in Eclipse this results in some tests failing. The tests all pass when running the test suite from the command line via Ant.
Can I enable 'test isolation' somehow in Eclipse? I don't mind if it takes longer to run.
Long term, I will clean up the misbehaving tests, but in the short term I'd like to get the tests working.
If you use Ant in Eclipse, you can set the JUnit task to fork a new JVM process for each test, providing isolation.
http://ant.apache.org/manual/Tasks/junit.html
I also had similar needs and developed small maven plugin Jute which has been published in maven central. It starts external JVM process for each JUnit test method.