Is there a way to run Karate test during maven's integration test phase? It seems that surefire plugin is hardcoded into Karate. I have tried to override it using failsafe plugin but with no luck. I don't want test to run along with unit tests.
Thank in advance!
It seems that surefire plugin is hardcoded into Karate
I'm not sure where you got that impression, but no, the surefire plugin is not hardcoded into Karate.
Keep in mind that the simplest way to not run a JUnit test via surefire is to not use the *Test.java naming convention.
I think the solution for you is simple, whichever JUnit test is the "entry-point" for your Karate tests (the parallel runner is recommended) - just use the failsafe naming conventions.
And then, just include the failsafe plugin as per the examples and it should work. If you have trouble getting that to work (unlikely), then you should look at maven profiles.
EDIT: also see this comment: Is there a way to run Karate tests as an integration test suite against a pre-booted spring boot server?
Turns out that I cannot be done and it is a limitation of Maven, not Karate. Howto add another test source folder to Maven and compile it to a separate folder? - Here is my test project to prove it out: https://github.com/djangofan/spring-boot-hello - Thanks for leading me down what appears to have been the correct path to discover the limitation. Using Gradle would likely solve my issue but that is not an option on my project. If I use Karate for "separated integration tests", I need a separate mvn test module.
We have currently set up our application set up with Junit 4 and Maven with the surefire plugin.
Now we are migrating to gradle and we want to run the test case in parallel with the exception of the ones annotated with #NotThreadSafe. Given the long running nature of our test cases it is hard to find out if they are running in parallel or not.
Has anybody used the #NotThreadSafe annotation together with gradle?
In the gradle docs I wasn't able to find out if it is supported or not.
I am running Jenkins on Windows 7 as service.
I have approximately 100 selenium test cases in package, but I want to run only 1 test case.
Is it possible to configure Jenkins to run not all, but 1 test?
Depending on how your Jenkins build is configured, somewhere you are specifying the maven goals. These can be configured to specify the test you want to execute, as described here:
mvn -Dtest=TestCircle#mytest test
Note that this require surefire 2.7.3 or later, and Junit 4.X or TestNG:
As of Surefire 2.7.3, you can also run only a subset of the tests in a test class.
NOTE: This feature is supported only for Junit 4.x and TestNG.
For older versions of the Surefire plugin/JUnit, it seems it is only possible to control this on class level, i.e. you cannot specify a specific test in a class:
mvn -Dtest=TestCircle test
I have a multi module maven project
JAR module A
JAR module B depends on A
WAR module C depends on B
Each of the modules has three types of tests.
Unit tests that don't depend on any external resources being present
Database integration tests that only depend on external database being present
REST api tests that depend on the application running in tomcat before they can run
I would like maven to execute the tests in the following order.
All unit tests for modules A,B,C
All database tests for modules A,B,C
All REST api tests for module A,B,C
I know how to use the surefire and the failsafe plugins to separate the unit tests from the integration tests. However i don't know how to make the failsafe plugin split the integration tests into two groups the database tests and the rest api tests.
How can the failsafe plugin be configured to split integration tests into groups that can be run separately?
All tests are written with TestNG.
In our project, we have a plenty of unit tests. They help to keep project rather well-tested.
Besides them, we have a set of tests which are unit tests but depends on some kind of external resource. We call them external tests. For example, they can sometimes access web-services.
While unit tests are easy to run, the integrational tests couldn't pass sometimes: for example, due to timeout error. Also, these tests can take too much time to run.
Currently, we keep integration/external unit tests just to run them when developing corresponding functionality.
For plain unit tests, we use TeamCity for continuous integration.
How do you run the integration unit tests and when do you run them?
In our project we have separate suite for regular/plain unit tests and separate suite for integration tests. The are two reasons for that:
performance: integration tests are much slower,
test fragility: integration tests fail more often due to environment-related conditions (give false positives).
We use TeamCity as our main Continuous Integration server and Maven as build system. We use the following algorithm to run the tests:
We run unit tests at within Eclipse IDE and before every commit.
We run unit tests automatically after each commit on TeamCity agents using Maven's mvn clean install
We run integration tests automatically on TeamCity agent after "main" build is completed.
The way we trigger integration tests execution is by configuring TeamCity's integration.tests task to be dependent on "main" continous.build task, see here for details: http://confluence.jetbrains.net/display/TCD4/Dependencies+Triggers
We run only integration tests (excluding unit tests) by:
using separate directory named
"src/it/java" to keep integration
tests,
excluding by default this source folder from maven-surefire-plugin configuration (configuration/excludes element),
using Maven profile called "integration" to exclude regular unit tests and include tests from "src/it/java" (this profile is configured by passing -Pintegration in integration.tests task).
We're using Maven2: maven-surefire-plugin to run unit tests (in the test phase) and maven-failsafe-plugin for integration tests (integration-test phase).
By default, all tests run when the project is built, however integration tests can be turned off using profiles.
In many cases integration tests are the part of the module, n some cases there are also dedicated modules which only do integration tests.
One of the teams also uses Fitnesse for acceptance testing. These tests are also in dedicated modules.
We're using Hudson for CI.
We run all the tests in one huge suite. It takes 7 minutes to run.
Our integration tests create mock servers. They never time out -- except when the test requires the server to time out.
So we have the following kinds of things. (The code sample is Python)
class SomeIntegrationTest( unittest.TestCase ):
def setUp( self ):
testclient.StartVendorMockServer( 18000 ) # port number
self.connection = applicationLibrary.connect( 'localhost', 18000 )
def test_should_do_this( self ):
self.connection.this()
self.assert...
def tearDown( self ):
testClient.KillVendorMockServer( 18000 )
This has some limitations -- it's always forking the client mock server for each test. Sometimes that's okay, and sometimes that's too much starting and stopping.
We also have the following kinds of things
class SomeIntegrationTest( unittest.TestCase ):
def setUp( self ):
self.connection = applicationLibrary.connect( 'localhost', 18000 )
def test_should_do_this( self ):
self.connection.this()
self.assert...
if __name__ == "__main__":
testclient.StartVendorMockServer( 18000 ) # port number
result= unittest.TextTestRunner().run()
testclient.KillVendorMockServer( 18000 )
system.exit( result.failures + result.errors )
To support this testing, we have a number of mocked-up servers for various kinds of integration tests.
We use Jenkins to run our tests automatically.
Be careful of differencing between Unit and Integration - Tests. It is confusing to talk about "Integration Unit Tests"
Maven offers good support to distinguish between Unit and Integration Tests-
Failsafe & Surefire Plugin.
From Apache Maven Project:
The Failsafe Plugin is designed to run integration tests while the Surefire Plugin is designed to run unit tests.
(see: http://maven.apache.org/surefire/maven-failsafe-plugin/)
You need to configure these Plugins in your pom.xml
You then only use mvn test - to run unit tests or mvn verify to run integration tests.
Unit test should run periodically f.i. every 15 min.
Integration Test, normally take long, and should run f.i. every 24 hours.
Hope that helps others.