We're migrating some ant build scripts to gradle and are diagnosing issues along the way. One problem that has popped up is that on the CI server (jenkins running gradle) we occasionally get test failures. We think the issue is related to test execution order because one of the tests that is failing uses thread local storage in some library code.
I would like to be able to reproduce the problem locally before fixing the broken tests. However. I can't reproduce the problem locally because gradle always runs the tests in an order that happens to work.
So, is there a way to force gradle to run test class X before test class Y? The tests need to run in the same JVM - one test right after the other.
In case it matters, the tests are JUnit tests.
Yes, it is possible. One of the possibilities is to create additional task for test Y run
task YTest(type: Test) {
include '**/Y.*'
}
test {
exclude '**/Y.*'
}
test.dependsOn YTest
Per Peter's response on the gradle forum, this is not possible.
http://forums.gradle.org/gradle/topics/can_gradle_run_two_tests_in_a_specific_desired_order_in_the_same_jvm?utm_content=reply_link&utm_medium=email&utm_source=reply_notification#reply_13187620
An alternative is to use a test suite.
testNG (a Junit alternative, also supported by gradle) also allows you to have control on test order. See this post for more details.
Related
Is there a way to run Karate test during maven's integration test phase? It seems that surefire plugin is hardcoded into Karate. I have tried to override it using failsafe plugin but with no luck. I don't want test to run along with unit tests.
Thank in advance!
It seems that surefire plugin is hardcoded into Karate
I'm not sure where you got that impression, but no, the surefire plugin is not hardcoded into Karate.
Keep in mind that the simplest way to not run a JUnit test via surefire is to not use the *Test.java naming convention.
I think the solution for you is simple, whichever JUnit test is the "entry-point" for your Karate tests (the parallel runner is recommended) - just use the failsafe naming conventions.
And then, just include the failsafe plugin as per the examples and it should work. If you have trouble getting that to work (unlikely), then you should look at maven profiles.
EDIT: also see this comment: Is there a way to run Karate tests as an integration test suite against a pre-booted spring boot server?
Turns out that I cannot be done and it is a limitation of Maven, not Karate. Howto add another test source folder to Maven and compile it to a separate folder? - Here is my test project to prove it out: https://github.com/djangofan/spring-boot-hello - Thanks for leading me down what appears to have been the correct path to discover the limitation. Using Gradle would likely solve my issue but that is not an option on my project. If I use Karate for "separated integration tests", I need a separate mvn test module.
Two tests fail in my build service that will not fail when ran locally.
What I've discovered is that when I execute mvn clean test locally the sort order of my test classes are alphabetical (package and class), but when ran in the build service they're seemingly random.
It is not my intention of needing a specific "order" so I'm definitely concerned that one test prior to another is hinting at an isolation issue. However, I don't think I can just tell maven to run classes in a certain order.
What can I do to try to reproduce?
EDIT
I cloned my build plan, but pointed at a forked repo (same code) and it ran successfully with no failed tests...
You should try to execute mvn -Dsurefire.runOrder=random clean test for random order locally.
But even if your build fails locally, random order is not the best option to reproduce failures because of test order. If the tests are all green when run in alphabetical order, you might be able to make your build consistently fail with mvn -Dsurefire.runOrder=reversealphabetical clean test
There are also a couple of more options -- see runOrder documentation
In Java, you can create a junit test suites and put all your junit test cases in it. This allows you to run all your test cases all at once and get the testing results immediately (e.g. how many tests passsed and failed, and which tests failed). Is there something equivalent in Scala within the ScalaTest?
Thanks
I don't know if there's anything equivalent for ScalaTest but in general this is a bad idea. It requires you to keep your test suite class up to date when you add new tests. Both your IDE and your build tool should let you be able to automatically discover and run all tests at once.
If you're using maven or gradle, just placing all your tests under the src/test/scala directory should be enough that running the test target will execute all test
I'm using maven-cobertura-plugin in order to calculate code coverage in my project. As I understand, this plugin starts a new/forked build cycle in order to compile and test the code base. When it's done the plugin calculates code coverage. As I understand, this is the only approach the plugin can use, and it's fine for me.
The problem is that before cobertura plugin my code base is compiled and tested already. Thus, I'm experiencing duplicated compilation and testing. Is it possible to avoid compilation and testing before cobertura? Or maybe there is some other workaround?
Is it possible to avoid compilation and testing before cobertura? Or maybe there is some other workaround?
There are several issues about this (see MCOBERTURA-83, MCOBERTURA-76) but AFAIK, there is no perfect workaround (due to the way the life cycle is constructed - things might be improved in Maven 3).
The only one I'm aware of (works with CI servers) would be to run:
mvn clean install -Dmaven.test.skip=true
and then
mvn cobertura:check
Instead of binding cobertura:check on the build lifecycle.
Note that compiling twice shouldn't be an issue as all classes should be up to date.
As far as I know, cobertura needs to do bytecode weaving on your code to be able to work.
The only way I was able to work around that was to instrument the byte code as part of my build (by binding the cobertura:instrument goal to the verify phase and also bind the default-test execution from maven-surefire-plugin to the verify phase so it doesn't get executed as part of the test phase on every cobertura goal execution.
I have a project with nearly 500 individual tests in around 200 test classes. Some of these tests don't do a great job of tearing down their own state after they're finished, and in Eclipse this results in some tests failing. The tests all pass when running the test suite from the command line via Ant.
Can I enable 'test isolation' somehow in Eclipse? I don't mind if it takes longer to run.
Long term, I will clean up the misbehaving tests, but in the short term I'd like to get the tests working.
If you use Ant in Eclipse, you can set the JUnit task to fork a new JVM process for each test, providing isolation.
http://ant.apache.org/manual/Tasks/junit.html
I also had similar needs and developed small maven plugin Jute which has been published in maven central. It starts external JVM process for each JUnit test method.