I have a Java application which uses Gradle as build system. This is multi-project built (several Java projects).
When I run gradlew test and some of the tests are failing I've noticed not all tests actually run.
If I add the following code to build.gradle it does run all tests but of course, the built always complete successfully:
test {
ignoreFailures = true
}
I have two questions:
Does gradle stop executing tests after the first failure?
Is there a way to run all tests (even when some fail) and still have the built fail if some of them failed?
Gradle offers the --continue parameter gradlew build --continue which executes all tasks regardless of their success. However, if any executed task failed the whole build is marked as failed.
I think you are looking for that switch.
Make sure you are running tests from the root of you project.
The test task is part of Java Plugin for Gradle. As from the official documentation:
Projects with large test suites can take a long time to execute even though a failure occurred early on leading to unnecessary wait times (especially on CI). To short circuit this behavior, the Test.getFailFast() property allows you to cause the test task to fail after the first test failure instead of running all tests. When this property is true, the resulting output will only show the results of tests that have completed up to and including the failure. To enable this fail fast behavior in your build file, set the failFast property to true:
test {
failFast = true
}
The --fail-fast command line option enables the behavior from the command line. An invocation looks like:
gradle test --fail-fast
The default value for the failFast property is false.
Official documentation can be found here.
So to answer your questions:
By default, Gradle does not stop executing tests after the first failure. This behaviour can be modified using failFast property.
Default behaviour should do the trick. By running gradle test all tests will be run. If any of them fail, the build will fail. Since you have a multi-project build, make sure you are running Gradle from your root project.
Related
"Test" tasks in my multi-module build are not being cached by Gradle. Incremental build works fine and reports everything as up-to-date, but if I perform clean build tests are always executed.
I am seeing this message Non-cacheable because 'Task outputs cacheable' not satisfied [ENABLE_CONDITION_NOT_SATISFIED] for all the test tasks if org.gradle.caching.debug parameter set to true. All fingerprints are identical between executions.
Any ideas what does it mean and/or how to fix it?
P.S. Using Gradle 6.7 with Junit 5 and JaCoCo. Disabling JaCoCo does not change anything.
I'm working on splitting a custom test task into build and test phase (I want to build the sources with Java 8 and run them with Java 11 without recompilation).
The task is defined here
Running the task with -x test flag doesn't disable the tests. I want to be able to do it all from command line in order to execute the task as part of a Jenkins job.
So far I failed to come up with a solution other than chaining 5 bash commands and adding the tasks from --dry-run to the ./gradle command and later excluding them with -x, but I believe there must be a more elegant approach.
If I missed some information required to provide some answers, please let me know, I'll be happy to fill you in.
If I understand correctly, what you are looking for is exactly what the Java plugin does with its default tasks:
compileTestJava is the task responsible for compiling the tests.
test is the task responsible for running the tests, and it expresses a dependency on compileTestJava
So you would model your tasks the same way:
One JavaCompile task, configured with Java 8, to compile the tests.
One Test task, configured with Java 11, to run the tests, with a dependency on the previous task.
Then the up-to-date system of Gradle should do the trick, that is only run the Test task when the JavaCompile task is up-to-date.
And in order to run only the test compilation, you would invoke just the matching task. Assuming it defines its own dependencies properly, it would do only what's required.
If you need more specific guidance on how to defined these tasks, add more information to the question.
We have currently set up our application set up with Junit 4 and Maven with the surefire plugin.
Now we are migrating to gradle and we want to run the test case in parallel with the exception of the ones annotated with #NotThreadSafe. Given the long running nature of our test cases it is hard to find out if they are running in parallel or not.
Has anybody used the #NotThreadSafe annotation together with gradle?
In the gradle docs I wasn't able to find out if it is supported or not.
Two tests fail in my build service that will not fail when ran locally.
What I've discovered is that when I execute mvn clean test locally the sort order of my test classes are alphabetical (package and class), but when ran in the build service they're seemingly random.
It is not my intention of needing a specific "order" so I'm definitely concerned that one test prior to another is hinting at an isolation issue. However, I don't think I can just tell maven to run classes in a certain order.
What can I do to try to reproduce?
EDIT
I cloned my build plan, but pointed at a forked repo (same code) and it ran successfully with no failed tests...
You should try to execute mvn -Dsurefire.runOrder=random clean test for random order locally.
But even if your build fails locally, random order is not the best option to reproduce failures because of test order. If the tests are all green when run in alphabetical order, you might be able to make your build consistently fail with mvn -Dsurefire.runOrder=reversealphabetical clean test
There are also a couple of more options -- see runOrder documentation
I have a project with nearly 500 individual tests in around 200 test classes. Some of these tests don't do a great job of tearing down their own state after they're finished, and in Eclipse this results in some tests failing. The tests all pass when running the test suite from the command line via Ant.
Can I enable 'test isolation' somehow in Eclipse? I don't mind if it takes longer to run.
Long term, I will clean up the misbehaving tests, but in the short term I'd like to get the tests working.
If you use Ant in Eclipse, you can set the JUnit task to fork a new JVM process for each test, providing isolation.
http://ant.apache.org/manual/Tasks/junit.html
I also had similar needs and developed small maven plugin Jute which has been published in maven central. It starts external JVM process for each JUnit test method.