So i have a gradle test task which runs all my tests. How can I set up gradle to run this task 100 times? It works and runs all my tests, I just need an option to choose how many times to run this.
The task in build.gradle:
test {
// enable JUnit Platform (a.k.a. JUnit 5) support
useJUnitPlatform()
// set a system property for the test JVM(s)
systemProperty 'some.prop', 'value'
// explicitly include or exclude tests
include 'com/company/calculator/**'
// show standard out and standard error of the test JVM(s) on the console
testLogging.showStandardStreams = true
// set heap size for the test JVM(s)
minHeapSize = "128m"
maxHeapSize = "512m"
// set JVM arguments for the test JVM(s)
jvmArgs '-XX:MaxPermSize=256m'
// listen to events in the test execution lifecycle
beforeTest { descriptor ->
logger.lifecycle("Running test: " + descriptor)
}
// Fail the 'test' task on the first test failure
failFast = true
// listen to standard out and standard error of the test JVM(s)
onOutput { descriptor, event ->
logger.lifecycle("Test: " + descriptor + " produced standard out/err: " + event.message )
}
The use case is that i want to test performance of different assertions and mocking libraries (i have multiple branches with tests written using different libraries), to do that i need to run test suite multiple times.
To test performance i need to measure the time it takes to run these tests for example 100 times (maybe 1000), on each libraries set.
One option might be this --rerun-tasks flag.
gradle test --rerun-tasks
From the Gradle user guide.
Another option, from a similar question, is creating a subclass of the Test class that returns a task with multiple copies of all tests, code here: https://stackoverflow.com/a/41650455/1686615 .
There really are many ways to do this at different levels, with Gradle code as in that link, or perhaps in .gradle files, with a parameter passed into the test code, or on the command line. Maybe indicate more about your use case or if there is a particular level at which you'd like to make the change.
Related
I have a separate task to run the test that generates open-api.yml specification for my application. Here is the Gradle configuration below:
task generateOpenApiYml(type: Test) {
group = 'verification'
useJUnitPlatform {
includeTags 'openApi'
}
testLogging {
showExceptions true
showStandardStreams = false
showCauses true
showStackTraces true
exceptionFormat "full"
events("skipped", "failed", "passed")
}
}
So, I have one test with openApi JUnit tag. It works very well, but there is a slight thing I want to approve.
The whole idea of this test is that the result open-api.yml file is used to generate Java client to invoke REST endpoints. This client is used in other tests in the project.
I want to put those generated Java classes to .gitgnore because they are generated anyway and there is no need to index those files. The problem is that the generateOpenApiYml task compiles all the tests in src/test/java directory. Therefore, some of them use generated classes. Which leads to compiling errors.
Is it possible to tune the generateOpenApiYml task to make it compile and run only the single test with openApi JUnit tag? Then I could easily put generated Java classes to .gitignore and don't bother about their temporary absence, because other tests won't be compiled.
I figured out the solution. Firstly, I installed the gradle-testsets-plugin.
Then I configured a separate source set like this:
testSets {
generateOpenApiYml
}
So, generateOpenApiYml is the new Gradle task that looks for sources in src/generatedOpenApiYml/java directory.
Afterwards, we need to tune all tasks of test type to bind them with JUnit 5 platform.
tasks.withType(Test) {
group = 'verification'
useJUnitPlatform()
testLogging {
showExceptions true
showStandardStreams = false
showCauses true
showStackTraces true
exceptionFormat "full"
events("skipped", "failed", "passed")
}
}
generateOpenApiYml.outputs.upToDateWhen { false }
I put the upToDateWhen option for convenience to make sure the generateOpenApiYml task is always run on demand and never cached.
Then I have the open-api-generator.
openApiGenerate {
inputSpec = "$buildDir/classes/java/generateOpenApiYml/open-api.json".toString()
outputDir = "$buildDir/generated/openapi".toString()
apiPackage = "..."
invokerPackage = "..."
modelPackage = "..."
configOptions = [
dateLibrary : "java8",
openApiNullable: "false",
]
generatorName = 'java'
groupId = "..."
globalProperties = [
modelDocs: "false"
]
additionalProperties = [
hideGenerationTimestamp: true
]
}
tasks.named('openApiGenerate') {
dependsOn generateOpenApiYml
}
Finally, I just need to run two commands to build and run tests for my whole project.
./gradlew openApiGenerate
./gradlew build
The first one creates the open-api.yml file and generates Java client according to the provided specification. The second one runs tests and build the project normally. The tests running during the build phase uses classes generated by openApiGenerate task. Therefore, I can put them to .gitignore safely.
Hope this will be helpful.
When I build my project with gradle, the test outputs are huge and I would like to keep them. Therefore, I activated showing the output streams:
test {
testLogging.showStandardStreams = true
}
Unfortunately, gradle does not seem to be able to handle big standard outputs. For demonstration, I created a minimal example: https://github.com/DaGeRe/stdout-test It is a project creating big standard output in a test by
#Test
public void test() {
long start = System.currentTimeMillis();
for (int i = 0; i < 200; i++) {
for (int j = 0; j < 10000; j++) {
long current = System.currentTimeMillis() - start;
System.out.println("This is a simple logging output: " + i + " " + j + " " + current);
}
}
}
If I run this in a standard maven project, it finishes in about 2 minutes:
reichelt#reichelt-desktop:~/workspaces/stdout-test$ time mvn test &> mvn.txt
real 1m34,130s
user 0m31,333s
sys 1m12,296s
If I run it in gradle by time ./gradlew test &> gradle.txt, it does not finish at all (in reasonable time) and the output contains many Expiring Daemon because JVM heap space is exhausted. A way to solve this temporarily would be increasing heap memory (like suggested here: JVM space exhausted when building a project through gradle ), but -Xmx4g does not change anything according to my experiments, and this obviously will not scale for bigger outputs. Also, running ./gradlew -i test does not change the behavior.
The project also contains example files for outputs from maven (https://github.com/DaGeRe/stdout-test/blob/master/mvn.txt) and gradle (https://github.com/DaGeRe/stdout-test/blob/master/gradle.txt - aborted process after ~10 minutes) and one of the heap dumps (https://github.com/DaGeRe/stdout-test/blob/master/java_pid10812.hprof.tar) gradle created. There are only minor increases in the current time in the gradle-log (third output number of every line). Therefore, I assume that gradle mainly has problems printing to stdout and not executing the program.
This shows that, while gradle has some problems printing to stdout, it seems to not block the test execution. Is there any switch or parameter I could give gradle, which forces gradle to directly print to stdout instead of doing its memory-intense processing? Unfortunately, I did not find any in the documentation (https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html).
EDIT Just finished a test run on a server:
reichelt#r147:~/workspaces/dissworkspace/stdout-test$ time ./gradlew test &> gradle.txt
real 28m17,959s
user 216m37,351s
sys 0m12,410s
Ends with an exception:
This is a simple logging output: 89 7842 1416
This is a simple logging output: 89 7843 1416
This is a simple logging output: 89 7844 1416
This is a simple logging output: 89 7845 1416
This is a simple logging output: 89 7846 1416
FAILURE: Build failed with an exception.
* What went wrong:
GC overhead limit exceeded
I don't really have a solution for you, but I want to share a few observations that are too long to put into a comment.
First of all, I can reproduce your OutOfMemory problem from your Github repository. I googled it a bit, and while there are other reports on OOM on this out there, none had a solution. I think it is just a limitation in Gradle when enabling showStandardStreams. I tried fiddling around with the console output type and a few other parameters, but none had an effect.
However, with disabling showStandardStreams, I would not get an OOM. Not even after bumping the number of iterations from 200*10000, that you specified, to 1000*10000. It worked fine and the output got saved to both a .bin, .xml and a .html file for later inspection.
What's more, Gradle ran it more than twice as fast as Maven on my machine:
λ time ./gradlew test &> gradle.txt
real 1m23.113s
user 0m0.015s
sys 0m0.031s
λ time mvn test &> mvn.txt
real 3m6.671s
user 0m0.183s
sys 0m0.566s
Not sure why there is such a big difference between the two.
While I completely agree that it would be nice to use showStandardStreams for large outputs, just like Maven defaults to, it appears it is just not possible unless you can afford to raise the maximum heap size accordingly. On the other hand, having the output saved in the report is also rather nice, which is something you don't get from the Surefire plugin in Maven.
I am a Gradle rookie and I am not sure whether Gradle will start the new JVM when it runs the test set.
Like Passing JVM arguments to Gradle test task I want to pass some parameters to the JVM running the test set.
I added the following lines to build.gradle:
...
test {
groovy {
jvmArgs '-agentpath:/usr/lib/code_dependency_capturer.so' // add line
srcDirs = ['src/test']
if (!JavaVersion.current().isJava8Compatible()) {
exclude '**/v8/*'
exclude '**/vm8/*'
}
}
resources {
srcDirs = ['src/test-resources']
}
}
...
But it tells me:
A problem occurred evaluating root project 'groovy'.
Could not find method jvmArgs() for arguments[-agentpath:/usr/lib/code_dependency_capturer.so] on source set 'test' of type org.gradle.api.internal.tasks.DefaultSourceSet.
I googled this error but failed to solve it.
Try setting the jvmArgs of the enclosing test task rather than trying to set them on groovy.
The error you are getting suggests that jvmArgs isn’t present on groovy.
Example:
...
test {
jvmArgs '-agentpath:/usr/lib/code_dependency_capturer.so' // add line
groovy {
srcDirs = ['src/test']
...
}
...
}
This is just a guess as I don’t have a gradle setup handy on which to confirm but worth a try as jvmArgs is documented as a property for test:
https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:jvmArgs
List<String> jvmArgs
The extra arguments to use to launch the JVM for the process. Does not include system properties and the minimum/maximum heap size.
Since jvmArgs is a list of String you can pass it multiple arguments, refer to:
http://docs.groovy-lang.org/next/html/documentation/working-with-collections.html#_list_literals
Example:
jvmArgs ["-Xarg1", "-Xarg2"]
For "-Dprop=value" system properties use the systemProperties of the test task instead:
https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:systemProperties
Is it possible to perform some high-level operation at Unit testing end?
Like grabbing all test results and logs and sending them by email to developer?
Would like solution either in JUnit or with Gradle.
A pure JUnit approach could be to implement your own RunListener to be able to grab the information of each of your tests.
To invoke your listener, you need to run your tests through JUnitCore.
public void main(String... args) {
JUnitCore core= new JUnitCore();
core.addListener(new MyRunListener());
core.run(MyTestClass.class);
}
A pure Gradle approach could be to implement your own TestListener to be able to grab the information of each of your tests and send the final report.
In build.gradle you will need to add gradle.addListener(new my.package.MyTestListener()). Your listener must be in the build source directory also known as buildSrc.
test.finalizedBy(someHighLevelOperationTask)
Example: Here is a task that you can call gradle emailTestResults that would depend on 'zipTestResults' which depends on test. I have not done the email part of it, there's a question that covers that: Sending email using Gradle
So this just spits out a println in place of the email task:
apply plugin: 'java'
task zipTestResults(dependsOn: 'test', type: Zip){
from 'build/reports/tests'
baseName 'TestReport'
destinationDir file('build/reports')
}
task emailTestResults(dependsOn: 'zipTestResults') << {
println 'Emailing...' + file('build/reports/TestReport.zip')
}
repositories{
jcenter()
}
dependencies{
testCompile 'junit:junit:4.12'
}
But that won't work if the test fails, so, there maybe a better way, but you can add this:
test.ignoreFailures = true
Then if you then always want to email the test results add:
test.finalizedBy(emailTestResults)
Then a gradle test would finish by emailing the results.
I'm trying to exclude a 'quarantine' folder that I set up for Selenium tests that need to be updated and I do not wish to have run. I know that one solution is to set up and assign test groups for the tests in these classes but given the sheer size and volume of tests that will be in here, I'd rather do it using an Ant-style filter.
Here is a snippet of my build.gradle file:
apply plugin: 'java'
apply plugin: 'idea'
apply plugin: 'eclipse'
repositories {
mavenCentral()
}
dependencies {
compile "org.seleniumhq.selenium:selenium-java:2.35.0"
compile "org.testng:testng:5.14.10"
testCompile('org.uncommons:reportng:1.1.2') {
exclude group: 'org.testng'
}
testCompile "junit:junit:4.8.2"
compile "com.jayway.restassured:rest-assured:1.8.1"
}
//initialize thread count variable for parallel testing and default to 1
def threadCount = System.getProperty("MAXTHREADS", "1")
tasks.withType(Test) {
maxParallelForks = 1
forkEvery = 1000
ignoreFailures = false
// Pass all system properties to the tests
systemProperties = System.getProperties()
// Makes the standard streams (err and out) visible at console when running tests
testLogging.showStandardStreams = true
exclude '**/tasks/'
exclude '**/disabled/'
classpath += configurations.testCompile
}
task firefox(type: Test) {
maxParallelForks = Integer.valueOf(threadCount) //default is 1 if not specified
testLogging.events "started"
testLogging {
events "started", "passed", "skipped", "failed", "standardOut", "standardError"
exceptionFormat "full" // default is "short"
}
useTestNG() {
excludeGroups 'chrome'
useDefaultListeners = false
listeners << 'org.uncommons.reportng.HTMLReporter'
listeners << 'org.uncommons.reportng.JUnitXMLReporter'
listeners << 'com.xmatters.testng.Listener'
}
testResultsDir = file("${buildDir}/test-results/firefox")
testReportDir = file("${reporting.baseDir}/firefox")
systemProperties.BROWSER = System.getProperty('BROWSER', 'firefox')
exclude '**/selenium/'
exclude '**/setupscripts/'
}
task chrome(type: Test) {
maxParallelForks = Integer.valueOf(threadCount) //default is 1 if not specified
testLogging.events "started"
useTestNG() {
useDefaultListeners = false;
listeners << 'org.uncommons.reportng.HTMLReporter'
listeners << 'org.uncommons.reportng.JUnitXMLReporter'
listeners << 'com.xmatters.testng.Listener'
}
testResultsDir = file("${buildDir}/test-results/chrome")
testReportDir = file("${reporting.baseDir}/chrome")
systemProperties.BROWSER = System.getProperty('BROWSER', 'chrome')
exclude '**/selenium/'
exclude '**/setupscripts/'
}
On line 34 you can see exclude '**/disabled/' that I added. This folder is a couple levels up from the root folder. The preceding like with exclude '**/tasks/' was already in the build file and seems to work fine with a similar directory structure.
When I run the build, tests in the /disabled/ folder are still getting run. Is there something I'm doing wrong here? I'm assuming that with that syntax, a directory named 'exclude' a couple levels up would be ignored by scanForTestClasses which is true by default. Any idea what is up here?
One other thing I've noticed in Gradle test report is that the package name listed in the report is default-packagefor the excluded tests that are not 'excluding' whereas the other tests that are meant to be run are listing the correct package names. The package names in the Java files match their folder structure correctly so I'm not sure why this is being reported this way. I've checked for duplicates, typos, etc, and am not getting anywhere.
If anyone could shed some light on this that would be great as having these incomplete / broken test classes running is causing failures that should be ignored until these tests are updated.
These test are being run using the Gradle wrapper generated bash script on our test CI (Jenkins) box running on Linux.
Looks like the exclude pattern is applied to the relative path of the files (i.e. relative to your root folder), which explains why it works for folders under your root folder.
Using an excludeSpec (see Gradle Test task DSL) should work fine:
exclude { it.file.canonicalPath.contains('/disabled/')}
Of course, pay attention to / vs \ according to your OS.