I've got a Jenkins CI server that is set up with a Selenium test project running with maven-surefire. I need the project to be a parameterized build, so that I can trigger the build via URL with a Dtest as a parameter (and only run the tests I specify in the url). This works great.
Unfortunately, I've been unable to figure out how to run ALL of the tests, while in this parameterized configuration. Since it is in parameterized build mode, I must ALWAYS specify the -Dtest parameter.
Based on the Surefire documentation, it seems like I should be able to wildcard the test names, and everything will be run:
-Dtest=* or -Dtest=Test*
The odd result of running these parameters is a print statement (that I created) from all 6 of the tests (denoting that they were all started):
"Test <test_name> started, click here to see the SauceLabs video"
And then the standard test result (below) for only 4/6 tests
Running <test_class_path>
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.048 sec
Followed by the summary:
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
If it matters, the tests are being run in parallel using surefire, and one other odd thing is that while printing out the individual test results, after the 4th one, the 5th result starts printing, but never shows a result, and includes a $1 at the end:
Running <test_class_path>$1
Please let me know if I can clarify anything or answer any questions.
Thanks in advance for any help!
I think that it's a regular expession:
mvn -Dtest=.*
works for me.
Related
I follow the evosuite maven documentation to generate unit test cases for my project. Below is the command I've used:
mvn -DmemoryInMB=2000 -Dcores=2 evosuite:generate evosuite:export test
The tool takes around 1 hour to generate test cases for my project (with 9700 lines of Java code). However when it proceed to mvn test phase, all test cases failed with the same message:
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec <<< FAILURE! - in com.xxx.yyy.util.Messages_ESTest
com.xxx.yyy.util.Messages_ESTest Time elapsed: 0 sec <<< ERROR!
java.lang.IllegalStateException: Trying to set up the sandbox while executing a test case
at <evosuite>.<evosuite>(<evosuite>)
at <evosuite>.<evosuite>(<evosuite>)
at <evosuite>.<evosuite>(<evosuite>)
While running the generated test case in Intellij IDEA most of them can pass the test:
Any one got any idea?
this is clearly a bug, which would be best to report on
https://github.com/EvoSuite/evosuite/issues
furthermore, currently in 1.0.3 the tests generated by EvoSuite do not work well with "mvn test". The fix is already in the SNAPSHOT though, and will be part of next release. However, the "Trying to set up the sandbox" might be a new bug as I haven't seen it before... :(
My project: testng-surefire-maven.
In one of the modules I runt mvn clean install.
When all tests are green, I have a result:
Tests run: 277, Failures: 0, Errors: 0, Skipped: 0
Then I in turn make an intentional mistake in one of 3 tests that I am refactoring right now. And as a result I have 3 totally different outputs:
Test1>AbstractTestNGSpringContextTests.springTestContextPrepareTestInstance:149 » BeanCreation
Tests run: 344, Failures: 1, Errors: 0, Skipped: 100
Test2>AbstractTestNGSpringContextTests.springTestContextPrepareTestInstance:149 » BeanCreation
Tests run: 282, Failures: 1, Errors: 0, Skipped: 8
Test3>AbstractTestNGSpringContextTests.springTestContextPrepareTestInstance:149 » BeanCreation
Tests run: 416, Failures: 1, Errors: 0, Skipped: 205
How is that possible???
All I've done is a one-line change in one of the test classes in turn. I didn't touch testng.xml nor pom.xml.
Additionally, if I make a mistake in all 3 of them simultaneously, only one will pop up. I didn't set a custom skipAfterFailureCount in surefire nor any other testng property. Why doesn't it run though all of them and show me the list of all failing tests at once? All tests are in the same package.
Ok, I don't know what influences test count, but I can answer a second part of my question.
Additionally, if I make a mistake in all 3 of them simultaneously, only one will pop up. I didn't set a custom skipAfterFailureCount in surefire nor any other testng property. Why doesn't it run though all of them and show me the list of all failing tests at once? All tests are in the same package.
It happens because testng has a property - configfailurepolicy: Whether TestNG should continue to execute the remaining tests in the suite or skip them if an #Before* method fails. Default behavior is skip.
And that's what happened in my case. I had problems on test init stage, not in the test method itself.
I'm running Cucumber tests (with Maven, in Jenkins) and outputting failed scenario's to target/rerun.txt, as per these instructions: https://github.com/cucumber/cucumber-jvm/issues/890
It creates the rerun.txt and it inputs failed scenario's, but enters them as: features/name.feature:2. The rerun then tries to run them as classpath/name.feature, and fails to see they are in src/test/resources/features.
If I put the features files in the classpath instead, I get an inconsistent filters error (probably because I'm using tags in the first call and the text in the second call).
How to get Cucumber to either output the correct path for features, or to get it to find the features based on the rerun.txt?
The error:
Running com.xxx.cucumber._RunCukesTest
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 sec <<< FAILURE!
initializationError(com.xxx.cucumber._RunCukesTest) Time elapsed: 0 sec <<< ERROR!
java.lang.IllegalArgumentException: Neither found on file system or on classpath: Not a file or directory: classpath\SubFolder\Test1.feature,
No resource found for: classpath:SubFolder/Test1.feature
Additional info:
All initial calls to tests are done via tags.
All features are in subfolders in classpath/src/test/resources/features
This test has to run in Jenkins, automatically, every day.
The first run uses this:
#RunWith(Cucumber.class)
#CucumberOptions(
strict = false,
features={"src/test/resources/features/"},
snippets=SnippetType.CAMELCASE,
plugin = { "pretty", "json:target/cucumber-reports/test-report.json", "html:target/cucumber-reports",
"rerun:target/rerun.txt"})
public class RunCukesTest {
}
The second run of failed scenario's uses this:
#RunWith(Cucumber.class)
#CucumberOptions(
strict = false,
features={"#target/rerun.txt"},
snippets=SnippetType.CAMELCASE,
plugin = {"pretty", "json:target/cucumber-reports/test-report.json", "html:target/cucumber-reports"})
public class _RunCukesTest {
}
Change the feature file location of first test runner to features = "."
This will make 'rerun.txt' to be updated with correct path to the feature file.
I'm trying to run FEST Swing tests on Jenkins as a service on Windows 7.
I have already allowed the service to interect with the desktop with the necessary permission to do so. It was no use, though.
Most of what I found on the internet either shows how to do it on Linux (through the CLI shell) or how to run it on the prompt, which is not what I'm looking for.
I have also found some vague discussions on how to run the tests letting the desktop available (which was by running it on a VM). But that's not what I need either.
I need to run the GUI tests just as they are run on Eclipse, with the graphical appearance on the desktop. When I try to do it, the following happens:
execguitest:
[echo] ----------------------------------------------
[echo] project-name: Executing project's GUI tests
[echo] ----------------------------------------------
[junit] Running dev.AllGUITests
[junit] null
[junit] null
[junit] null
[junit] Tests run:33, Failures: 3, Errors: 30, Skipped: 0, Time elapsed: 112,375 sec
[junit Test dev.AllGUITests FAILED
Does anyone have a clue on how can it be done?
P.S.: Sorry for any english mistakes. English is not my mother language.
The Documentation of FEST shows you how to configure Hudson (almost the same as Jenkins) unter Windows and how to run the tests on virtual Xvfb devices (Linux) for execution without real display devices.
I did it with the Jenkins-XVNC-Plugin.
Look at this.
I'm applying BDD methodology using Cucumber, which is GREAT!
The problem is that my test suite getting bigger and bigger and now I get the following exception which fails my test from the wrong reason...
I'm using all sort or Cucumber features, such as: Background, Scenario Outline and simple scenarios.
I run the tests like this:
#RunWith(Cucumber.class)
#Cucumber.Options(features={"...../controller1"})
public class RunCukes1Test {
}
I split my feature files to different directories (controller1, controller2...) and runners (RunCukes1Test, RunCukes2Test...), but this didn't help.
When I run each test itself everything is ok, but when I run it all using mave lifecycle test, it fails. Does anyone know of any best practices in Java Cucumber to avoid problems of such.
Tests run: 5896, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.082 sec
Running com.kenshoo.urlbuilder.appservice.controller.RunCukes4Test
Tests run: 11838, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 80.833 sec
Exception in thread "Thread-73" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515)
at java.lang.StringBuffer.append(StringBuffer.java:306)
at java.io.BufferedReader.readLine(BufferedReader.java:345)
at java.io.BufferedReader.readLine(BufferedReader.java:362)
at org.codehaus.plexus.util.cli.StreamPumper.run(StreamPumper.java:129)
Exception in thread "ThreadedStreamConsumer" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3209)
at java.lang.String.<init>(String.java:215)
at java.lang.StringBuffer.toString(StringBuffer.java:585)
at org.apache.maven.surefire.report.PrettyPrintXMLWriter.escapeXml(PrettyPrintXMLWriter.java:167)
at org.apache.maven.surefire.report.PrettyPrintXMLWriter.addAttribute(PrettyPrintXMLWriter.java:178)
at org.apache.maven.surefire.shade.org.codehaus.plexus.util.xml.Xpp3DomWriter.write(Xpp3DomWriter.java:50)
at org.apache.maven.surefire.shade.org.codehaus.plexus.util.xml.Xpp3DomWriter.write(Xpp3DomWriter.java:55)
at org.apache.maven.surefire.shade.org.codehaus.plexus.util.xml.Xpp3DomWriter.write(Xpp3DomWriter.java:39)
at org.apache.maven.surefire.report.XMLReporter.testSetCompleted(XMLReporter.java:128)
at org.apache.maven.surefire.report.MulticastingReporter.testSetCompleted(MulticastingReporter.java:51)
at org.apache.maven.surefire.report.TestSetRunListener.testSetCompleted(TestSetRunListener.java:115)
at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:97)
at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:67)
at java.lang.Thread.run(Thread.java:662)
Results :
Tests run: 11790, Failures: 0, Errors: 0, Skipped: 0
I got an answer to another java-heap-space exception I had, after the cucumber tests running.
You can see it here - related problem
My theory is that the -XX:MaxPermSize is a factor for during Cucumber running, as Cucumber generates tests code and PermSize is related to amount of code as described what is permsize in java
The -Xmx is a factor for the post Cucumber running, while parsing the tests results.
So the solution is to find the balance between them both and the actual available memory.