I have found that the following :
<junit haltonfailure="no" haltonerror="no" >
<!-- some tests guaranteed to call Assert.fail(); -->
</junit>
Will not prematurely halt (all tests will run).
HOWEVER ...... The following
<junit haltonfailure="no" haltonerror="yes" >
Will fail after the FIRST failed test. Note that in this case, I'm guaranteeing that the test is failing via Assert.fail();
This, I am assuming that either
1) Ant considers an ERROR to be a subclass of a FAILURE. or
2) There is a oddity in the way ant percieves failures - that is, failures are treated (erroneously) as errors when deciding to halt.
My Question : Why would ant would halt on a test failure (even when explicitly configured with haltonfailure="no"). Any thoughts could be very enlightening ! Thanks.
UPDATE
I now realize that this is clearly an RTFM question which is covered in the ant http://ant.apache.org/manual/Tasks/junit.html (see correct answer below).
Quote from junit ant task documentation:
haltonfailure: Stop the build process if a test fails (errors are considered failures as well).
So the answer is: ant junit considers failures as errors, hence the stop of the build
Related
I've recently started the upgrade from JUnit 4 to JUnit5 and the subsequent update to Ant 1.10.8 as well.
What I'm seeing when I run the tests is that it appears we're not forking the JVM like it did in previous versions. The results are indicating that we're reusing the JVM between tests which means we're hitting issue with static classes between runs.
<junitlauncher>
<classpath>
...
</classpath>
<testclasses>
<fork dir="${baseDirectory}">
....
</fork>
</testclasses>
</junitlauncher>
Is there something I'm missing here with the fork functionality on Ant?
Or is there another way around this that operates like the 'old ant & junit combo'?
*sigh
So for those interested parties - the answer is in the details:
The fork nested element can be used to run the tests in a newly forked
JVM. All tests that are part of this testclasses element will run
in one single instance of the newly forked JVM.
So JUnit5 doesn't fork for each test - it forks for all the items in the testclasses attribute. In order to have a resource to be looped through with a JVM for each test class - that is going to require some magic on the outside of that testclasses block.
I have test in java that run on Jenkins.
Is it no possible to debug that test and therefore I want to add dummy comment to the java and also to see values of variables
I want to see it in the console output of the Jenkins ?
How I can do it ?
System.out.println() did not work for me
If you use a java testing framework like JUnit, then any tests run will be logged by Jenkins, including any print statements that the tests execute.
Jenkins also has a plugin specifically for JUnit (found here), which will publish the results so you can easily access them in Jenkins, allowing you to see if tests fail, and then inspect anything printed out, as well as the stack trace, for any given test.
If your junit tests are being run by an ant task, then you can simply add a formatter to your task. For example:
<target name="unit-tests">
<junit>
...
<formatter type="plain" usefile="false" />
</junit>
</target>
Copied directly from this post
I need to cause my build to fail based only on the junit report. Is there a way to do this. I know how to fail the build using haltonfailure in junit tag, but in my build.xml I have access only to the junit report. I am using ant.
Try something like
<fail if="testFail" message="At least one unit test failed"/>
in your unit test target.
Refer to this: http://ant.apache.org/manual/Tasks/junit.html ,
try using failureproperty="failed.unit.test.property" inside junit ant task & check if its set & fail when your run completes with :
<fail if="failed.unit.test.property" message="Unit tests failed with more custom msg to your test class/suite"/>
The core of my question is that I am concerned that my Ant build file is missing something that will allow a test to finish and clean itself up. The details are below.
I have a suite of tests that always passes when I run it through Eclipse, but sometimes passes or fails when I run it using my Ant build. The tests use openCL via JOCL so I have limited memory on the GPU and it has to be managed correctly. I get this in my output sometimes when I run my Ant build,
[junit] Caused an ERROR
[junit] CL_MEM_OBJECT_ALLOCATION_FAILURE
[junit] org.jocl.CLException: CL_MEM_OBJECT_ALLOCATION_FAILURE
The problem can not be in the test itself. I think it is that my most memory hungry test is invoked at the end of the suite. When this last test is invoked, somehow the GPU is left in a bad state from my previous tests. This doesn't happen when I run the tests through Eclipse. It has never failed in my Ant build when I make the memory hungry test the first test in the suite. Is this a familiar case? Why does running the tests through Eclipse always work? Is there anything I can try?
Here is the testing target in my Ant build:
<target name="test" if="testing.enabled">
<mkdir dir="${test.bin.dir}" />
<javac srcdir="test" destdir="${test.bin.dir}" debug="true" classpathref="testclasspath" source="1.6"/>
<junit haltonerror="true" haltonfailure="true">
<classpath refid="testclasspath"/>
<formatter type="plain" usefile="false" />
<batchtest>
<fileset dir="test">
<include name="*Test.java"/>
</fileset>
</batchtest>
</junit>
</target>
If you are really sure no left-over cleanup is missed in your code, you can create a JUnit test suite and run that from both eclipse and ant. By creating a test suite you make yourself independent of the sequence of tests that eclipse (order within the project?) and ant (order within the filesystem?) use and determine the order yourself in both cases.
If you are not really really sure your code is issue-free you could make a test suite which starts of by calling Collections.shuffle() on the list of test classes to introduce unknown test execution order in both eclipse see if your tests still never fail.
The problem could be that you do not free memory in your test cases. JUnit instanciates all test classes when it is started and then runs them, as far as I know. If you have fields that reference objects in your test classes all fields will stay assigned through the whole testrun when you don't assign null to them in a tearDown() method. For example:
class Test extends TestCase {
private Data data;
public void setUp() {
data = new Data();
}
public void tearDown() {
data = null; // required to allow garbage collection of the Data object
}
}
Maybe Eclipse unreferences the Test instances after they are executed, so that the fields can be garbage collected. But using the standard JUnit TestRunner you will end up with a lot of objects that are not used anymore but that are still referenced and eat up all your memory.
If the tests are passing in Eclipse and failing elsewhere, then you're suffering from one of the many kinds of developer syndrome: "...but It works when I run it here...!"
You have managed to configure Eclipse to let you work with your code, the functionality is in, yet your code is, this far, not deployable, which means it's not done.
Shelve Eclipse for a while (stop blaming it), and drop to the command line (or use a different IDE) until things work. Try the code on a different computer, even!
Then go back to Eclipse, and repeat the above cycle until you're certain that any dependencies on Eclipse or your hard disk/setup have been removed. In the end, your code must be able to run on who-knows-which server.
Have you tried having a clean Eclipse installation (on a different computer) take a shot at a source-only snapshot of the code? It would be a good configurations management test that I'm quite sure your code won't pass as it stands.
Seriously try to get Eclipse doing its magic on a clean (virtual) machine. It won't work on a first run, but you'll learn what you did to make it work under your setup.
Let me google that for you:
Ant looks for an environment variable called ANT_OPTS which is use to set Java parameters. > Just set the environment variable and off you go. So I added the following to increase the > heap size:
export ANT_OPTS=-Xmx256m
When running withing Eclipse, the JVM (and therefore Ant) most likely already has more memory than default.
I'm trying to define a task that emits (using echo) a message when a target completes execution, regardless of whether that target was successful or not. Specifically, the target executes a task to run some unit tests, and I want to emit a message indicating where the results are available:
<target name="mytarget">
<testng outputDir="${results}" ...>
...
</testng>
<echo>Tests complete. Results available in ${results}</echo>
</target>
Unfortunately, if the tests fail, the task fails and execution aborts. So the message is only output if the tests pass - the opposite of what I want. I know I can put the task before the task, but this will make it easier for users to miss this message. Is what I'm trying to do possible?
Update: It turns out I'm dumb. I had haltOnFailure="true" in my <testng> task, which explains the behaviour I was seeing. Now the issue is that setting this to false causes the overall ant build to succeed even if tests fail, which is not what I want. The answer below using the task looks like it might be what I want..
You can use a try-catch block like so:
<target name="myTarget">
<trycatch property="foo" reference="bar">
<try>
<testing outputdir="${results}" ...>
...
</testing>
</try>
<catch>
<echo>Test failed</echo>
</catch>
<finally>
<echo>Tests complete. Results available in ${results}</echo>
</finally>
</trycatch>
</target>
According to the Ant docs, there are two properties that control whether the build process is stopped or not if the testng task fails:
haltonfailure - Stop the build process
if a failure has occurred during the
test run. Defaults to false.
haltonskipped - Stop the build process
if there is at least on skipped test.
Default to false.
I can't tell from the snippet if you're setting this property or not. May be worth trying to explicitly set haltonfailure to false if it's currently set to true.
Also, assuming you're using the <exec> functionality in Ant, there are similar properties to control what happens if the executed command fails:
failonerror - Stop the buildprocess if the command exits with a return code
signaling failure. Defaults to false.
failifexecutionfails - Stop the build if we can't start the program.
Defaults to true.
Can't tell based on the partial code snippet in your post, but my guess is that the most likely culprit is failonerror or haltonfailure being set to true.
The solution to your problem is to use the failureProperty in conjunction with the haltOnFailure property of the testng task like this:
<target name="mytarget">
<testng outputDir="${results}" failureProperty="tests.failed" haltOnFailure="false" ...>
...
</testng>
<echo>Tests complete. Results available in ${results}</echo>
</target>
Then, elsewhere when you want the build to fail you add ant code like this:
<target name="doSomethingIfTestsWereSuccessful" unless="tests.failed">
...
</target>
<target name="doSomethingIfTestsFailed" if="tests.failed">
...
<fail message="Tests Failed" />
</target>
You can then call doSomethingIfTestsFailed where you want your ant build to fail.
Although you are showing a fake task called "testng" in your example I presume you are using the junit target.
In this case, it is strange you are seeing these results because the junit target by default does NOT abort execution on a test failure.
There is a way to actually tell ant to stop the build on a junit failure or error by using the halt attributes, eg. haltonfailure:
<target name="junit" depends="junitcompile">
<junit printsummary="withOutAndErr" fork="yes" haltonfailure="yes">
However, both haltonfailure and haltonerror are by default set to off. I suppose you could check your build file to see if either of these flags have been set. They can even be set globally, so one thing you could try is to explicitly set it to "no" on your task to make sure it is overridden in case it is set in the global scope.
http://ant.apache.org/manual/Tasks/junit.html
Can you fork the testng task ? If yes, then, you might want to use that feature so that the testng task will run on a different JVM.