We Keep Coding

JUnit tests pass but PIT says the suite isn't green

While trying to run a PIT mutation test I get the following error:
mutationCoverage failed: All tests did not pass without mutation when calculating line coverage. Mutation testing requires a green suite.
The tests run just fine when I do a normal test build but while running the mutation tests phase they supposedly fail but no details are provided as to why. I have gone through the reasons listed on the PIT Testing FAQ but I still have no clue what could be wrong.
I tried:
adding the -Dthreads=1 option to rule of any multi threading issue
could not find any system properties unique the couple tests that are failing
the tests are not ignored under normal runs
What are some other things I should try? Or other ways to debug what could be going on here?

The common causes of tests failing at the coverage stage are
PIT picking up tests not included/are excluded in the normal test config
Tests rely on an environment variable or other property set in the test config, but not set in the pitest config
Tests have a hidden order dependency that is not revealed during the normal test run
PIT doesn't like something in your tech stack - possibly a JUnit test runner
It sounds like you've eliminated 1 & 2. so that leaves 3 and 4.
Test order dependencies can be hard to spot. If the answer is yes to any of these you may have one.
Does your codebase include mutable static state? (e.g in singletons)
Do your tests hit a database (in memory or otherwise) where it is possible for state to persist between tests?
Do your tests modify files on disk?
There are probably also many other causes not listed above.
If you are confident that order dependencies are impossible in your code base, that leaves a problem with these particular tests.
It's hard to guess what this might be without some code. Can you post a simplified version of the test that still fails?

Code coverage for Fitnesse

I have just inherited a old java codebase (around 10 - 15 years old). It does not have any automated testing coverage, or at least the contemporary world knows about it. I am planning to write some Fitnesse scripts around it, to begin with.
I know about Concordion etc. And I have my reason to pick Fitnesse. I will keep away from that since that is not the topic of this question.
My question is, I don't know of a quick way to measure the code coverage done by the Fitnesse tests as I write them. I know that jacoco (or similar libraries) should be able to report them but I just can't figure out exactly how.
So, if anyone of you have worked on Fitnesse test scripts and have managed to have Jenkins report on the coverage achieved by the scripts, please help.
Thanks.

I have not done this myself, because I tend to use FitNesse to test deployed applications (and I cannot measure code coverage in the 'real installation'). But I believe this should be fairly straightforward if you run your FitNesse tests as part of a Jenkins (or any other build server's) jUnit run, which measures code coverage.
To have your FitNesse tests be executed as part of a jUnit run: create a Java class annotated with #RunWith(FitNesseRunner.class) and give it a #Suite("MyPageOrSuite.That.IWantToRun") annotation to indicate which tests to run. This will execute the specified page(s) in the same Java process that the jUnit process is using, so if that is instrumented in some way to determine code coverage the coverage of your FitNesse tests will be included in the report.
Sample jUnit test class, running FitNesse.SuiteAcceptanceTests.SuiteSlimTests.TestScriptTable:
#RunWith(FitNesseRunner.class)
#FitNesseRunner.Suite("FitNesse.SuiteAcceptanceTests.SuiteSlimTests.TestScriptTable")
#FitNesseRunner.FitnesseDir(".")
#FitNesseRunner.OutputDir("../target/fitnesse-results")
public class FitNesseRunnerTest {
}

Narrowing down a bad test by running junit tests in various combinations using Maven or other tool

I have 1000 junit tests and one "bad" test that is modifying a shared resource causing a subsequent test to fail. It passes if run alone. I'm looking for a Maven plugin or Java application or tool that will take as input a test class name. It will then run the 1000 tests in various combinations until it finds the "bad" test. Assume it is one "bad" test.

As you may noticed, writing dependant tests IS the problem ! Every solutions would be a workaround, maybe with side effectfs, complexity the first one.
Assuming you REALLY can't change that, you may have different strategies to solve your issue, but they are not necessary related to maven :
Ordering
Order your test to run the "bad one" at the end ... be carefull that you me affected again if you have a second bad test !
User TestNG instead of JUnit and #Groups and #AfterGroups annotations to split your tests and run them as you want
Use #AfterClass and BeforeClass with a test suite: Cleanup after all junit tests
Manually describe a Test Suite (not sure if you can achieve what you want)
Provide good data
Use setUp and tearDown methods to prepare and clean data in order to always have a stable environment, on each tests classes
Rollback the test that modify you resource (pretty the same thing indeed)
Step back thoughts :
Just a piece of minds :
If you cannot run tests independently, they are not unit tests.

Retesting only recompiled tests using dependencies?

My testing framework has hundreds of tests. The average test takes a 30 seconds so right there that's 50 minutes.
If I change ONE file I should only have to re-test the dependencies.
The way I was thinking about doing this was to checkout rev0 from version control. Compile it. Then update to rev1, then look at the unit tests that needed to be recompiled after the task in ant kicks in and deletes the classes in the dependency graph.
In a trivial example I just did I found that I would only need to run 2 tests.
I imagine I can just do this with the hashes of the files. This way I can do cool things like tweak javadoc without triggering lots of retesting.
I could HACK something together but I don't think there's any clean way to do this in Junit/Ant.
Anyone have any ideas?

As said in a comment: if you have a unit test that takes 30 seconds, your tests are not good unit tests. They are probably not unit tests at all. You would be better off redesigning your tests.
That said, I have a large C++ software system with 25000 unit, integration and system tests. It uses make for building and cppunit for unit tests. Each module has its own suite of tests, which write a report file for each module. I have the make dependencies set up so only the modules that have changed rerun their tests.

Set JUnit timeout in eclipse

Question
When I run all our JUnit tests, using eclipse, can I set a default timeout?
Background
My manager insists on writing Unit tests that sometimes take up to 5 minutes to complete. When I try to run our entire test suite (only about 300 tests) it can take over 30 minutes. I want to put something in place that will stop any test that takes longer than 10 seconds.
I know an individual test can be annotated with:
#Test(timeout=10000)
But doing this would make his long tests always fail. I want them to work when he runs them on his box (if I have to make minor adjustments to the project before checking it in, that's acceptable. However, deleting the timeouts from 40 different test files is not practical).
I also know I can create an ant task to set a default timeout for all tests, along the lines of:
<junit timeout="10000">
...
</junit>
The problem with that we typically run our tests from inside eclipse with Right Click > Run As > JUnit Test.
Summary
So is there a relatively painless way to set a timeout for all tests, perhaps using a Run Configuration setting, or project setting, or JUnit preference, or environment variable, or something? I'd even settle for installing some other plugin that lets me right click on particular test folders and run all the tests in some other manner like through ant or something...

Possible solution:
Extend all your Test classes from another class: TestBase for example
Add to TestBase global timeout. This timeout will be applied to all extended classes:
public class TestBase {
#Rule
public Timeout globalTimeout = new Timeout(10000);
}

So maybe a combination of using Infinitest with the "Slow test warning" enabled together with the filtering feature would do the trick. You could identify tests that exceed your time-limit and add them to the filter list, this would only affect testing from inside Eclipse. Running the tests via a possible build script via CLI/CI etc would not be affected at all.
You can find more on setting this up here: http://improvingworks.com/products/infinitest/infinitest-user-guide/

If you want to configure the tests to run for a maximum of ten seconds you can try this:
#Test(timeout=10000)

My manager insists on writing Unit tests that sometimes take up to 5 minutes to complete
This almost certainly indicates that those tests are not in fact unit tests. Cut that Gordian knot: try refactoring your testsuite to provide equivalent test coverage without requiring a test-case that runs for that long.

Almost certainly your bosses tests are system tests pretending to be unit tests. If they are suppsoed to be unit tests and are just slow they should be refactored to use mocks so that they run quicker.
Anyway, a more pragmatic and diplomatic approach than confronting your boss over this might be to just try and run the faster ones yourself. I've seen a hack to do this in a project where slow tests had SytemTest in their names. Then there were two ant targets created in the build file. One that ran all tests and one that filtered out by class name the SytemTests . To implement this all you would have to do is rename some of the tests and write your ant target.

It sounds like test suites would help you out.
You can have two test suites; QuickTests and AllTests. Include QuickTests in the AllTests suite, as well as the tests that take a long time. Then all other tests would go into the quick tests suite.
From eclipse you can run an entire test suite at once. So you would run QuickTests and that way all the other slow tests will not run.
Or see this question on how to apply a timeout to a suite which will apply to nested suites and classes in the suite. Which can achieve similar to what you want when combined with my above suggestion.

I know this doesnt really answer your question but the simple answer is don't!
Setting timeouts conditionally is wrong beacsue then you would have Unit tests on your machine that you are always going to fail. The point of Unit tests is to be able to quickly see that you havnt broken anything. Having to check through the failed test list to make sure it's just the long running tests is ust going to make some bugs pass through the cracks.
As some of the commenters mentioned you should split out the tests into unit tests that run quickly and the slower runnning integration tests i.e. have a source folder called src/main/java for your code, src/test/java for unit tests and src/integration-test/java for the longer running tests.