I'm working with a lot of legacy code. There was a JUnit-TestSuite to begin with. When running all tests with gradle, they failed. When running the test in IntelliJ, they worked. We configured gradle to use the test suite.
Now someone reported tests working locally without gradle, but not with gradle. It's time we fix this mess.
Is there a smart way to figure out which test leaves some configuration behind or which tests relies on the other tests?
The most likely cause of this "bleed" from one test into another is mutable static values. By default, all tests are run by the same JVM so a static variable which is "mutated" by one test will be "dirty" in another test.
Mutable statics are evil! I'm working on a codebase currently with mutable statics everywhere and it's a mess. If possible you should refactor to use dependency injection and store mutable state in instances and not statics.
The best workaround is to find the tests which "dirty" the static mutable variables and do
#After
public void cleanup() {
SomeStatic.reset();
}
If you can't find the "dirty" test which is causing the issue, you might be forced to do the following in the "failing" test. This is not preferred, and a little hacky
#Before
public void cleanBefore() {
SomeStatic.reset();
}
But this has a slight code "smell". Better to find the offending test which "dirties" the mutable static
The "nuclear" option is to run each test in its own jvm. This is a total hack and should be avoided at all costs. It will drastically increase the time it takes to run your tests
test {
forkEvery = 1
}
See Test.forkEvery
I recently diagnosed a similar issue in a Gradle Java project, where a test was working when run individually, but not when run as part of the Gradle build.
In order to track down the offending test that was breaking the subsequent test, I first configured Gradle to print out the tests that were run using beforeTest, so I could tell what all was running prior to my test:
test {
beforeTest { TestDescriptor descriptor ->
logger.lifecycle("$descriptor.className#$descriptor.name")
}
}
This printed the tests in the order in which they ran:
$./gradlew test
> Task :test
com.example.Test1#firstTest()
com.example.Test1#secondTest()
com.example.Test2#firstTest()
com.example.Test2#secondTest()
com.example.Test3#firstTest()
com.example.Test3#secondTest()
com.example.BrokenTest#brokenTest()
BrokenTest > brokenTest() FAILED
java.lang.AssertionError at BrokenTest.java:34
com.example.Test4#firstTest()
com.example.Test4#secondTest()
Now that I knew which test classes were running before the broken test, I knew that one (or more) of those tests were causing the test to break.
Next, I added test filtering in my Gradle build to only run those tests that ran before the broken test, plus the broken test itself:
test {
filter {
includeTestsMatching 'com.example.Test1'
includeTestsMatching 'com.example.Test2'
includeTestsMatching 'com.example.Test3'
includeTestsMatching 'com.example.BrokenTest'
}
}
After doing a sanity check Gradle build to confirm that the test was still broken, I commented out groups of tests and reran the build until I was able to narrow it down to the test that caused the build to break:
test {
filter {
// includeTestsMatching 'com.example.Test1'
includeTestsMatching 'com.example.Test2'
// includeTestsMatching 'com.example.Test3'
includeTestsMatching 'com.example.BrokenTest'
}
}
In this example, the build broke when Test2 was run before BrokenTest, but not when Test1 or Test3 were. Armed with this information, I could dive deeper into what specifically about Test2 was affecting the system in such a way as to break the other test when it was run after it.
Related
When running a specific junit test in the Apache Jackrabbit project, Eclipse goes through the motions of running the test, but the test is never run, debugger breakpoints are never fired, and the results are that zero out of eight tests were run, and no errors were reported.
https://github.com/apache/jackrabbit/blob/trunk/jackrabbit-spi2dav/src/test/java/org/apache/jackrabbit/spi2dav/ConnectionTest.java
The confusion is that the tests, after the run, don't indicate whether they succeeded, failed, or were explicitly skipped, and I cannot see anything in the source to indicate they should be skipped.
What is Eclipse trying to tell me?
This behaviour was triggered by the unit test overriding Junit's run method as follows:
https://github.com/apache/jackrabbit/blob/ed3124e5fe223dada33ce6ddf53bc666063c3f2f/jackrabbit-jcr-server/src/test/java/org/apache/jackrabbit/webdav/server/WebDAVTestBase.java#L234
public void run(TestResult testResult) {
if (Boolean.getBoolean("jackrabbit.test.integration")) {
super.run(testResult);
}
}
This was causing the test to not run at all.
In this case, Eclipse was picking up the test methods that should have been run and listed them, but with the tests not having been run, the Eclipse UI remains in a "vanilla" state.
This unit test obviously needs to be fixed to follow maven's best practise of splitting unit and integration tests into surefire and failsafe respectively, and not relying on a private mechanism.
I have some junit test cases within a test class. It runs fine when you run the file as a whole from IntelliJ.
But when I run the maven build, it fails.
TestCase Code Snippet {
doReturn(DummyObject with values).when(springRepo).call()
executeTest()
}
We have a TestUtil which has some static methods to create DummyObject at runtime based on different input keys. What I see that Junit always runs testcases sequentialy and there are no parallel test cases run unless specified.
Now, Dummy object is not null, but few of the internal data is null only from Maven run. What could be the issue?
I have a project that itself is a set of test cases for an application. The test cases are tested itself using a mockup of the application to ensure the tests itself is correct. This is important to ensure the test cases justify some sort of specifications that are hard to follow and easy to mess up.
Now I want the final tests (those who are tested before) being in src in Maven. As expected mvn test just executes the test cases of the test cases not the test cases themself.
So basically how do I execute test cases that reside inside the 'src' folder using a maven goal?
Try this :
<testSourceDirectory>src</testSourceDirectory>
So I edited the name of a JUnit test and now it wont work. Instead I get Unrooted Tests: initializationError.
This is a simple test. Infact it is a test for JUnit tests as I am just starting to use it.
#Test
public void testRun()
String s = null;
assertNull(s);
}
and all i did was change it to testRun2(). Also when I run the file not the individual test, it still runs the old testRun(), not testRun2().
My project has Maven not sure if that is a factor. And I have updated the project
So it turned out that I needed to rebuild using Maven to update the classes. Now it works fine and I can add/modify test cases.
In my case, i changed the method name and it didn't update it automatically, so the above solution of Project> Clean worked well for me.
Another way this error would occur is forgetting the Test annotation. Encountered when right click method name in Eclipse and Run As -> Junit Test.
I am looking to obfuscate our Java web app code within our existing Ant build script, but am running into problems around unit testing. I am obfuscating the code right after it has been compiled, before it is jar-ed and before the unit tests are ran.
However, if I obfuscate my production code and not my test code, all my tests fail because they are trying to call methods that no longer exist because they have been renamed by the obfuscator. I can mark certain methods to not obfuscate so they can be used by external systems such as our test suite, but since we are shooting for high unit test coverage we will need to mark all of our methods as un-obfuscatable.
If I obfuscate the test classes as well, I run into two problems:
1: The production classes and the test classes get merged into the same output directory and I am unable to exclude the test classes from the production .jar files
2: I cannot run my normal Ant batchtest call:
<batchtest todir="${basedir}/reports">
<fileset dir="${basedir}/components/common/build-zkm">
<include name="**/*Test.class"/>
</fileset>
</batchtest>
because the obfuscator has changed the names of the tests.
I could just run the obfuscator on the resulting .war/.ear files, but I want to have our unit tests run against the modified code to drive out any bugs caused by the obfuscator.
I am currently working with Zelix KlassMaster, but I am still in the evaluation phase so I would be open to other options if they would work better.
I use yguard (it is free, which is why I mention it).
You should be able to tell the obfuscator not to obfuscate certain things (looking here it seems you can).
Some as others have said, don't obfuscate the tests, but do obfuscate the rest.
However, I would suggest that you do the following:
compile
jar the un-obfuscated files (if desired)
test the un-obfuscated files
if they pass the tests then obfuscate jar the obfuscated files
test the obfuscated files
It will be slower, but if the tests fail in step 3 it'll be easier to fix (potentially) and if the tests fail at 5 then you know there is an issue with the obfuscation not your source code.
Can you tell it to run the obfuscator such that it effectively refactors the code including the references from the tests (i.e. when a production name changes, the test code changes its reference) but not to obfuscate the tests themselves (i.e. don't change the names of the test classes or their methods)? Given previous experience with obfuscators I'd expect that to work.
So for example, suppose we had unobfuscated source of:
public class ProductionCode
{
public void productionMethod() {}
}
public class ProductionCodeTest
{
public void testProductionMethod()
{
new ProductionCode().productionMethod();
}
}
You want to set the options of the obfuscator to make it effectively:
public class Xyzzy
{
public void ababa() {}
}
public class ProductionCodeTest
{
public void testProductionMethod()
{
new Xyzzy(). ababa();
}
}
That way your "run the tests" Ant tasks should be able to stay the same, because the API of the tests hasn't changed - merely the implementation of the methods.
The obfuscator should not change your public calls. It seems that yo should run the other tests before obfuscation because they check internal functionality that should not change after obfuscation.
So if that is the case, why not just run the tests that call public functionality? All you need to do is have a separate class with those calls and re-build it using the obfuscated code and then run that dll.