In my project there are 2 types of tests: simple unit tests and acceptance tests with UI automation. All acceptance tests are extending a common base class:
import org.junit.Test;
public class MyAcceptanceTest extends AbstractAcceptanceTest {
#Test
public void someTest() {...}
}
Is it possible with gradle to filter tests so that only the acceptance tests are executed?
My main goal is that with gradle build only the normal unit tests are executed (all test classes except those that extend from AbstractAcceptanceTest).
The second goal is to have a separate task that only executes the acceptance tests (only test classes that extend AbstractAcceptanceTest). However, it would also be ok to execute all tests with this task (no filtering).
As far as I've found in the docs and by googling, filtering is only possible by file name patterns. But maybe there is some groovy magic that allows filtering like this?
EDIT
I've found a solution that works but feels a little hacky:
I've added a #Category annotation from JUnit to the base class (AbstractAcceptanceTest). The category needs an identifier class for which I've created a class com.example.AcceptanceTest. In my gradle.build file I'm using this configuration:
task acceptanceTests (type: Test) {
test {
useJUnit {
includeCategories 'com.example.AcceptanceTest'
}
}
}
test {
useJUnit {
excludeCategories 'com.example.AcceptanceTest'
}
}
This way the acceptance tests aren't executed in normal build but only with gradle acceptanceTests.
Related
I have a legacy test class using old Junit 3 way:
public class MyTestUtil extends TestCase {
//class has helper methods but no method starting with "test"
}
I have other test classes which extends this class:
public class MyTests extends MyTestUtil {
public void testSomething() {
}
}
I am trying to run this using gradle build file. And the build fails complaining with a warning:
junit.framework.AssertionFailedError: No tests found in myPackage.MyTestUtil
The build obviously runs fine when I exclude this class from test task:
test {
exclude '**/MyTestUtil.class'
}
But I don't know if excluding like this is the only solution.
Is there a way to do away with this warning?
From the JUnit FAQ:
Why do I get the warning "AssertionFailedError: No tests found in XXX" when I run my test?
Make sure you have more or more method annotated with #Test.
Either add at least one test or exclude the class like you described.
I have JUnit tests located in different test folders, when I'm running them one by one everything is green all tests are passed in particular folder, but when its done in scope (all at once), some tests are failing due to some data is changed during previous tests execution. So it's a way to run JUnit tests from scratch, I've tried
mvn "-Dtest=TestClass1,TestClass2" test
but some tests are failed. When its done like:
mvn "-Dtest=TestClass1" test
all passed. Or when:
`mvn "-Dtest=TestClass2" test
all passed.
As long as TestClass1 and TestClass2 share common state there might be no way to run them together e.g. it could be a static field somewhere in the JVM. You must refactor the tests so they are isolated and have no side effects e.g. use #Before and #After to clean up resources after the test.
You could play with Maven Surefire Plugin options to spawn a new JVM for each test but it would be very inefficient.
For this specific problem, you can create a TestSuite.
Create Two Test Suite Classes
Attach #RunWith(Suite.class) Annotation with the class.
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
#RunWith(Suite.class)
#Suite.SuiteClasses({
TestJunit1.class
})
public class JunitTestSuite1 {
}
#RunWith(Suite.class)
#Suite.SuiteClasses({
TestJunit2.class
})
public class JunitTestSuite2 {
}
public class TestRunner {
public static void main(String[] args) {
Result result1 = JUnitCore.runClasses(JunitTestSuite1.class);
Result result2 = JUnitCore.runClasses(JunitTestSuite2.class);
for (Failure failure : result1.getFailures()) {
System.out.println(failure.toString());
}
for (Failure failure : result2.getFailures()) {
System.out.println(failure.toString());
}
System.out.println(result1.wasSuccessful());
System.out.println(result2.wasSuccessful());
}
}
I have the classical structure for tests, I have a test suite of different suites like DatabaseTests, UnitTests etc. Sometimes those suites contains other suites like SlowDatabaseTests, FastDatabaseTests etc.
What I want is to randomize the running order of tests so I will make sure they are not dependent to each other. Randomization should be at every level, like suite should shuffle test class order, and test class should shuffle test method order.
If it is possible to do this in Eclipse that will be the best.
You do have a Sortable but I can't see how you would use it.
You could extend BlockJUnit4ClassRunner and have computeTestMethods() return a randomized copy of super.computeTestMethods(). Then use the #RunWith to set that as the runner to use.
e.g.
package com.stackoverflow.mlk;
import java.util.Collections;
import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runners.model.InitializationError;
public class RandomBlockJUnit4ClassRunner extends BlockJUnit4ClassRunner {
public RandomBlockJUnit4ClassRunner(Class<?> klass)
throws InitializationError {
super(klass);
}
protected java.util.List<org.junit.runners.model.FrameworkMethod> computeTestMethods() {
java.util.List<org.junit.runners.model.FrameworkMethod> methods = super.computeTestMethods();
Collections.shuffle(methods);
return methods;
}
}
Then
#RunWith(com.stackoverflow.mlk.RandomBlockJUnit4ClassRunner.class)
public class RandomOrder {
#Test
public void one() {
}
#Test
public void two() {
}
#Test
public void three() {
}
}
https://github.com/KentBeck/junit/pull/386 introduces some orders but not RANDOM. Probably you do not really want this; tests should run deterministically. If you need to verify that different permutations of tests still pass, either test all permutations; or, if this would be impractically slow, introduce a “random” seed for shuffling that is determined by an environment variable or the like, so that you can reproduce any failures. http://hg.netbeans.org/main/file/66d9fb12e98f/nbjunit/src/org/netbeans/junit/MethodOrder.java gives an example of doing this for JUnit 3.
In general what you need to do is to write your own test runner and in the test runner class aggregate the methods and randomly run each test (make sure you don't run a test twice).
Read more about the test framework and how to write your own test runner here:
http://www.ddj.com/architect/184415674
In JUnit 4.13, to run the tests within a test class in random order, write a small helper class:
import org.junit.runner.manipulation.Ordering;
import java.util.Random;
public class RandomOrder implements Ordering.Factory {
#Override
public Ordering create(Ordering.Context context) {
long seed = new Random().nextLong();
System.out.println("RandomOrder: seed = " + seed);
return Ordering.shuffledBy(new Random(seed));
}
}
Then, annotate your test class with:
#OrderWith(RandomOrder.class)
This way, the test methods of this one class are run in random order. Plus, if they unexpectedly fail, you know the random seed to repeat exactly this order.
I don't know though how to configure this for a whole project or a test suite.
I will make sure they are not dependent to
each other
You should make sure that this is the case without relying on random execution order. What makes you fear that dependencies may exist?
This issue is open on JUnit GitHub since 2 years, and point out 2 independent issues:
- Tests depending on the execution order;
- Non repeatable tests.
Consider adressing the issue at the root, rather than trying to use the framework to do the job afterwards. Use setUp and tearDown method to guarantee isolation, and test at the smallest level.
Here is a solution with Gradle and JUnit 5.8.0
Step 1 : Ensure that you have latest JUnit version dependency.
Step 2 : Define the required properties under build.gradle test section
test {
useJUnitPlatform()
systemProperties([
//Random in method level
'junit.jupiter.testmethod.order.default': 'org.junit.jupiter.api.MethodOrderer$Random',
// Random in class level
'junit.jupiter.testclass.order.default' : 'org.junit.jupiter.api.ClassOrderer$Random',
// Log configuration to see the seed
'java.util.logging.config.file' : file('src/test/resources/logging.properties')
])
//To print the JUnit logs in console
testLogging {
events "passed", "skipped", "failed", "standardOut", "standardError"
}
}
Step 3: Define logging.properties under src/test/resources
.level=CONFIG
java.util.logging.ConsoleHandler.level=CONFIG
org.junit.jupiter.api.ClassOrderer$Random.handlers=java.util.logging.ConsoleHandler
org.junit.jupiter.api.MethodOrderer$Random.handlers=java.util.logging.ConsoleHandler
Step 4 : Run test. gradlew clean test
You can see the seed used for the random test in the console
CONFIG: ClassOrderer.Random default seed: 65423695211256721
CONFIG: MethodOrderer.Random default seed: 6542369521653287
In case of flaky test, you can reproduce it by configuring the same seed where the JUnit tests were failing
systemProperties([
'junit.jupiter.execution.class.order.random.seed' : '65423695211256721'
'junit.jupiter.execution.order.random.seed' : '6542369521653287'
])
References : how-to-randomize-tests-in-junit , Random
I was looking for a verification test for custom implemented collections, and stumbled upon this: http://www.gamlor.info/wordpress/2012/09/google-guava-collection-test-suite/
I haven't used junit before (so I'm a total noob with junit). I added junit4 to my test project and... got stuck on how to actually run the test suite created by the Google Guava Collection Test Suite. I run annotated test from my test class just fine, but not the test suite from guava.
The junit docs say that suites are created by annotating a suite class with the Suite annotation, listing the cases they should include, but obviously I can't list a dynamically generated class that way. I would be happy to just create a simple test and run the entire suite as a single test, only... how do I get junit to run the suite instance?
If you have a factory method that returns a Test object (let's call it TestMaker.foo(), then you can write something like the following to take the Test and add it to your own TestSuite class:
import junit.framework.JUnit4TestAdapter;
import junit.framework.TestSuite;
import org.junit.runner.RunWith;
import org.junit.runners.AllTests;
#RunWith(AllTests.class)
public class TestRunner {
public static TestSuite suite() {
TestSuite ts = new TestSuite();
ts.addTest(TestMaker.foo());
return ts;
}
}
Now, when you run the TestRunner tests, it will also run the Test returned by TestMaker.foo() along with any other tests that might be defined directly in TestRunner.
For related discussion (for example, how to refer to a Test Class rather than pass a Test object), check out: How do I Dynamically create a Test Suite in JUnit 4?
I need to do some per-suite initialisation (starting a web-server). It is working fine except that when I run all tests in my project in eclipse my tests run twice. My test suite looks a bit like this:
#RunWith(Suite.class)
#Suite.SuiteClasses({
SubtestOne.class,
SubtestTwo.class
})
public class TestSuite
{
[...]
}
public class SubtestOne
{
#Test public void testOne() { [...] }
}
public class SubtestTwo
{
#Test public void testTwo() { [...] }
}
When I run all test in project in eclipse this causes the junit plugin to run the tests twice like this:
SubtestOne
SubtestTwo
TestSuite
SubtestOne
SubtestTwo
Is it possible to make "run all test in project" not run the sub-tests twice? I want my sub tests to be only ever run as part of the suite.
No, the test class will always be started directly and then through the "link" in the suite. This is as expected.
One workaround might to set in the run configuration to only run tests from the package which contains your suites. Open the run configuration and select Run all tests in the selected project, package or source folder then click Search... and select the package.
I realize that this has been asked over 5 years ago, but as quite a few folks up-voted the question I thought I'd still chime in with a solution. Skip right to the end if you just want the solution; read the whole text if you also want to understand it ;-)
First of all, it is indeed possible to ensure that a particular JUnit test class gets only run inside a test suite. Also, it is irrelevant whether you want to run that test suite inside Eclipse (as asked here) or any other tool or environment; this is really a pure JUnit issue for the most part.
Before I sketch out the solution, it might be a good idea to revisit what the exact problem is here. All JUnit tests need to be visible and instantiable to be picked up by the JUnit framework and its various runners. This also applies to test suites and the individual tests that are part of a test suite. As a consequence, if JUnit picks up the test suite it will also pick up the individual tests, and all tests in the suite will be executed twice, once individually and once as part of the suite.
So, the trick, if you will, is to prevent JUnit from picking up the individual tests while still being able to instantiate and execute them as part of the suite.
One thing that comes to mind is to make the test classes static inner classes nested inside the test suite. However, the nested classes still need to be public (otherwise they can't be run in the suite either), and if they are public classes they will also be picked up individually, despite being nested inside the suite's public class. JUnit will not try to run test classes that are not considered visible, though. So, nesting the test classes inside a non-public class would presumably be sufficient to hide them, but we can't make the suite class non-public because then JUnit would not execute it. What we can do, however, is to nest the individual tests inside another non-public class that's nested inside the test suite, which leads us to the solution of this conundrum:
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
import org.junit.runners.Suite.SuiteClasses;
#RunWith(Suite.class)
#SuiteClasses({AllTests.InSuiteOnly.Test1.class, AllTests.InSuiteOnly.Test2.class})
public class AllTests
{
static class InSuiteOnly
{
public static class Test1
{
#Test
public void test1()
{
//...
}
}
public static class Test2
{
#Test
public void test2()
{
//...
}
}
}
}
A lot of folks will probably object to all tests needing to be inside a single source file now. What if I want to maintain separate JUnit test classes that don't get executed by themselves but still get executed inside the test suite? A simple solution is to make the individual test classes abstract (public/non-public doesn't matter) so that JUnit won't execute them, and inside the test suite we simply use concrete subclasses of the original abstract test classes:
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
import org.junit.runners.Suite.SuiteClasses;
#RunWith(Suite.class)
#SuiteClasses({AllTests.InSuiteOnly.SuiteTest1.class, AllTests.InSuiteOnly.SuiteTest2.class})
public class AllTests
{
static class InSuiteOnly
{
public static class SuiteTest1 extends Test1 {}
public static class SuiteTest2 extends Test2 {}
}
}
abstract class Test1
{
#Test
public void test1()
{
//...
}
}
abstract class Test2
{
#Test
public void test2()
{
//...
}
}
This scheme works with Maven, Eclipse, and all other environments that either directly leverage JUnit's runners or implement their own runners that closely follow JUnit's original behavior and semantics.
I have an idea for you. Actually you do not want to run these test case as stand-alone test cases. You can do the following.
Mark the test cases with annotation #RunWith(DoNothingRunner.class)
Implment DoNothingRunner as following:
public class DoNothingRunner extends Runner {
public Description getDescription() {
return "do nothing";
}
public void run(RunNotifier notifier) {
// indeed do nothing
}
}
I have not tried this personally but I hope this will work.
do you need the suite in the first place ? depending on when you click for run all (class, package, or src/test/java), all underlying tests will be executed. So what's the point of having a suite ?
There is a solution, it's a bit tricky, but it may easily resolve your problem: create one suite class, and include all your suite classes in it. Then you can use this suite class to run all your tests.
#RunWith(Suite.class)
#Suite.SuiteClasses({
AXXSuite.class,
BXXSuite.class,
CXXSuite.class
})
public class AllSuites {
}