We Keep Coding

Start single Serenity scenario from command line

My team received ownership of a webapp. Tests are written with junit suites and serenity. Good things, there a good test coverage. Problem come when you need to run that single test/scenario that is still failing and you need to wait >30min to run everything.
How can I run a single scenario of this suite using mvn command line?
From code editor, it's hard to start single scenario as both suite and test classes contains important initialization code.
I've also tried argument '-Dtest=T1Test#T1Scenario1' without success.
Code snipplet:
#RunWith(Suite.class)
#Suite.SuiteClasses({
UserConfigASuite.class,
UserConfigBSuite.class,
UserConfigCSuite.class
})
public class AllTestSuite {
}
#RunWith(Suite.class)
#Suite.SuiteClasses({
T1Test.class,
T2Test.class,
//... Lots of other tests
})public class UserConfigASuite {
#BeforeClass
public static void beforeClass() {
//Required init code
}
#AfterClass
public static void afterClass() {
//Cleanup after test suite
}
}
#RunWith(SerenityRunner.class)
public class T1Test {
#Test
#Title("T1: scenario 1")
public void T1Scenario1() {
}
//... Lots of other scenarios
}

Just confirm first that your using supported surefire and junit version. For more details refer https://maven.apache.org/surefire/maven-surefire-plugin/examples/single-test.html
In case your using maven failsafe plugin then the syntax will vary little bit. Something like this
mvn -Dit.test=ITCircle#test* verify
Refer https://maven.apache.org/surefire/maven-failsafe-plugin/examples/single-test.html for more details.

JUnit4 Categories results are different between #SuiteClasses vs. TestSuite

I am trying to run a JUnit4 test using categories against an AllTests suite of tests. In example 1 running Suite3 works and example 2 running Suite2 produces the following exception.
java.lang.Exception: Category annotations on Parameterized classes
are not supported on individual methods.
I need to generate the TestSuite as the test is executed.
Any suggestions for how to correct the problem?
Thanks
example 1
#RunWith(Categories.class)
#IncludeCategory(SlowTest.class)
#SuiteClasses(AllTests3.class)
public class Suite3 {
}
#RunWith(Suite.class)
#SuiteClasses({
MathUtilTest.class, MathUtil2Test.class
})
public class AllTests3 {
}
example 2
#RunWith(Categories.class)
#IncludeCategory(SlowTest.class)
#SuiteClasses(AllTests2.class)
public class Suite2 {
}
public final class AllTests2 {
public static TestSuite suite() {
final TestSuite result = new TestSuite();
result.addTest(new JUnit4TestAdapter(MathUtilTest.class));
result.addTest(new JUnit4TestAdapter(MathUtil2Test.class));
return result;
}
private AllTests2() {
}

There are various related bugs in the latest stable version of JUnit 4 (4.12):
https://github.com/junit-team/junit4/issues/1203
Somebody committed a fix for the unreleased 4.13-SNAPSHOT version, which you can build and check out yourself by cloning their GitHub repo over at https://github.com/junit-team/junit4
However, I ran a couple of tests myself and have to conclude that the #Category annotation does work when used on your MathUtilTest class, but only if the class is annotated, the annotation is ignored when used on individual test methods.

Gradle: Exclude/Include test classes that extend a specific class

In my project there are 2 types of tests: simple unit tests and acceptance tests with UI automation. All acceptance tests are extending a common base class:
import org.junit.Test;
public class MyAcceptanceTest extends AbstractAcceptanceTest {
#Test
public void someTest() {...}
}
Is it possible with gradle to filter tests so that only the acceptance tests are executed?
My main goal is that with gradle build only the normal unit tests are executed (all test classes except those that extend from AbstractAcceptanceTest).
The second goal is to have a separate task that only executes the acceptance tests (only test classes that extend AbstractAcceptanceTest). However, it would also be ok to execute all tests with this task (no filtering).
As far as I've found in the docs and by googling, filtering is only possible by file name patterns. But maybe there is some groovy magic that allows filtering like this?
EDIT
I've found a solution that works but feels a little hacky:
I've added a #Category annotation from JUnit to the base class (AbstractAcceptanceTest). The category needs an identifier class for which I've created a class com.example.AcceptanceTest. In my gradle.build file I'm using this configuration:
task acceptanceTests (type: Test) {
test {
useJUnit {
includeCategories 'com.example.AcceptanceTest'
}
}
}
test {
useJUnit {
excludeCategories 'com.example.AcceptanceTest'
}
}
This way the acceptance tests aren't executed in normal build but only with gradle acceptanceTests.

Can I avoid running junit tests twice in eclipse when using a TestSuite?

I need to do some per-suite initialisation (starting a web-server). It is working fine except that when I run all tests in my project in eclipse my tests run twice. My test suite looks a bit like this:
#RunWith(Suite.class)
#Suite.SuiteClasses({
SubtestOne.class,
SubtestTwo.class
})
public class TestSuite
{
[...]
}
public class SubtestOne
{
#Test public void testOne() { [...] }
}
public class SubtestTwo
{
#Test public void testTwo() { [...] }
}
When I run all test in project in eclipse this causes the junit plugin to run the tests twice like this:
SubtestOne
SubtestTwo
TestSuite
SubtestOne
SubtestTwo
Is it possible to make "run all test in project" not run the sub-tests twice? I want my sub tests to be only ever run as part of the suite.

No, the test class will always be started directly and then through the "link" in the suite. This is as expected.
One workaround might to set in the run configuration to only run tests from the package which contains your suites. Open the run configuration and select Run all tests in the selected project, package or source folder then click Search... and select the package.

I realize that this has been asked over 5 years ago, but as quite a few folks up-voted the question I thought I'd still chime in with a solution. Skip right to the end if you just want the solution; read the whole text if you also want to understand it ;-)
First of all, it is indeed possible to ensure that a particular JUnit test class gets only run inside a test suite. Also, it is irrelevant whether you want to run that test suite inside Eclipse (as asked here) or any other tool or environment; this is really a pure JUnit issue for the most part.
Before I sketch out the solution, it might be a good idea to revisit what the exact problem is here. All JUnit tests need to be visible and instantiable to be picked up by the JUnit framework and its various runners. This also applies to test suites and the individual tests that are part of a test suite. As a consequence, if JUnit picks up the test suite it will also pick up the individual tests, and all tests in the suite will be executed twice, once individually and once as part of the suite.
So, the trick, if you will, is to prevent JUnit from picking up the individual tests while still being able to instantiate and execute them as part of the suite.
One thing that comes to mind is to make the test classes static inner classes nested inside the test suite. However, the nested classes still need to be public (otherwise they can't be run in the suite either), and if they are public classes they will also be picked up individually, despite being nested inside the suite's public class. JUnit will not try to run test classes that are not considered visible, though. So, nesting the test classes inside a non-public class would presumably be sufficient to hide them, but we can't make the suite class non-public because then JUnit would not execute it. What we can do, however, is to nest the individual tests inside another non-public class that's nested inside the test suite, which leads us to the solution of this conundrum:
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
import org.junit.runners.Suite.SuiteClasses;
#RunWith(Suite.class)
#SuiteClasses({AllTests.InSuiteOnly.Test1.class, AllTests.InSuiteOnly.Test2.class})
public class AllTests
{
static class InSuiteOnly
{
public static class Test1
{
#Test
public void test1()
{
//...
}
}
public static class Test2
{
#Test
public void test2()
{
//...
}
}
}
}
A lot of folks will probably object to all tests needing to be inside a single source file now. What if I want to maintain separate JUnit test classes that don't get executed by themselves but still get executed inside the test suite? A simple solution is to make the individual test classes abstract (public/non-public doesn't matter) so that JUnit won't execute them, and inside the test suite we simply use concrete subclasses of the original abstract test classes:
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
import org.junit.runners.Suite.SuiteClasses;
#RunWith(Suite.class)
#SuiteClasses({AllTests.InSuiteOnly.SuiteTest1.class, AllTests.InSuiteOnly.SuiteTest2.class})
public class AllTests
{
static class InSuiteOnly
{
public static class SuiteTest1 extends Test1 {}
public static class SuiteTest2 extends Test2 {}
}
}
abstract class Test1
{
#Test
public void test1()
{
//...
}
}
abstract class Test2
{
#Test
public void test2()
{
//...
}
}
This scheme works with Maven, Eclipse, and all other environments that either directly leverage JUnit's runners or implement their own runners that closely follow JUnit's original behavior and semantics.

I have an idea for you. Actually you do not want to run these test case as stand-alone test cases. You can do the following.
Mark the test cases with annotation #RunWith(DoNothingRunner.class)
Implment DoNothingRunner as following:
public class DoNothingRunner extends Runner {
public Description getDescription() {
return "do nothing";
}
public void run(RunNotifier notifier) {
// indeed do nothing
}
}
I have not tried this personally but I hope this will work.

do you need the suite in the first place ? depending on when you click for run all (class, package, or src/test/java), all underlying tests will be executed. So what's the point of having a suite ?

There is a solution, it's a bit tricky, but it may easily resolve your problem: create one suite class, and include all your suite classes in it. Then you can use this suite class to run all your tests.
#RunWith(Suite.class)
#Suite.SuiteClasses({
AXXSuite.class,
BXXSuite.class,
CXXSuite.class
})
public class AllSuites {
}

JUnit: Enable assertions in class under test

I've been bit a few times by Java assert statements that didn't fail in the JUnit test suite because assertions weren't enabled in JUnit's JVM instance. To be clear, these are "black box" assertions inside implementations (checking invariants, etc) not the assertions defined by the JUnit tests themselves. Of course, I'd like to catch any such assertion failures in the test suite.
The obvious solution is to be really careful to use -enableassertions whenever I run JUnit, but I'd prefer a more robust solution. One alternative is to add the following test to every test class:
#Test(expected=AssertionError.class)
public void testAssertionsEnabled() {
assert(false);
}
Is there a more automatic way to accomplish this? A system-wide configuration option to JUnit? A dynamic call I could put in the setUp() method?

In Eclipse you can go to Windows → Preferences → Java → JUnit, which has an option to add -ea everytime a new launch configuration is created. It adds the -ea option to the Debug Configuration as well.
The full text next to a check box is
Add '-ea' to VM arguments when creating a new JUnit launch
configuration

I propose three possible (simple?) fixes which work for me after a quick test (but you might need to check the side effects of using a static-initializer-block)
1.) Add a static-initializer block to those testcases which rely on assertions being enabled
import ....
public class TestXX....
...
static {
ClassLoader.getSystemClassLoader().setDefaultAssertionStatus(true);
}
...
#Test(expected=AssertionError.class)
...
...
2.) Create a base-class which all of your test-classes extend which need assertions enabled
public class AssertionBaseTest {
static {
//static block gets inherited too
ClassLoader.getSystemClassLoader().setDefaultAssertionStatus(true);
}
}
3.) Create a test suite which runs all your test
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
#RunWith(Suite.class)
#Suite.SuiteClasses({
//list of comma-separated classes
/*Foo.class,
Bar.class*/
})
public class AssertionTestSuite {
static {
//should run before the test classes are loaded
ClassLoader.getSystemClassLoader().setDefaultAssertionStatus(true);
}
public static void main(String args[]) {
org.junit.runner.JUnitCore.main("AssertionTestSuite");
}
}

Alternatively, you may compile your code such that assertions cannot be turned off. Under Java 6, you may use "fa.jar – Force assertion check even when not enabled", a small hack of mine.

As a friend of mine says... why take the time to write an assert if you are just going to turn it off?
Given that logic all assert statements should become:
if(!(....))
{
// or some other appropriate RuntimeException subclass
throw new IllegalArgumentException(".........");
}
To answer your question a way you probably want :-)
import org.junit.BeforeClass;
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
#RunWith(Suite.class)
#Suite.SuiteClasses({
FooTest.class,
BarTest.class
})
public class TestSuite
{
#BeforeClass
public static void oneTimeSetUp()
{
ClassLoader.getSystemClassLoader().setDefaultAssertionStatus(true);
}
}
Then run the test suite rather than each test. This should (works in my testing, but I did not read the internals of the JUnit framework code) result in the assertion status being set before any of the test classes are loaded.