We Keep Coding

Gradle build testing fails all of my tests, but individual running does fine

I am adding unit test classes to production code, and I have told IntelliJ IDEA to use the Gradle tester. I run the tests independently, and they are passing. However, the Gradle test task fails with practically all of my tests failing.
I've tried setting the tester back to the platform tester in the settings, to no apparent change.
#RunWith(MockitoJUnitRunner.class)
public class ChatTranscriptTest extends GeneralUnitTest {
#Test
public void testDiscordMessageSend() {
GuildMessageReceivedEvent guildMessageReceivedEvent = testGuildMessageReceivedEvent();
DiscordChatListener listenerUnderTest = new DiscordChatListener(mockPlugin);
listenerUnderTest.onGuildMessageReceived(guildMessageReceivedEvent);
assertServerBroadcast("<TestUser> Testing");
}
private GuildMessageReceivedEvent testGuildMessageReceivedEvent() {
return new GuildMessageReceivedEvent(jdaMock, 0L, new MessageMock());
}
private void assertServerBroadcast(String broadcast) {
Server targetServer = mockPlugin.getServer();
verify(targetServer, atLeastOnce()).broadcastMessage(broadcast);
}
#Test
public void testMinecraftMessageSend() {
Player playerMock = new MockPlayer();
AsyncPlayerChatEvent playerChatEvent = new AsyncPlayerChatEventTestImpl(playerMock);
MainListener mainListener = new MainListener(mockPlugin);
mainListener.onPlayerChat(playerChatEvent);
assertDiscordMessageSent("**TestUser**: Testing");
}
#SuppressWarnings("ResultOfMethodCallIgnored")
private void assertDiscordMessageSent(String message) {
JDA pluginJDA = mockPlugin.getJDA();
TextChannel targetTextChannel = pluginJDA.getTextChannelById(0L);
verify(targetTextChannel, atLeastOnce()).sendMessage(message);
}
}
This is one of my tests. It's just general behavior testing.
What could be going wrong?

Check that JUnit Extension throws specific Exception

Suppose I develop an extension which disallows test method names to start with an uppercase character.
public class DisallowUppercaseLetterAtBeginning implements BeforeEachCallback {
#Override
public void beforeEach(ExtensionContext context) {
char c = context.getRequiredTestMethod().getName().charAt(0);
if (Character.isUpperCase(c)) {
throw new RuntimeException("test method names should start with lowercase.");
}
}
}
Now I want to test that my extension works as expected.
#ExtendWith(DisallowUppercaseLetterAtBeginning.class)
class MyTest {
#Test
void validTest() {
}
#Test
void TestShouldNotBeCalled() {
fail("test should have failed before");
}
}
How can I write a test to verify that the attempt to execute the second method throws a RuntimeException with a specific message?

Another approach could be to use the facilities provided by the new JUnit 5 - Jupiter framework.
I put below the code which I tested with Java 1.8 on Eclipse Oxygen. The code suffers from a lack of elegance and conciseness but could hopefully serve as a basis to build a robust solution for your meta-testing use case.
Note that this is actually how JUnit 5 is tested, I refer you to the unit tests of the Jupiter engine on Github.
public final class DisallowUppercaseLetterAtBeginningTest {
#Test
void testIt() {
// Warning here: I checked the test container created below will
// execute on the same thread as used for this test. We should remain
// careful though, as the map used here is not thread-safe.
final Map<String, TestExecutionResult> events = new HashMap<>();
EngineExecutionListener listener = new EngineExecutionListener() {
#Override
public void executionFinished(TestDescriptor descriptor, TestExecutionResult result) {
if (descriptor.isTest()) {
events.put(descriptor.getDisplayName(), result);
}
// skip class and container reports
}
#Override
public void reportingEntryPublished(TestDescriptor testDescriptor, ReportEntry entry) {}
#Override
public void executionStarted(TestDescriptor testDescriptor) {}
#Override
public void executionSkipped(TestDescriptor testDescriptor, String reason) {}
#Override
public void dynamicTestRegistered(TestDescriptor testDescriptor) {}
};
// Build our test container and use Jupiter fluent API to launch our test. The following static imports are assumed:
//
// import static org.junit.platform.engine.discovery.DiscoverySelectors.selectClass
// import static org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder.request
JupiterTestEngine engine = new JupiterTestEngine();
LauncherDiscoveryRequest request = request().selectors(selectClass(MyTest.class)).build();
TestDescriptor td = engine.discover(request, UniqueId.forEngine(engine.getId()));
engine.execute(new ExecutionRequest(td, listener, request.getConfigurationParameters()));
// Bunch of verbose assertions, should be refactored and simplified in real code.
assertEquals(new HashSet<>(asList("validTest()", "TestShouldNotBeCalled()")), events.keySet());
assertEquals(Status.SUCCESSFUL, events.get("validTest()").getStatus());
assertEquals(Status.FAILED, events.get("TestShouldNotBeCalled()").getStatus());
Throwable t = events.get("TestShouldNotBeCalled()").getThrowable().get();
assertEquals(RuntimeException.class, t.getClass());
assertEquals("test method names should start with lowercase.", t.getMessage());
}
Though a little verbose, one advantage of this approach is it doesn't require mocking and execute the tests in the same JUnit container as will be used later for real unit tests.
With a bit of clean-up, a much more readable code is achievable. Again, JUnit-Jupiter sources can be a great source of inspiration.

If the extension throws an exception then there's not much a #Test method can do since the test runner will never reach the #Test method. In this case, I think, you have to test the extension outside of its use in the normal test flow i.e. let the extension be the SUT.
For the extension provided in your question, the test might be something like this:
#Test
public void willRejectATestMethodHavingANameStartingWithAnUpperCaseLetter() throws NoSuchMethodException {
ExtensionContext extensionContext = Mockito.mock(ExtensionContext.class);
Method method = Testable.class.getMethod("MethodNameStartingWithUpperCase");
Mockito.when(extensionContext.getRequiredTestMethod()).thenReturn(method);
DisallowUppercaseLetterAtBeginning sut = new DisallowUppercaseLetterAtBeginning();
RuntimeException actual =
assertThrows(RuntimeException.class, () -> sut.beforeEach(extensionContext));
assertThat(actual.getMessage(), is("test method names should start with lowercase."));
}
#Test
public void willAllowTestMethodHavingANameStartingWithAnLowerCaseLetter() throws NoSuchMethodException {
ExtensionContext extensionContext = Mockito.mock(ExtensionContext.class);
Method method = Testable.class.getMethod("methodNameStartingWithLowerCase");
Mockito.when(extensionContext.getRequiredTestMethod()).thenReturn(method);
DisallowUppercaseLetterAtBeginning sut = new DisallowUppercaseLetterAtBeginning();
sut.beforeEach(extensionContext);
// no exception - good enough
}
public class Testable {
public void MethodNameStartingWithUpperCase() {
}
public void methodNameStartingWithLowerCase() {
}
}
However, your question suggests that the above extension is only an example so, more generally; if your extension has a side effect (e.g. sets something in an addressable context, populates a System property etc) then your #Test method could assert that this side effect is present. For example:
public class SystemPropertyExtension implements BeforeEachCallback {
#Override
public void beforeEach(ExtensionContext context) {
System.setProperty("foo", "bar");
}
}
#ExtendWith(SystemPropertyExtension.class)
public class SystemPropertyExtensionTest {
#Test
public void willSetTheSystemProperty() {
assertThat(System.getProperty("foo"), is("bar"));
}
}
This approach has the benefit of side stepping the potentially awkward setup steps of: creating the ExtensionContext and populating it with the state required by your test but it may come at the cost of limiting the test coverage since you can really only test one outcome. And, of course, it is only feasible if the extension has a side effect which can be evaulated in a test case which uses the extension.
So, in practice, I suspect you might need a combination of these approaches; for some extensions the extension can be the SUT and for others the extension can be tested by asserting against its side effect(s).

After trying the solutions in the answers and the question linked in the comments, I ended up with a solution using the JUnit Platform Launcher.
class DisallowUppercaseLetterAtBeginningTest {
#Test
void should_succeed_if_method_name_starts_with_lower_case() {
TestExecutionSummary summary = runTestMethod(MyTest.class, "validTest");
assertThat(summary.getTestsSucceededCount()).isEqualTo(1);
}
#Test
void should_fail_if_method_name_starts_with_upper_case() {
TestExecutionSummary summary = runTestMethod(MyTest.class, "InvalidTest");
assertThat(summary.getTestsFailedCount()).isEqualTo(1);
assertThat(summary.getFailures().get(0).getException())
.isInstanceOf(RuntimeException.class)
.hasMessage("test method names should start with lowercase.");
}
private TestExecutionSummary runTestMethod(Class<?> testClass, String methodName) {
SummaryGeneratingListener listener = new SummaryGeneratingListener();
LauncherDiscoveryRequest request = request().selectors(selectMethod(testClass, methodName)).build();
LauncherFactory.create().execute(request, listener);
return listener.getSummary();
}
#ExtendWith(DisallowUppercaseLetterAtBeginning.class)
static class MyTest {
#Test
void validTest() {
}
#Test
void InvalidTest() {
fail("test should have failed before");
}
}
}
JUnit itself will not run MyTest because it is an inner class without #Nested. So there are no failing tests during the build process.
Update
JUnit itself will not run MyTest because it is an inner class without #Nested. So there are no failing tests during the build process.
This is not completly correct. JUnit itself would also run MyTest, e.g. if "Run All Tests" is started within the IDE or within a Gradle build.
The reason why MyTest was not executed is because I used Maven and I tested it with mvn test. Maven uses the Maven Surefire Plugin to execute tests. This plugin has a default configuration which excludes all nested classes like MyTest.
See also this answer about "Run tests from inner classes via Maven" and the linked issues in the comments.

JUnit 5.4 introduced the JUnit Platform Test Kit which allows you to execute a test plan and inspect the results.
To take a dependency on it from Gradle, it might look something like this:
testImplementation("org.junit.platform:junit-platform-testkit:1.4.0")
And using your example, your extension test could look something like this:
import org.junit.jupiter.api.extension.ExtendWith
import org.junit.jupiter.api.fail
import org.junit.platform.engine.discovery.DiscoverySelectors
import org.junit.platform.testkit.engine.EngineTestKit
import org.junit.platform.testkit.engine.EventConditions
import org.junit.platform.testkit.engine.TestExecutionResultConditions
internal class DisallowUpperCaseExtensionTest {
#Test
internal fun `succeed if starts with lower case`() {
val results = EngineTestKit
.engine("junit-jupiter")
.selectors(
DiscoverySelectors.selectMethod(ExampleTest::class.java, "validTest")
)
.execute()
results.tests().assertStatistics { stats ->
stats.finished(1)
}
}
#Test
internal fun `fail if starts with upper case`() {
val results = EngineTestKit
.engine("junit-jupiter")
.selectors(
DiscoverySelectors.selectMethod(ExampleTest::class.java, "TestShouldNotBeCalled")
)
.execute()
results.tests().assertThatEvents()
.haveExactly(
1,
EventConditions.finishedWithFailure(
TestExecutionResultConditions.instanceOf(java.lang.RuntimeException::class.java),
TestExecutionResultConditions.message("test method names should start with lowercase.")
)
)
}
#ExtendWith(DisallowUppercaseLetterAtBeginning::class)
internal class ExampleTest {
#Test
fun validTest() {
}
#Test
fun TestShouldNotBeCalled() {
fail("test should have failed before")
}
}
}

Failed TestNG tests with DataProvider in IntelliJ IDEA

i recently started playing around with tdd and ran into a problem where i do not understand why one thing is working and the other one doesnt.
the following code works for me:
public class Ant {
public Ant(Point startLocation, Point hive) {
this.currentLocation = new Point(startLocation);
this.hive = new Point(hive);
}
public void goHome() {
if (hive.x > currentLocation.x) {
currentLocation.x++;
} else if (hive.x < currentLocation.x){
currentLocation.x--;
}
if (hive.y > currentLocation.y) {
currentLocation.y++;
} else if (hive.y < currentLocation.y){
currentLocation.y--;
}
}
}
The corresponding test:
#DataProvider(name = "goneHome")
public static Object[][] goHome() {
return new Object[][] {
{new Point(2,1), new Point(3,2), new Point(7,8)},
{new Point(20,1), new Point(19,2), new Point(7,8)},
{new Point(23,10), new Point(22,9), new Point(7,8)},
{new Point(2,10), new Point(3,9), new Point(7,8)},
{new Point(2,8), new Point(3,8), new Point(7,8)},
{new Point(7,1), new Point(7,2), new Point(7,8)}
};
}
#Test(dataProvider = "goneHome")
public void testGoHome(Point currentPosition, Point nextPosition, Point hive)
throws Exception {
Ant ant = new Ant(currentPosition, hive);
ant.move();
assertEquals(ant.getCurrentLocation(), nextPosition);
}
the test fails if i change the ant constructor like this:
public Ant(Point startLocation, Point hive) {
this.currentLocation = startLocation;
this.hive = hive;
}
By failing i mean that the test with the first two sets of the DataProvider work correctly, the rest is failing/not finishing.
Although i am not quite sure what failed. If i remove the first two sets of data in the DataProvider, still only the first two datasets (which where the 3rd and 4th data set before) do not fail.
I use IntelliJ and the symbol besides the "failed" test is still the "loading icon".
Debugging each single test case shows that the points are set correctly. Removing the assert from the test does not change anything.
Can someone explain this behavior to me please?
Thanks in advance
Egon
Edit: corrected the version of the constructor that failed

Maybe it's a bug in IntelliJ IDEA. Sometimes I also facing with this problem. Unfortunatelly it's still (2014-11-24) unresolved: https://youtrack.jetbrains.com/issue/IDEA-100752
Try run your tests with alternate runner (as Maven goal, for instance).

Run Webdriver Junit browser specific test cases

I have selenium.properties in which i specify the test configuration (baseURL, browser etc). This is used by ant script to kick off webdriver junit test cases. Now I have few junit test methods, that i want to run only on Firefox. I was wondering if there a way i can accomplish this using JUnit annotations? can i create custom annotations?
my setup
public class TestBase{
public static String baseURL = null;
public static String browser = null;
#BeforeClass
public static void webdriverSetUp() {
try {
FileInputStream fn = new FileInputStream(SELENIUM_PROP_FILE);
Properties selenium_properties = new Properties();
selenium_properties.load(fn);
baseURL = selenium_properties.getProperty("baseUrl");
browser = selenium_properties.getProperty("browser");
} catch (Exception e) {
e.printStackTrace();
}
if(browserg.equalsIgnoreCase("firefox")){
File profileDirectory = new File("./profile");
FirefoxProfile profile = new FirefoxProfile(profileDirectory);
driver = new FirefoxDriver(profile);
}
}
//Test Class
public class TestCase1 extends TestBase{
#Test //run this case only if browser = firefox
public void test1(){
}
#Test //do not run this case if browser = chrome
public void test2(){
}
}
any pointers?

You can easily do this with JUnit with your own runner. In fact, there is a similar working code in Selenium WebDriver test - it's just backwards. The Selenium guys wanted to skip some tests for particular browsers, so they introduced a custom #Ignore annotation.
Take a look at JUnit4TestBase, SeleniumTestRunner and finally TestIgnorance.
You can use their idea to make the opposite and only run the tests with the desired drivers. However, I think you'll need to write it yourself as I am not aware of any good solution out there.

JUnit Test Suite: Way to create a dataset first before tests start running

I want to setup data for my entire test suite before any of the tests start running. I understand maven runs the test one by one and not a suite, so I cannot use #SuiteClasses. Also I dont want to create the dataset through dbunit-maven-plugin, the dataset has to be created over REST. Is there a way where I can run specific classes as part of maven pre-integration-test and post-integration-test to setup and clean?
For example
public class TestInit
{
public void setUp()
{
//Data setup
}
public void tearDown()
{
//Data clean up
}
}
make setup run before test suite starts and tearDown after it ends. Or can I run 2 separate classes like, TestInitSetup and TestInitTearDown?

Here is a Rule based solution. It may be useful.
The syntax looks like this:
public class SimpleWayToUseDataSetTest {
#Rule
public DataSetRule rule = new DataSetRule(); // <-- this is used to access to the testVectors from inside the tests
public static class MyDataSet extends SimpleTestVectors {
#Override
protected Object[][] generateTestVectors() {
return new Object[][] {
{true, "alpha", new CustomProductionClass()}, // <-- this is a testVector
{true, "bravo", new CustomProductionClass()},
{false, "alpha", new CustomProductionClass()},
{false, "bravo", new CustomProductionClass() }
};
}
}
#Test
#DataSet(testData = MyDataSet.class) // <-- annotate the test with the dataset
public void testFirst() throws InvalidDataSetException { // <-- any access to testData may result in Exception
boolean myTextFixture = rule.getBoolean(0); // <-- this is how you access an element of the testVector. Indexing starts with 0
String myAssertMessage = rule.getString(1); // <-- there are a couple of typed parameter getters
CustomProductionClass myCustomObject = (CustomProductionClass) rule.getParameter(2); // <-- for other classes you need to cast
Assert.assertTrue(myAssertMessage, true);
}
}

If you can't find a solution in JUnit, TestNG supports #BeforeSuite and #AfterSuite, which seem to do what you want.