We Keep Coding

Creating a configurable JUnit library to test same features across several microservices

A set of tests should be run on every microservice. Current solution is to have an abstract class and extend in every service, providing the necessary properties in abstract getters.
public abstract class AbstractTest {
#LocalServerPort
protected int serverPort;
protected abstract String getPath();
#Test
void someTest() {}
#Test
void conditionalTest() {}
}
#SpringBootTest(
webEnvironment = SpringBootTest.WebEnvironment.DEFINED_PORT,
classes = {...})
#ActiveProfiles(...) // etc
public class MyTest extends AbstractTest {
// ... implement getPath()
// tests from parent will be executed
}
The goal:
Ditch inheritance and have the AbstractTest's logic get executed automatically with conditional #Test execution based on beans/properties etc.
The possible solution:
A concrete class with all the tests or some sort of Configuration/TestFactory to create the necessary tests. It should take into account available properties and beans to determine which tests to run.
The problem:
How can those tests (created in runtime) be discovered and registered for execution?
How to inject all the properties that are part of the current context of the #SpringBootTest?
Failed attempts:
TestInstanceFactory extension doesn't seem to be the solution as it requires an instance of the class which it annotates.
Using the Launcher API seems overkill, and also doesn't seem to work, since the library class won't be created with the Spring context configs.
using cglib and a base class Spring Contract-style is not a desirable solution
Ideally I don't want the client of this lib to implement/create anything, so abstract String getPath(); would be a test.lib.path property, and if it's present, a test from the library which uses it will run.
Any thoughts on this would be great, because right now this just seems impossible to me.

What is the reason to have the inheritance for tests?
In case you need to share some common logic within the tests you may try JUnit features (custom rules/extensions), for example
For junit < 5.x.x #Rule functionality https://junit.org/junit4/javadoc/4.12/org/junit/rules/TemporaryFolder.html https://stackoverflow.com/a/34608174/6916890
For junit >= 5.x.x (jupiter) there is an extension API
https://junit.org/junit5/docs/current/user-guide/#writing-tests-built-in-extensions-TempDirectory

Spring Boot / JUnit, run all unit-tests for multiple profiles

I have a BaseTest class which consists of several tests. Each test shall be executed for EVERY profile I list.
I thought about using Parameterized values such as:
#RunWith(Parameterized.class)
#SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
// #ActiveProfiles("h2-test") // <-- how to iterate over this?
public abstract class BaseTest {
#Autowired
private TestRepository test;
// to be used with Parameterized/Spring
private TestContextManager testContextManager;
public BaseTest(String profile) {
System.setProperty("spring.profiles.active", profile);
// TODO what now?
}
#Parameterized.Parameters
public static Collection<Object[]> data() {
Collection<Object[]> params = new ArrayList<>();
params.add(new Object[] {"h2-test" });
params.add(new Object[] {"mysql-test" });
return params;
}
#Before
public void setUp() throws Exception {
this.testContextManager = new TestContextManager(getClass());
this.testContextManager.prepareTestInstance(this);
// maybe I can spinup Spring here with my profile?
}
#Test
public void testRepository() {
Assert.assertTrue(test.exists("foo"))
}
How would I tell Spring to run each test with these different profiles? In fact, each profile will talk to different datasources (in-memory h2, external mysql, external oracle, ..) so my repository/datasource has to be reinitialized.
I know that I can specify #ActiveProfiles(...) and I can even extend from BaseTest and override the ActiveProfile annotation. Although this will work, I only show a portion of my test-suite. Lots of my test-classes extend from BaseTest and I don't want to create several different profile-stubs for each class. Currently working, but ugly solution:
BaseTest (#ActiveProfiles("mysql"))
FooClassMySQL(annotation from BaseTest)
FooClassH2(#ActiveProfiles("h2"))
BarClassMySQL(annotation from BaseTest)
BarClassH2(#ActiveProfiles("h2"))
Thanks

For what it's worth:
My use case was to run a specific test class for multiple spring profiles, this is how I achieved it:
#SpringBootTest
abstract class BaseTest {
#Test void doSomeTest() {... ARRANGE-ACT-ASSERT ...}
}
#ActiveProfiles("NextGen")
class NextGenTest extends BaseTest {}
#ActiveProfiles("Legacy")
class LegacyTest extends BaseTest {}

If you use Maven you can actually specify active profile from command line (or env variable if needed):
mvn clean test -Dspring.profiles.active=h2-test
The approach with parameterized test may not work in this case, because profile has to be specified before Spring boots up its context. In this case when you run parameterized integration test the context will be already booted up before test runner starts running your test. Also JUnit's parameterized tests were invented for other reasons (running unit tests with different data series).
EDIT: Also one more thing - when you decide to use #RunWith(Parameterized.class) you wont be able to use different runner. In many cases (if not all if it comes to integration testing) you want to specify different runner, like SpringRunner.class - with parameterized test you wont be able to do it.

Spring profiles are not designed to work in this way.
In your case, each profile uses a specific datasource.
So each one requires a Spring Boot load to run tests with the expected datasource.
In fact, what you want to do is like making as many Maven build as Spring profiles that you want to test.
Besides, builds in local env should be as fast as possible.
Multiplying automated tests execution by DBMS implementation that requires a Spring Boot reload for each one will not help.
You should not need to specify #ActiveProfiles .
It looks rather like a task for a Continuous Integration tool where you could define a job that executes (sequentially or parallely) each Maven build by specifying a specific Spring Boot profile :
mvn clean test -Dspring.profiles.active=h2
mvn clean test -Dspring.profiles.active=mysql
etc...
You can also try to perform it in local by writing a script that performs the execution of the maven builds.
But as said, it will slowdown your local build and also complex it.

getting TestNG to treat class variables like JUnit with Guice

I am trying to setup TestNG so that it gives me new instances of my class variable for each test (basically like JUnit). I need this as I intend to parallelize my tests at the method level. I have been experimenting with both standalone Guice and the built in Guice functionality that TestNG provides to try to accomplish this but I have had no luck. I know that I can use ThreadLocal, but calling .get() for every variable in the test is pretty unappealing. I am weary of using GuiceBerry as it does not really have a lot of updates/activity and it's last release is not even acquirable via Maven. I am pretty set on TestNG as for all the inconvenience this is causing me it still does a lot of great things. I am open to things other tools though to accomplish my goal. Basically I want things setup so the below tests would work consistently. Any help would be greatly appreciated.
// just has a variable thats a class called child with a simple string variable
// with a value of "original
Parent p;
#Test
public void sometest1(){
p.child.value = "Altered";
Assert.assertTrue(p.child.value.equals("Altered"));
}
#Test
public void sometest2(){
Assert.assertTrue(p.child.value.equals("original"));
}

TestNG doesn't create a new instance for each test. If you want such a behavior than I recommend creating separate test classes. e.g.:
public class SomeTest1 {
Parent p;
#Test
public void something(){
p.child.value = "Altered";
Assert.assertTrue(p.child.value.equals("Altered"));
}
}
public class SomeTest2 {
Parent p;
#Test
public void something(){
Assert.assertTrue(p.child.value.equals("original"));
}
}
Note that TestNG can run JUnit 3 and JUnit 4 tests (you might maintain a mixed suite depending on the style you want to use in a given test class).

How to rollback DB changes in #AfterClass?

I know how to configure Spring/JUnit to rollback after each test case.
What I am after is a way to start and roll back one transaction for all test cases.
I am using #BeforeClass to prepare my HSQL DB for several test cases. Then I want to rollback the changes after the end of all test cases in #AfterClass.
What is the best way to achieve this rollback?
Here is my code example:
#BeforeClass
public static void setupDB(){
ApplicationContext context = new ClassPathXmlApplicationContext(
"classpath:/spring/applicationContext-services-test.xml");
//- get beans and insert some records to the DB
...
}
#AfterClass
public static void cleanUp(){
??? what should go here?
}
Any idea on the best way to do rollback in AfterClass?
Thanks to all..

In case it's acceptable to you to roll back after each test and to use Spring, the following snippet from my project might help you:
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(locations = { "classpath:/net/sukharevd/shopzilla/model/application-context-dao.xml" })
#TestExecutionListeners(DependencyInjectionTestExecutionListener.class)
#TransactionConfiguration(transactionManager = "transactionManager", defaultRollback = true)
#Transactional
public class HibernateCategoryDaoTest extends AbstractTransactionalJUnit4SpringContextTests {

You probably don't want to be doing a rollback after the class, but after each test. Tests are not guaranteed to run in the same order each time, so you might get different results. Unit tests should be isolated.
You can use the spring test runner to annotate your class and rollback transactions
#RunWith(SpringJUnit4ClassRunner.class)
#TransactionConfiguration(defaultRollback=true)
public static class YourTestClass {
#Test
#Transactional
public void aTest() {
// do some db stuff that will get rolled back at the end of the test
}
}
In general though you should also try to avoid hitting a real database in unit tests. Tests that hit the database are generally integration level tests (or even more coarse grain tests like acceptance tests). The DBUnit http://www.dbunit.org/ framework is used to stub out databases for unit tests so you dont' need to use a real database.

I managed to solve this issue using just Spring/JUnit (without using DBUnit).
In short, the solution was to call transactionManager.getTransaction(def).setRollbackOnly(); in #BeforeClass.
Let me first explain what I was trying to do.
My main motive was around this flow:
1. Start transaction
2. Insert load test data
3. run several test cases on the same test data
4. rollback test data.
Since I am building my load test data in #BeforeClass, I was looking to rollback in #AfterClass. This seems to be unnecessary as I can simply instruct the transaction to be rollback only in my #BeforeClass!
So here is how I did it:
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(locations = "classpath:/spring/applicationContext-services-test.xml")
#TestExecutionListeners(inheritListeners = false, listeners = { SpecialDependencyInjectionTestExcecutionListener.class })
#TransactionConfiguration(defaultRollback = true)
#Transactional
public class loadTest {
...
private static HibernateTransactionManager transactionManager;
...
#BeforeClass
public static void setupDB() {
//- set the transaction to rollback only. We have to get a new transaction for that.
DefaultTransactionDefinition def = new DefaultTransactionDefinition();
def.setPropagationBehavior(TransactionDefinition.PROPAGATION_REQUIRES_NEW);
transactionManager.getTransaction(def).setRollbackOnly();
...
//- start loading the data using the injected services.
...
}
This helped to rollback at the end of the class.
P.S. The SpecialDependencyInjectionTestExcecutionListener is an extension to DependencyInjectionTestExecutionListener which I used to override beforeTestClass to force the application.context to to be loaded before calling #BeforeClass. Credit goes to Dmitriy in highlighting this Listener which was the hint to solve another problem which i had in my mind.
Thanks to everyone who helped in highlighting and suggestions which collectively led me to this solution.
Dhafir

Let's pretend you weren't using Spring. (Because using Spring the other answers here are better.) Then you would have three choices:
Use dbunit to take care of the data load/cleanup for you. (The site is down at the moment but if you Google it, you can see some tutorials.)
Create manual deletes for the manual updates
Create a rollback point in your database as the first step of setup. Here's how to do so in Oracle.

JUnit 4 test suite problems

I have a problem with some JUnit 4 tests that I run with a test suite.
If I run the tests individually they work with no problems but when run in a suite most of them, 90% of the test methods, fail with errors. What i noticed is that always the first tests works fine but the rest are failing. Another thing is that a few of the tests the methods are not executed in the right order (the reflection does not work as expected or it does because the retrieval of the methods is not necessarily in the created order). This usually happens if there is more than one test with methods that have the same name. I tried to debug some of the tests and it seems that from a line to the next the value of some attributes becomes null.
Does anyone know what is the problem, or if the behavior is "normal"?
Thanks in advance.
P.S.:
OK, the tests do not depend on each other, none of them do and they all have the #BeforeClass, #Before, #After, #AfterClass so between tests everything is cleared up. The tests work with a database but the database is cleared before each test in the #BeforeClass so this should not be the problem.
Simplefied example:
TEST SUITE:
import org.junit.BeforeClass;
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
importy testclasses...;
#RunWith(Suite.class)
#Suite.SuiteClasses({ Test1.class, Test2.class })
public class TestSuiteX {
#BeforeClass
public static void setupSuite() { System.out.println("Tests started"); }
#AfterClass
public static void setupSuite() { System.out.println("Tests started"); }
}
TESTS:
The tests are testing the functionalily on a server application running on Glassfish.
Now the tests extend a base class that has the #BeforeClass - method that clears the database and login's and the #AfterClass that only makes a logoff.
This is not the source of the problems because the same thing happened before introducing this class.
The class has some public static attributes that are not used in the other tests and implements the 2 controll methods.
The rest of the classes, for this example the two extends the base class and does not owerride the inherited controll methods.
Example of the test classes:
imports....
public class Test1 extends AbstractTestClass {
protected static Log log = LogFactory.getLog( Test1.class.getName() );
#Test
public void test1_A() throws CustomException1, CustomException2 {
System.out.println("text");
creates some entities with the server api.
deletes a couple of entities with the server api.
//tests if the extities exists in the database
Assert.assertNull( serverapi.isEntity(..) );
}
}
and the second :
public class Test1 extends AbstractTestClass {
protected static Log log = LogFactory.getLog( Test1.class.getName() );
private static String keyEntity;
private static EntityDO entity;
#Test
public void test1_B() throws CustomException1, CustomException2 {
System.out.println("text");
creates some entities with the server api, adds one entities key to the static attribute and one entity DO to the static attribute for the use in the next method.
deletes a couple of entities with the server api.
//tests if the extities exists in the database
Assert.assertNull( serverapi.isEntity(..) );
}
#Test
public void test2_B() throws CustomException1, CustomException2 {
System.out.println("text");
deletes the 2 entities, the one retrieved by the key and the one associated with the static DO attribute
//tests if the deelted entities exists in the database
Assert.assertNull( serverapi.isEntity(..) );
}
This is a basic example, the actual tests are more complex but i tried with simplified tests and still it does not work.
Thank you.

The situation you describe sounds like a side-effecting problem. You mention that tests work fine in isolation but are dependent on order of operations: that's usually a critical symptom.
Part of the challenge of setting up a whole suite of test cases is the problem of ensuring that each test starts from a clean state, performs its testing and then cleans up after itself, putting everything back in the clean state.
Keep in mind that there are situations where the standard cleanup routines (e.g., #Before and #After) aren't sufficient. One problem I had some time ago was in a set of databases tests: I was adding records to the database as a part of the test and needed to specifically remove the records that I'd just added.
So, there are times when you need to add specific cleanup code to get back to your original state.

It seems that you built your test suite on the assumption that the order of executing methods is fixed. This is wrong - JUnit does not guarantee the order of execution of test methods, so you should not count on it.
This is by design - unit tests should be totally independent of each other. To help guaranteeing this, JUnit creates a distinct, new instance of your test class for executing each test method. So whatever attributes you set in one method, will be lost in the next one.
If you have common test setup / teardown code, you should put it into separate methods, annotated with #Before / #After. These are executed before and after each test method.
Update: you wrote
the database is cleared before each test in the #BeforeClass
if this is not a typo, this can be the source of your problems. The DB should be cleared in the #Before method - #BeforeClass is run only once for each class.

Be careful in how you use #BeforeClass to set up things once and for all, and #Before to set up things before each individual test. And be careful about instance variables.
We may be able to help more specifically, if you can post a simplified example of what is going wrong.