We Keep Coding

How to monitor the invocation of methods in abstract class using java agent and ASM?

What I want to do is to monitor the invocation of JUnit 4 test methods. The reason I must do this by myself is: I need to record the executed classes during the execution of each test method. So I need to insert some instructions to the test method so that I know when the test start/end and those recorded classes are executed by which test entity. So I need to filter the test methods on my own.
Actually, the reason why I am doing this is not relevant to the question, as I did not mention "JUnit", "test" in the title. The problem can still be a problem in other similar cases.
The case I have is like this:
public abstract class BaseTest {
#Test
public void t8() {
assert new C().m() == 1;
}
}
public class TestC extends BaseTest{
// empty
}
I have also modified Surefire's member argLine so that my agent will be attached (premain mode) when Surefire launch a new JVM process to execute tests.
In my agent class:
public static void premain(String args, Instrumentation inst){
isPreMain = true;
agentArgs = args;
log("args: " + args);
parseArgs(args);
inst.addTransformer(new TestTransformer(), true);
}
My transformer class:
public class TestTransformer implements ClassFileTransformer {
public byte[] transform(ClassLoader loader, String className,
Class<?> classBeingRedefined, ProtectionDomain protectionDomain,
byte[] classfileBuffer) throws IllegalClassFormatException {
log("TestTransformer: transform: " + className);
...
ClassReader cr = new ClassReader(classfileBuffer);
ClassWriter cw = new ClassWriter(cr, ClassWriter.COMPUTE_MAXS);
RecordClassAdapter mca = new RecordClassAdapter(cw, className);
cr.accept(mca, 0);
return cw.toByteArray();
}
}
In my ClassVisitor adapter class:
class RecordClassAdapter extends ClassVisitor {
...
#Override
public MethodVisitor visitMethod(int access, String name, String desc, String signature, String[] exceptions) {
MethodVisitor mv = cv.visitMethod(access, name, desc, signature, exceptions);
mv = new RecordMethodAdapter (...);
return mv;
}
}
In my MethodVisitor adapter class:
class RecordMethodAdapter extends MethodVisitor {
public void visitCode() {
mv.visitCode();
if (isTestMethod){
mv.visitLdcInsn(methodName);
mv.visitMethodInsn(INVOKESTATIC, MyClass, "entityStarted",
"(Ljava/lang/String;)V", false);
}
}
}
Sadly, I found that the abstract class will not get into the transform method, thus I can not instrument the t8 method. TestC should be executed as a test class, but I can never monitor the invocation of TestC.t8.

There are several opportunities to inject logging into the test via the JUnit API. There is no need for instrumentation.
For a very simple setup:
public class BaseTest {
#Test
public void t8() {
System.out.println("Running test "+getClass().getName()+".t8() [BaseTest.t8()]");
}
#Test
public void anotherMethod() {
System.out.println("Running test "
+getClass().getName()+".anotherMethod() [BaseTest.anotherMethod()]");
}
}
public class TestC extends BaseTest {
#Rule
public TestName name = new TestName();
#Before
public void logStart() throws Exception {
System.out.println("Starting test "+getClass().getName()+'.'+name.getMethodName());
}
#After
public void logEnd() throws Exception {
System.out.println("Finished test "+getClass().getName()+'.'+name.getMethodName());
}
}
which will print
Starting test class TestC.t8
Running test TestC.t8() [BaseTest.t8()]
Finished test class TestC.t8
Starting test class TestC.anotherMethod
Running test TestC.anotherMethod() [BaseTest.anotherMethod()]
Finished test class TestC.anotherMethod
You can also implement your own rule. E.g. ad-hoc:
public class TestB extends BaseTest {
#Rule
public TestRule notify = TestB::decorateTest;
static Statement decorateTest(Statement st, Description d) {
return new Statement() {
#Override public void evaluate() throws Throwable {
System.out.println("Starting test "+d.getClassName()+"."+d.getMethodName());
st.evaluate();
System.out.println("Finished test "+d.getClassName()+"."+d.getMethodName());
}
};
}
}
Or as a reusable rule that can be inserted via a single-liner into a test class
public class LoggingRule implements TestRule {
public static final LoggingRule INSTANCE = new LoggingRule();
private LoggingRule() {}
#Override
public Statement apply(Statement base, Description description) {
Logger log = Logger.getLogger(description.getClassName());
log.setLevel(Level.FINEST);
Logger.getLogger("").getHandlers()[0].setLevel(Level.FINEST);
String clName = description.getClassName(), mName = description.getMethodName();
return new Statement() {
#Override
public void evaluate() throws Throwable {
log.entering(clName, mName);
String result = "SUCCESS";
try {
base.evaluate();
}
catch(Throwable t) {
result = "FAIL";
log.throwing(clName, mName, t);
}
finally {
log.exiting(clName, mName, result);
}
}
};
}
}
used as simple as
public class TestB extends BaseTest {
#Rule
public LoggingRule log = LoggingRule.INSTANCE;
}
A different approach is implementing a custom test runner. This allows to apply a behavior to an entire test suite, as test suites are implemented via runners as well.
public class LoggingSuiteRunner extends Suite {
public LoggingSuiteRunner(Class<?> klass, RunnerBuilder builder)
throws InitializationError {
super(klass, builder);
}
#Override
public void run(RunNotifier notifier) {
notifier.addListener(LOG_LISTENER);
try {
super.run(notifier);
} finally {
notifier.removeListener(LOG_LISTENER);
}
}
static final RunListener LOG_LISTENER = new RunListener() {
public void testStarted(Description d) {
System.out.println("Starting test "+d.getClassName()+"."+d.getMethodName());
}
public void testFinished(Description d) {
System.out.println("Finished test "+d.getClassName()+"."+d.getMethodName());
}
public void testFailure(Failure f) {
Description d = f.getDescription();
System.out.println("Failed test "+d.getClassName()+"."+d.getMethodName()
+": "+f.getMessage());
};
};
}
This may get applied to an entire test suite, i.e. still inheriting test methods from BaseTest, you may use
#RunWith(LoggingSuiteRunner.class)
#SuiteClasses({ TestB.class, TestC.class })
public class TestA {}
public class TestB extends BaseTest {}
public class TestC extends BaseTest {}
which will print
Starting test TestB.t8
Running test TestB.t8() [BaseTest.t8()]
Finished test TestB.t8
Starting test TestB.anotherMethod
Running test TestB.anotherMethod() [BaseTest.anotherMethod()]
Finished test TestB.anotherMethod
Starting test TestC.t8
Running test TestC.t8() [BaseTest.t8()]
Finished test TestC.t8
Starting test TestC.anotherMethod
Running test TestC.anotherMethod() [BaseTest.anotherMethod()]
Finished test TestC.anotherMethod
These are only pointers, to suggest studying the API which allows even more. Another point to consider, is that depending on the method you’re using for launching the tests (you mentioned a maven plugin), there might be support for adding a global RunListener right there, without the need to alter the test classes.

Repeat runs from a test suite only displays the final run in eclipse JUnit view

I have test suite with around 100 tests, and it needs to be run with different configuration, I looked at Theories and Parameterized tests, however they are repeated for each and every test, which is very expensive in my case. So I went ahead and created a custom TestSuite to setup the config once and run all the tests, then setup the next config and run all the tests ...
However the Junit window displays only the last run and I am unable to find out the test that failed in a previous run https://pasteboard.co/IsyA71q.png
Is there a way to display all the repeated runs from the suite in the Junit window in eclipse?
I've looked around
org.junit.runner.notification.RunNotifier with little luck
public class ExtendedSuite extends Suite
{
public ExtendedSuite(Class<?> klass, RunnerBuilder builder) throws InitializationError
{
super(klass, builder);
}
#Override
protected void runChild(Runner runner, RunNotifier notifier)
{
Object newInstance = null;
try
{
newInstance = super.getTestClass().getJavaClass().newInstance();
}
catch (InstantiationException | IllegalAccessException e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
List<RepeatConfig> repeatConfigs = super.getTestClass()
.getAnnotatedFieldValues(newInstance, ConfigureSuite.class, RepeatConfig.class);
for (RepeatConfig repeatConfig : repeatConfigs)
{
repeatConfig.beforeSuiteRun();
super.runChild(runner, notifier);
repeatConfig.afterSuiteRun();
}
}
}
public interface RepeatConfig
{
void beforeSuiteRun();
void afterSuiteRun();
}
#Retention(RetentionPolicy.RUNTIME)
#Target(ElementType.FIELD)
#Inherited
public #interface ConfigureSuite
{
}
#RunWith(ExtendedSuite.class)
#SuiteClasses({ MessagingTest.class })
public class AllTests
{
#ConfigureSuite
public RepeatConfig withConfig1 = new RepeatConfig()
{
#Override
public void beforeSuiteRun()
{
System.out.println("beforeSuiteRunwithConfig1");
}
#Override
public void afterSuiteRun()
{
System.out.println("afterSuiteRunwithConfig1");
}
};
#ConfigureSuite
public RepeatConfig withConfig2 = new RepeatConfig()
{
#Override
public void beforeSuiteRun()
{
System.out.println("beforeSuiteRunwithConfig2");
}
#Override
public void afterSuiteRun()
{
System.out.println("afterSuiteRunwithConfig2");
}
};
}
public class MessagingTest
{
static int count = 0;
#BeforeClass
public static void setUp()
{
System.out.println("setting up");
}
#AfterClass
public static void tearDown()
{
System.out.println("tearing down");
}
#Test
public void testMessage1()
{
System.out.println("Executing test1");
count++;
if (count == 1)
Assert.fail();
}
#Test
public void testMessage2()
{
System.out.println("Executing test2");
}
}
I would like to see the individual test results from all the runs in the junit window from eclipse so that I can narrow down the specific config, test pair for debugging purposes.

How to create a custom "ExpectedException" junit rule?

I need to create a Rule to check for exceptions with customized messages. Below is my attempt, but this is not quite correct since I am simply using methods from the standard "ExpectedException". How to do it right?
public class CustomExpectedExceptionRule implements TestRule {
private final ExpectedException delegate = ExpectedException.none();
public static CustomExpectedExceptionRule none() {
return new CustomExpectedExceptionRule();
}
private CustomExpectedExceptionRule() {
}
public void expect(Class<? extends Throwable> type) {
delegate.expect(type);
}
public void expectMessage(String message) {
delegate.expectMessage(message);
}
#Override
public Statement apply(Statement base, Description description) {
return delegate.apply(base, description);
}
Now I'm trying something like that:
private final ExpectedException expectedException = ExpectedException.none();
private Object exception;
private String expectedMessage;
#Override
public Statement apply(Statement base, Description description) {
return new Statement() {
#Override
public void evaluate() throws Throwable {
expectedException.expect((Class<? extends Throwable>) exception);
expectedException.expectMessage(expectedMessage);
base.evaluate();
}
};
}
public void expectedMessage(String expectedMessage) {
this.expectedMessage = expectedMessage;
}
public void expectedException(Object exception) {
this.exception = exception;
}
But this test does not work where the exception is thrown does not pass, though all fields here are passed.
How do I remake it in the correct form?

As I understand the requirement, in your tests you need to:
public class MyTest {
#Rule
ExpectedException expExc = ExpectedException.none();
#Test
public void throwsNothing() {
// "normal tests" not affected.
}
#Test
public void throwsExceptionWithSpecificTypeAndMessage() {
expExc.expect(MyCustomException.class);
expExc.expectMessage("substring, that passes test");// other matchers possible
// do something that (is expected to) raise(s)
// MyCustomException("substring, that passes test").
}
}
..where MyCustomException.class is a custom exception class (the lowest possible in inheritance hierarchy, which you want to "pass"), and substring, that passes test the (part of) the message, which you want to "pass".
Introducing a custom TestRule saves you 1 line/Test. In this simple case I would recommend you, not to implement the interface but extend ExternalResource (, see here)):
class CustomExpectedException extends ExternalResource /*implements (!) TestRule*/ {
private ExpectedException expExc = ExpectedException.none();
/* Parameterize the message and also the class, if it fits your needs,
* alternatively && additionally implement defaults/constants/more methods.*/
public void myExpect(String substr) {
expExc.expect(MyCustomException.class);
expExc.expectMessage(substr);// other matchers possible
}
}
...and then use it like:
public class MyTest {
#Rule
CustomExpectedException expExc = new CustomExpectedException();
...
#Test
public void throwsExceptionWithSpecificTypeAndMessage() {
expExc.myExpect("substring, that passes test");
// do something...
}
}
A rule-less approach(, see here) :
public class MyTest {
#Test
public void throwsExceptionWithSpecificTypeAndMessage() {
try { // !
// do something ...
// after that, fail the test:
org.junit.Assert.fail("expected exception!");
} catch (Exception exc) { // ! here i would recommend "the highest possible Exception" (in inheritance hierarchy) ...even better <code>Throwable</code>.
// this code can be moved to a (static) util method:
if (exc instanceof MyCustomException) {
// make assertions on ((MyCustomException) exc).getMessage();
} else {
org.junit.Assert.fail("UNexpected exception!");
// or rethrow:
// throw exc;
}
}
}
}

#BeforeGroups method ignores dependsOnGroups

I have a class with several tests split into two groups. I want a strict ordering to when the grouped tests are run, such that the tests in group A run first, then a setup method for group B is executed, and then group B runs. For example:
#Test(groups="A")
public void a1() {
// ...
}
#Test(groups="A")
public void a2() {
// ...
}
#BeforeGroups(value="B", dependsOnGroups="A")
public void setupB() {
// ...
}
#Test(groups="B")
public void b1() {
// ...
}
#Test(groups="B")
public void b2() {
// ...
}
The problem I'm running into is that TestNG doesn't seem to be honoring the setupB method. Instead of the expected execution order:
a1/a2
a2/a1
setupB
b1/b2
b2/b1
It executes something like this:
a1
setupB
b1
a2
b2
Any idea what I'm doing wrong with this setup? Am I missing something conceptually about how TestNG's groups work?

Try to specify dependsOnGroups for test methods as well.
public class TestClass {
#Test(groups="B")
public void b1() {
System.out.println("b1");
}
#Test(groups="B")
public void b2() {
System.out.println("b2");
}
#Test(groups="A", dependsOnGroups="B")
public void a1() {
System.out.println("a1");
}
#Test(groups="A", dependsOnGroups="B")
public void a2() {
System.out.println("a2");
}
#BeforeGroups(value="A", dependsOnGroups="B")
public void setupA() {
System.out.println("before");
}
}
I may be wrong about it, but seems that if a test method that belongs to a group has been picked for execution and it does not depend on any groups or methods, it just causes #BeforeGroups-annotated method to be run (ignoring dependsOnGroups specified there). Note that TestNG does not guarantee the execution order without some explicit declaration, e.g. using "depends" or "priority" mechanisms.
Hopefully, Cedric Beust will pay this question a visit.

This might be a workaround for now if it works. Not sure if you can use both annotations together.
#Test(groups="A")
public void a1() {
// ...
}
#Test(groups="A")
public void a2() {
// ...
}
#BeforeGroups(value="B")
#AfterGroups(value="A")
public void setupB() {
// ...
}
#Test(groups="B")
public void b1() {
// ...
}
#Test(groups="B")
public void b2() {
// ..
.
}

As mentioned earlier, when a test method is invoked, it just checks whether the corresponding #BeforeGroups annotated method has been executed for the test group the method belongs to, and if it hasn't, TestNG just invokes the #BeforeGroups method.
The workaround is to add a custom IMethodInterceptor which would add dependencies on the groups from #BeforeGroups#dependsOnGroups for the test methods, related to the same group:
public class TestMethodInterceptor implements IMethodInterceptor {
#Override
public List<IMethodInstance> intercept(List<IMethodInstance> methods, ITestContext context) {
if (!(context instanceof TestRunner)) {
return methods;
}
TestRunner testRunner = (TestRunner) context;
Collection<ITestClass> testClasses = testRunner.getTestClasses();
Map<String, List<String>> groupDependencies = MethodGroupsHelper.findGroupsMethods(testClasses, true).entrySet().stream()
.collect(Collectors.toMap(
Map.Entry::getKey,
groupBeforeMethods -> groupBeforeMethods.getValue().stream()
.flatMap(m -> Arrays.stream(m.getGroupsDependedUpon()))
.collect(Collectors.toList())
));
return methods.stream()
.map(IMethodInstance::getMethod)
.map(method -> {
Set<String> methodGroupDependencies = Stream.ofNullable(method.getGroups())
.flatMap(Arrays::stream)
.flatMap(group -> groupDependencies.getOrDefault(group, List.of()).stream())
.collect(Collectors.toSet());
return methodGroupDependencies.isEmpty() ? method : addGroupDependencies(method, methodGroupDependencies);
})
.map(MethodInstance::new)
.collect(Collectors.toList());
}
private ITestNGMethod addGroupDependencies(ITestNGMethod method, Collection<String> groups) {
String[] methodGroupDependencies;
if (method.getGroupsDependedUpon() == null || method.getGroupsDependedUpon().length == 0) {
methodGroupDependencies = groups.toArray(new String[0]);
} else {
methodGroupDependencies = Stream.concat(Arrays.stream(method.getGroupsDependedUpon()), groups.stream())
.distinct()
.toArray(String[]::new);
}
return new WrappedTestNGMethod(method) {
#Override
public String[] getGroupsDependedUpon() {
return methodGroupDependencies;
}
};
}
}
The TestMethodInterceptor is a listener that can be added to the execution by means of #Listeners annotation.

junit4 creating test suite with specific test methods

In junit4 I want to execute specific test methods from different classes i.e want create a test suite with specific test methods from different classes.
Lets say I have 2 classes:
public class Test_Login {
#Test
public void test_Login_001(){
System.out.println("test_Login_001");
}
#Test
public void test_Login_002(){
System.out.println("test_Login_002");
}
#Test
public void test_Login_003(){
System.out.println("test_Login_003");
}
}
public class Logout {
#Test
public void test_Logout_001(){
System.out.println("test_Logout_001");
}
#Test
public void test_Logout_002(){
System.out.println("test_Logout_002");
}
#Test
public void test_Logout_003(){
System.out.println("test_Logout_003");
}
}
From the above classes I want to execute test methods test_Login_001 , test_Login_003 , test_Logout_002 only.
How this can be achieved in junit4 ?

Since JUnit 4.8 introduced Categories there exists a clean solution, create a TestSuite:
#RunWith(Categories.class)
#IncludeCategory(MustHaveTests.class)
#SuiteClasses( { Test_Login.class, Test_Logout.class })
public class MustHaveTestsTestSuite {
public interface MustHaveTests { /* category marker */ }
}
And add the #Category(MustHaveTests.class) above every test you would like to run with the TestSuite, e.g.:
#Category(MustHaveTests.class)
#Test
public void test_Login_001(){
System.out.println("test_Login_001");
}
When running the TestSuite only the MustHaveTests-"tagged" tests will be executed. More Details on #Category: https://github.com/junit-team/junit4/wiki/categories

You need to create an org.junit.runner.Request and pass it to the JunitCore runner, or actually to any Runner.
JUnitCore junitRunner = new JUnitCore();
Request request = Request.method(Logout.class, "test_Logout_002");
Result result = junitRunner.run(request);
I actually created an Annotation and can search for methods with those annotations and dynamically create Request and run them
public class TestsSuite {
public static void main(String[] args) throws Exception {
Class annotation = MyTestAnnotation.class;
JUnitCore junitRunner = new JUnitCore();
Class testClass = Test_Login.class;
Method[] methods = testClass.getMethods();
for (Method method : methods) {
if (method.isAnnotationPresent(annotation)) {
if (method.isAnnotationPresent(org.junit.Test.class)) {
Request request = Request.method(testClass, method.getName());
Result result = junitRunner.run(request);
System.out.println(result.wasSuccessful());
}
}
}
}
}

This might not be the slickest implementation, but I solved a similar problem by created a new #SuiteMethods annotation as follows:
SuiteMethods.java
#Retention(RUNTIME)
#Target(TYPE)
public #interface SuiteMethods {
String[] value() default {""};
}
FilteredSuite.java
public class FilteredSuite extends Categories {
private static String[] TEST_METHODS_TO_RUN = {""}; // default behavior is to run all methods
private static Class<?> extractMethodNamesFromAnnotation(Class<?> clazz) {
SuiteMethods methodsAnnotation = clazz.getAnnotation(SuiteMethods.class);
if (methodsAnnotation != null) {
// if our MethodsAnnotation was specified, use it's value as our methods filter
TEST_METHODS_TO_RUN = methodsAnnotation.value();
}
return clazz;
}
public static Filter getCustomFilter() {
Filter f = new Filter() {
#Override
public boolean shouldRun(Description desc) {
String methodName = desc.getMethodName();
for (String subString : TEST_METHODS_TO_RUN) {
if (methodName == null || methodName.contains(subString)) {
return true;
}
}
return false;
}
#Override
public String describe() {
return null;
}
};
return f;
}
public FilteredSuite(Class<?> arg0, RunnerBuilder arg1) throws InitializationError {
super(extractMethodNamesFromAnnotation(arg0), arg1);
}
#Override
public void filter(Filter arg0) throws NoTestsRemainException {
// At test suite startup, JUnit framework calls this method to install CategoryFilter.
// Throw away the given filter and install our own method name filter
super.filter(getCustomFilter());
}
}
A Usage Example
#RunWith(FilteredSuite.class)
#SuiteClasses({
GroupRestTest.class,
ScenarioRestTest.class
})
#SuiteMethods({
"testReadOnlyFlag",
"testSheetWriteData",
"testAddScenarioMeta"
})
public class SubsetTestSuite {
}