We Keep Coding

Java and JUnit, find missing #Test decorators

Let's say we have a project full of unit tests (thousands) and they all should look like this
#Test
public void testExceptionInBla() {
// some test
}
But in one case someone forgot to put an #Test decorator on top of the test.
What would be an easy way to spot those tests, without looking through all the code manually?
I want to find code like this, it's a test without #Test:
public void testExceptionInBla() {
// some test
}

I were you I would look at some Sonnar rule here I found something that may can match requirement:
https://rules.sonarsource.com/java/RSPEC-2187

But in one case someone forgot to put an #Test decorator on top of the
test.
And
I want to find code like this, it's a test without #Test:
public void testExceptionInBla() { // some test }
Annotating the method with #Test or specifying a test prefix in the method name is about the same thing in terms of consequences if the developer forgets to do that.
If the #Test is the way today, that is not chance.
The #Test annotation brings two real advantages on the test prefix :
1) it is checked at compile test. For example #Tast will provoke a compilation error while tastWhen...() will not.
2) #Test makes the test method name more straight readable : it allows to focus on the scenario with a functional language.
should_throw_exception_if_blabla() sounds more meaningful than test_should_throw_exception_if_blabla().
About your issue : how to ensure that tests are effectively executed, I would take things in another way. Generally you want to ensure that unit tests execution covers a minimum level of the application source code (while you can go down at package or class level if makes sense).
And that is the coverage tools goal (Jacoco for example) to do that job.
You can even add rules to make the build fail if the level of coverage of classes belonging to some package are not covered at least at a specified minimum level (look at that post).
Small Adding :
If you really ensure that methods of test are correctly annotated, you have a way :
1) you have to choose a convention for test methods : for example all instance and not private methods in a test class are test methods.
2) Create a Sonar rule that retrieves all non private instance methods of test classes and ensure that all these methods are annotated with #Test.
3) Add that rule to your Sonar rules.

java annotations - automated use at runtime

So I have a question in relation to using custom annotations at runtime.
Let me fill you in on what I am trying to achieve. I have created a custom annotation and applied it to a method in a service class.
public class MyService
{
#MyCustomAnnotation
public String connect()
{
return "hello";
}
}
Now I can go and use reflection to process methods and apply some kind of logic to methods which have my custom annotation applied to them.
for(Method method : obj.getClass().getDeclaredMethods())
{
// Look for #MyCustomAnnotation annotated method
if(method.isAnnotationPresent(MyCustomAnnotation.class))
{
// Do something...
}
}
However, I seem to be missing a piece of the puzzle as I can't figure out how to apply the reflection processing step automatically at runtime.
For example if I have the following main method in my application, how/where do I automatically apply the reflection processing step when I run the following?
public static void main(String[] args)
{
MyService service = new MyService();
service.connect();
}
Obviously this is possible as other frameworks such as spring are able to achieve it but I can't find an example of how they do this.

Credit to #Beri
I used aspects to create a solution.

Reflection is useful if you need to do something at compile time, if you want to do something when the method is called, you should use Aspect Oriented Programming. The most common framework for Java is AspectJ.
I have a multimodule example here. It's an example with scala, but you can omit the scala dependencies and classes and implement it with Java.
If you want to use the aspect in more than one place you must implement it in a separate module in order to include it and avoid repeating logic.

Can I use annotation to call method after the annotated one?

I am wondering if I can use custom annotation to call some method right after annotated one. For example I have a class that holds some settings that can also notify objects that something has changed (for example user changed something in settings panel). Not all listeners are interested in all types of events, so MyEvent is enum. Now I have structure like this:
class Settings
{
private ArrayList<Listeners> listeners;
private void notifyListeners(MyEvent e)
{
// notify all listeners that something was changed
}
public void setSomeOption(int value)
{
// validate parameter, store it etc.
notifyListeners(MyEvent.SOME_INTEGER_SETTING_CHANGED);
}
}
Of course listening object has to check type of event and ignore it or perform some action, but it is not the case here.
I am interested if I can achieve this with annotations, like this
#NotifyAnnotation(MyEvent.SOME_INTEGER_SETTING_CHANGED)
public void setSomeOption(int value)
{
// validate parameter, store it etc.
// NO NEED TO CALL NOTIFY HERE - WILL BE HANDLED BY ANNOTATION
}
In JUnit for example, we have #Before or #After annotations, and I am wondering if JUnit has own annotations parser that handles method annotated this way, or this kind of behavior can be done simpler, since annotations can be #Retention(value=RUNTIME).
I know that in this example it might look over-complicated and calling notifyListeners() is much simper, but I wan't to know if annotation can be used the way I described, and if yes, can i get some tips? I don't expect ready solution, just a hint if this is possible and what should I take in consideration.

yes, you can do it but you have to use a framework or write one by yourself. you can use for example spring aspects and #After advice (or any other proxy mechanism). you can also use full aspectj for this. another option is to write it by yourself using reflection api. in last case you will need some kind of inversion of control - some mechanism that will launch your method and then the other method

In annotations you need a class that checks for it. they don't work on themselves.
The way systems check for them are with reflection.
Annotation<NotifyAnnotation> a = method.getAnnotation();
And explicitly call their methods
a.notifyListeners(a.evt);
I can't see any advantage with your case. but I see full of disadvantages. They should not be used in actual coding, just for test systems or similar scenarios, where an external system has control on your class.

It could be do that using bytecode manipulation (JAssist, Asm, Java Rocks ...). All the classes would be instantiated thru a Factory that would identify annotated methods and would inject in the first line of this method a call to the method specified in its annotation.

How can I run particular sets of JUnit tests under specific circumstances?

I have a Java codebase that is developed exclusively in Eclipse. There
are a set of JUnit4 tests that can be divided into two mutually exclusive
subsets based on when they are expected to run:
"Standard" tests should run when a developer right-clicks the test
class (or containing project) and selects Run As > JUnit Test. Nothing
unusual here—this is exactly how JUnit works in Eclipse.
"Runtime" tests should only run when called programmatically from
within the application when it is started up in a specific state.
The two types of tests might sit adjacent to each other in the same Java
package. (Ideally we could intermix them in the same class, though
that's not a hard requirement.)
My first solution was to annotate the "Runtime" test classes with a new
#TestOnLaunch annotation. The application is able to find these classes,
and was running the tests contained in them (annotated with #Test) using
JUnitCore.run(Class<?>...). However, these tests leak into the
"Standard" scenario above, because the Eclipse test runner will run any
method annotated with #Test, regardless of the intent of my custom class
annotation.
Next I tried moving the #TestOnLaunch annotation to the method level.
This prevents the "leakage" of the "Runtime" tests into the "Standard"
scenario, but now I can't seem to get JUnitCore to run those test
methods. run.(Request) with a Request targeted at the correct class and
method, for example, fails with "No runnable methods", presumably
because it can't find the #Test annotation (because it's not there).
I'm very interested to know if there's a "JUnit way" of solving this
kind of problem. Presumably I could write my own Runner (to run methods
annotated with #TestOnLaunch)—is this the right approach? If so, how do
I then kick off the testing programmatically with a bunch of classes,
analogous to calling JUnitCore.run(Class<?>...)?

If you don't mix the two type test method in the same test class, this below may help:
http://johanneslink.net/projects/cpsuite.jsp
You can use the filter feature to setup two test suite.
I setup three test suites in my project by defining several mark interfaces:
UnitTests, IntegrationTests, DeploySmokeTests, AcceptanceTests
And three test suites:
#RunWith(ClasspathSuite.class)
#SuiteTypes({UnitTests.class, IntegrationTests.class})
public class CommitTestSuite {}
#RunWith(ClasspathSuite.class)
#SuiteTypes({DeploySmokeTests.class})
public class DeploySmokeTestSuite {}
#RunWith(ClasspathSuite.class)
#SuiteTypes({AcceptanceTests.class})
public class AcceptanceTestSuite {}
Now you could achieve your goal by running specific test suite. An alternative solution is using junit category:
#Category(IntegrationTests.class)
public class SomeTest {
#Test
public void test1() {
...
}
#Test
public void test2() {
....
}
}
#RunWith(Categories.class)
#IncludeCategory(UnitTests.class, IntegrationTests.class)
#SuiteClasses( { //all test classes })
public class CommitTestSuite {}
As I said if you mix differenct type test method in one test class, the first one can't help you, but by using the seconde solution you could annotate your category interface on test method (I annotated it on test class in the example above). But if you choose the second solution, you have to maintain your test suite every time you add a new test class.

First, you should reevaluate why you're using JUnit tests at runtime; that seems like an odd choice for a problem that probably has a better solution.
However, you should look at using a Filter to determine which tests to run, possibly in conjunction with a custom annotation.

JUnit custom annotation

I wanted to create a custom JUnit annotation, something similar to expected tag in #Test, but I want to also check the annotation message.
Any hints how to do that, or maybe there is something ready?

JUnit 4.9 tightened up the library's use of "rules" for tests, which I think might work as well as a custom annotation. Take a look at TestRule as a starting point. You can implement a rule based on that interface, and then use either the #ClassRule or (method-level) #Rule annotations to put them into play in your tests.
A good concrete example is ExpectedException, which lets you specify exceptions like the expected parameter for #Test does (and then some).

To make JUnit4 pickup your custom annotations, you need to write your own custom Runner implementation, and then supply that to the RunWith-annotation on the Test class.
You can start out by having a look at the BlockJUnit4ClassRunner, which is the default implementation runner for JUnit 4 (if memory serves me well).
Assuming you would want to pick up a custom annotation named #MyTest with a custom runner MyRunner, your test class would look something like:
#RunWith(MyRunner.class)
class Tests {
...
#MyTest
public void assumeBehaviour() {
...
}
}
The answer by "Reid Mac" does a fairly good job at decribing how a custom annotation is implemented.

You can create custom TestRule as mentioned in first answer or you can use/extend TestWatcher that already have method for processing start/finish of test.
There is a method apply(Statement base, Description description) where description is actually a wrapper around your test method. Description has a great method getAnnotation(annotationClass) which will let you do what you want by specifying a custom annotation you want to process