We Keep Coding

Can Intellij-IDEA be configured to break on exceptions in JUnit tests that lets the test fail?

When I have a JUnit test, I'd like the debugger to stop right at the point where any exception in the tested code is thrown that lets the test fail, in order to inspect what's wrong. Is there a possibility to configure IntelliJ's debugger to do that, and not break on other exceptions?
If I set an exception breakpoint on any uncaught exception, this doesn't work since the exception is caught by JUnit. If I try to have the breakpoint break also on caught exceptions with catch class org.junit., that doesn't work either, since the exception is caught and wrapped by Javas reflection mechanisms before it reaches JUnit. So I'm a bit at loss here - Eclipse just stops at the original exception.
CLARIFICATION: I am talking about exceptions in the code I test or code called from there, not about assertion failures in the tests. For example, consider these tests:
#Test
public void thisShouldBreak() {
int i = 25 - 25;
int j = 17 / i;
}
private void neverBreakHereSinceThisIsCaught() {
int i = 14 - 14;
int j = 29 / i;
}
#Test
public void thisShouldNotBreak() {
try {
neverBreakHereSinceThisIsCaught();
} catch (ArithmeticException e) {
// ignored or handled
}
}
#Test
public void thisShouldNotBreakEither() {
try {
getClass().getDeclaredMethod("neverBreakHereSinceThisIsCaught").invoke(this);
} catch (Exception e) {
// ignored or handled
}
}
I want IntelliJ to stop when executing test thisShouldBreak at the place where the ArithmeticException is thrown, so that I can inspect the value of i that caused the exception. However, I do not want IntelliJ to stop in neverBreakHereSinceThisIsCaught since the exception thrown there doesn't reach JUnit. I tried unsuccessfully:
- an exception breakpoint on caught exceptions breaks in neverBreakHereSinceThisIsCaught, too, and loads of other places.
- an exception breakpoint only on uncaught exception is never hit at all, since JUnit catches and wraps those exceptions.
- a catch class filterorg.junit.*` breaks in lots of internal places of JUnit end Java reflection calls by JUnit, too.

This is what I did:
What you basically want to do is tell Intellij to stop on any exception (Caught & Uncaught) where the catch class filter is junit, so that caught exceptions only by junit cause the debugger the stop.
Here's my Intellij settings:
You might run into a similar issue when you use a app server like Tomcat, which catches exception. The same way, you would filter the catch class by Tomcat packages.

You could add a filter to check if the stack trace contains your package. It will probably make the execution slower, but it will not stop for JUnit initialisation exceptions that don't prevent test execution anyway, and it will stop only if the calls involve some of your classes. Something along the lines of:

How I deal with this:
Set Java Exception Breakpoint to java.lang.AssertionError
Debug a test that fails, it will break inside Assertion.java
Look at the debugger and navigate to the test you want to debug as shown here:
You can now check variables, evaluate expressions, etc in your test

If you only need exceptions that are caught in JUnit, you can use "catch class filters" in exception breakpoint and specify a class inside JUnit where they are catched.

After fiddling around for a while, the following seems to work best:
"Any exception" Breakpoint on caught and uncaught exceptions with a catch class filter org.junit.runners.* and a class filter -org.junit.* to get rid of caught exceptions that are thrown by the JUnit mechanics itself.
A unconditional breakpoint on caught and uncaught AssertionError seems to be a good idea, too, to inspect the data in the local variables in the test itself when an assertion fails.

Java: Testing if an exception is caught within a larger system

I know there are a number of questions on this topic, but all of them seem to assume one of two things:
You just want to test if an exception was thrown and not caught,
You should test the function that is inside of the try block
directly
I'm not sure how I can apply those options to this case. I have a small try/catch block, like so:
try {
o.getDataContainer().build(...);
o2.setDataContainer(o.getDataContainer());
} catch (final Exception e) {
LOGGER.error("Data set failed", e);
}
As you can see, if o.getDataContainer() returns null, an exception would be triggered. However, that exception is then caught, and the test tool considers it a successful test. Is it possible to test that the exception occurred without changing the code?
I ask because our logging system triggers a trouble ticket if it picks up an exception that is caught and logged. Since it is common human error to forget to guard something like this, I would like to write UTs that can test if an exception was triggered and caught. I can't remove the whole-program protection provided by the catch block, but the error would also cause a degradation of the user experience, since the data isn't being passed along. (I work in a place where minutes of site downtime equal millions of dollars lost.)
In other words: The exception is an error, and we want to log it and investigate it, but in case this manages to trigger on prod, we don't want to risk the whole site going down.
Note: This try/catch sits inside a much larger function with many such try/catch blocks. One could easily argue bad overall design, but fixing it is not an option (without a huge amount of free dev time, at least).
Update: As the task at hand does not allow me to spend a great deal of time on this, I went with a very simple generic test that would fail if the guard and catch were both removed so that I could move on. But, I'm leaving the question unanswered for now in hopes of continuing conversation. I would love to be able to write a simple UT for each new feature that fails if any exceptions are triggered and caught.

Ignoring the issues with this code (Sometimes you've gotta put lipstick on a pig, I guess), this is how I might handle the situation.
I'd use Mockito and mock o2, then use an Answer to ensure the method is invoked.
A test might look like this:
#RunWith(MockitoJUnitRunner.class)
public class TestClass{
#Mock
O2 o2;
#Mock
O1 o1;
boolean exceptionThrown = false;
#Test
public void test(){
Mockito.doAnswer(new Answer<Void>(){
public Void answer(InvocationOnMock invocation) throws Throwable {
exceptionThrown = true;
throw new RuntimeException("some message");
}
}).when(o2).setDataContainer(DataContainer.class);
}
}
Essentially, you can Mock out o2 in your example, and force the exception.
If this doesn't quite do what you want, you may need to mock LOGGER and verify that it's invoked with LOGGER.error("some message");. Unfortunately, mocking statics is not at all elegant, but it can be done with PowerMock.

You could add a custom handler to LOGGER that just throws when an error is logged. For java.util.logging you could do something like:
LOGGER.addHandler(new Handler() {
public void publish(LogRecord record) {
if ("Data set failed".equals(record.getMessage())) {
throw new RuntimeException(record.getThrown());
}
}
public void flush() {}
public void close() throws SecurityException {}
});
I think log4j calls it "Appender," but the same principle should work. See How to create a own Appender in log4j? or How to Create a Custom Appender in log4j2?

test exception is not thrown

I'm creating an integration test:
#RunWith(CdiRunner.class)
#AdditionalClasses({FollowUpActivityRepository.class, SettingsPropertiesProducer.class})
public class FollowUpActivityFeaturesTest {
#Inject protected FollowUpActivityService fuaService;
#Test
public void DigitalInputTOFollowUpActivityFIELDS()
{
FollowUpActivityDTO dto = new FollowUpActivityDTO();
dto.setId("id");
dto.setTimestamp(Date.from(Instant.now()));
dto.setDueTimestamp(Date.from(Instant.now()));
dto.setClosingTimestamp(Date.from(Instant.now()));
dto.setMatter("matter");
dto.setComment("comment");
this.fuaService.createOrUpdate(dto);
}
}
createOrUpdate is like:
public void createOrUpdate(FollowUpActivityDTO dto) throws RepositorySystemException
So, I need to check this exception is NOT thrown.
I'd like to do it elegantly.
Actually, I'm using junit 4.12 and hamcrest 2.0.0.0.
Any ideas?
Example
In .NET, I'm using NSubstitute in order to get that:
this.apiClient.Invoking(c => c.GrantAuthorization()).ShouldNotThrow();

Edit after you reversed the meaning of the question:
If you want your test to fail if an Exception is thrown, you have nothing more to do than just declare an Exception in the throws part of the test method signature (this is not mandatory if the Exception thrown is some kind of RuntimeException, but yours obviously isn't):
public void DigitalInputTOFollowUpActivityFIELDS() throws Exception
No need to specify any kind of Exception. Anyway, any jUnit test will fail as soon as an unhandled Exception is thrown (which is the behavior you're expecting).
From this blog:
Test methods that declare that they throw one particular type of
exception are brittle because they must be changed whenever the method
under test changes.
Old answer:
Just write your test annotation like this:
#Test(expected=RepositorySystemException.class)
This way, the test method will succeed as soon as this exception is thrown.
See javadoc.
Edit after your comment:
To validate the test against any Exception, just:
#Test(expected=Exception.class)
But as B. Dalton suggested, that seems kind of dangerous, as this test would then pass on any Exception, no matter if it's the one you're expecting or any other.
For the sake of completeness, you can also do something like this (based on this answer):
#Rule
public ExpectedException thrown = ExpectedException.none();
#Test
public void DigitalInputTOFollowUpActivityFIELDS()
{
FollowUpActivityDTO dto = new FollowUpActivityDTO();
dto.setId("id");
dto.setTimestamp(Date.from(Instant.now()));
dto.setDueTimestamp(Date.from(Instant.now()));
dto.setClosingTimestamp(Date.from(Instant.now()));
dto.setMatter("matter");
dto.setComment("comment");
thrown.expect(Exception.class);
thrown.expectMessage("something you can check"); // if needed
this.fuaService.createOrUpdate(dto);
}
This way, createOrUpdate will still be able to validate the test by throwing any kind of Exception, but at least the rest of the method won't.
See javadoc for ExpectedException.
Or, of course, the good old solution:
try {
this.fuaService.createOrUpdate(dto);
fail("this should throw an exception");
} catch (RepositorySystemException e){
// pass
} catch (Exception e){
// pass
}
This is less elegant, but allows you to tweak the exception handling as you need.

How should a unit test deal with expected and unexpected exceptions?

Should it pass the test when the expected exception takes place?
Should it fail the test when an unexpected exception arises?
Is it redundant to handle the exception since it'll fail the test and therefore act as a test?

Test Expected Exceptions
You have to add the expected attribute with the expected exception, so the test will pass if the specified exception is thrown. Otherwise, it will fail.
For example:
#Test(expected=NullPointerException.class)
public void cannotConvertNulls() {
service.convert(null);
}
or...
#Test(expected = ArithmeticException.class)
public void divisionWithException() {
int i = 1/0;
}
Documentation says:
The Test annotation supports two optional parameters. The first,
expected, declares that a test method should throw an exception. If it
doesn't throw an exception or if it throws a different exception than
the one declared, the test fails.
Test Timemouts
Just to let you know, you can also test timeouts.
The second optional parameter, timeout, causes a test to fail if it
takes longer than a specified amount of clock time (measured in
milliseconds). The following test fails:
#Test(timeout=100)
public void infinity() {
while(true);
}
Hope to help

For expected exceptions there are really nice ways to do this with JUnit:
#Test(expected=NullPointerException.class)
public void testNullPointerExceptionIsThrown() {
ArrayList emptyList;
emptyList.size(); //or add, remove etc.
}
The above test in JUnit would pass because it was declared with the #Test annotation that the test method should expect that a null pointer exception is thrown.

If the test is to expect a particular exception will arise with certain data, then yes, it should pass if that particular exception is thrown.
If the test is to expect a particular exception will arise with certain data, or there is no expectation of an exception, then yes, it should fail if any exception outside of expected is thrown.
Do not handle the thrown exceptions yourself unless you have to (and if you have to, that's a test smell - revisit why you're handling exceptions). The best way to indicate to JUnit that you expect an exception is to use the expected field on the #Test annotation.
For example, let's say you were testing a Roman numeral converter, and said that anything not in the normal Roman numerals was an illegal argument (namely, P).
#Test(expected = IllegalArgumentException.class)
public void method_takesIllegalArgument_throwsIllegalArgumentException() {
convertRomanNumeralToNumber("PXL");
}

Exceptions are part of your API, and sometimes explicitly so. As such, you should document the exceptions that your code may throw, and write tests to ensure that they are upheld.
#Test(expected = ThisException.class) is an excellent place to start, if you have JUnit4 and are writing a method that will throw new ThisException(...). Note the value here of picking the most appropriate exception: If you get lazy and use expected = Exception.class, your code would accept it if you change to throw new ThatException(...) instead. Your non-exception code should pass if and only if no exceptions are thrown (which JUnit will enforce for you), and your tests should carefully specify exactly which exception to expect so they can fail if and only if that specific exception is thrown.
As dhiller noted in the comments, the ExpectedException rule is also a very good choice for JUnit4, and allows further inspection of the thrown exception:
#Rule ExpectedException expectedException = ExpectedException.none();
#Test public void yourTest() {
// This is for demonstration. Don't actually verify the exact exception message
// unless you want to have to update your test if the text ever changes.
expectedException.expectMessage("Error 743: Toilet paper missing.");
systemUnderTest.doStuff();
}
But if you really want to check state after catching an exception, the best way to go is JUnit3-style:
#Test public void yourTest() {
try {
systemUnderTest.doStuff();
fail("ThisException expected.");
} catch (ThisException expected) {
assertEquals(743, expected.getErrorNumber());
}
// In both #Test(expected=...) and ExpectedException code, the
// exception-throwing line will be the last executed line, because Java will
// still traverse the call stack until it reaches a try block--which will be
// inside the JUnit framework in those cases. The only way to prevent this
// behavior is to use your own try block.
// This is especially useful to test the state of the system after the
// exception is caught.
assertFalse(systemUnderTest.hasToiletPaper());
}
Another library that claims to help here is catch-exception; however, as of May 2014, the project appears to be in maintenance mode (obsoleted by Java 8), and much like Mockito catch-exception can only manipulate non-final methods.

Correctly setting up Java exception handling

I've been attempting to write Java exception handlers for a while now, have tried multiple methods and have even visited/read through Oracle's "The Java Tutorials" and I still cannot get it straight. I'm unsure what I am doing wrong. I have a given class (TooLowException) for the exception that I am trying to use. In the method I am attempting to use it in I am using an argument that I need to catch if it is less than zero.
public int func(int num) throws TooLowException {
int blah = num + 1;
if ( blah < 0) {
return blah;
}
else {
String error = "Input is too low.";
throw new TooLowException(error);
}
}
This is the exception class:
public class TooLowException extends Exception {
public TooLowException(String response) {
super(response);
}
}
I'm getting the error in Oracle "Unhandled Exception type TooLowException". I've also attempted the try-catch method as well, but it also doesn't work for me. Hopefully this is enough information for someone to point out what I'm doing incorrectly. I need to be set right in my ways of exception handling.

Taken from what info you've given, it seems that you need to have a try/catch block somewhere in your code. Basically somewhere in your application that func(int) method is being called, or needs to be called if you're running into a compiler error telling you the "Unhandled Exception type TooLowException." General rule for exceptions is Handle/Catch or Declare. This can be broken down like this:
Handle/Catch: If you choose to handle the exception, then the "throws" declaration should be removed from the method signature ("public int func(int num) throws TooLowException" becomes "public int func(int num)"). The idea behind this approach is that you as the programmer intend to handle this type of exception because it's specific enough to the method that you don't want external code to have to worry about handling the exception outside of the scope of the method. This requires that you "handle" the exception yourself, by using a try/catch block.
Declare: This is the method you went with. You are stating that whatever class uses this function has the burden of handling the exception with the try/catch block. This would be used if the method you wrote is generic enough that many different applications can use it and that handling the exception should be application specific, i.e., it's up to other developers to handle it in their own way. Some people like to just log things, others like to have a different control flow execute upon receiving certain exceptions.
Here's what works, sorry if it basically answers an exercise you needed to do, but it's in the interest of helping you! Please take time to understand what is happening here:
public class YourClassThisStuffIsIn {
public static int func(int num) throws TooLowException {
int blah = num + 1;
if ( blah < 0) {
return blah;
}
else {
String error = "Input is too low.";
throw new Exception(error);
}
}
public static void main(String[] commandlineArgument){
try {
YourClassThisStuffIsIn.func(3);
} catch (TooLowException tle){
System.out.println("Caught " + e);
}
}
}
When you click run in your IDE, or you run through the console, the JVM looks for the main method with the correct signature. In this case it finds it, and it executes the main method. First line is a try, meaning the JVM has to prepare itself for the possibility of a problem in the application, allowing it to recover in case an exception is thrown. In this case, the only exception that can be thrown is a TooLowException, which you have written how to handle it inside the catch block. Your way of handling it is simply printing the stack trace out, which is fine I think in this situation.
I've changed your example slightly, making your method static just so it's quicker to write. I also suspect that the intent is that the commandLineArgument is meant to be the number passed into the func(int) method, so in that case you're looking at the func method to look like func(Integer.parseInt(commandlineArgument[0])).
Bonus points for you is noticing that parseInt throws a NumberFormatException too, but you will of course remember that java.lang.RuntimeException and its subclasses aren't checked exceptions so there is no requirement to catch them, though it is good practice!