We Keep Coding

How to properly abort improper test case?

I know you have fail() to mark that a test has failed under specific condition if used with if condition. However, it's my understanding that fail() is used to mark the failure of the assumption of the test, not the code of the test itself. For example, say I have expectedFile and actualFile, but if we don't have expectedFile, I want to abort the test case. I could:
if(expectedFile == null){
fail()
}
But the logging would imply that it's what we are testing that's failing, not the test case itself. I was wondering if there is a separate way of indicating that the failure comes from the test case itself rather than from the unit that we are testing.

You could use the Assume.assumeNotNull(...) method which would throw an AssumptionViolatedException for you. Something like:
import static org.junit.Assume.assumeNotNull;
// later in the code
void someMethod() {
assumeNotNull(expectedFile, actualFile);
// Rest of code goes here
}
An alternative might be to use Objects.requireNonNull(...), which will throw a NullPointerException if argument is null.

I wonder why you don't simply use assertEquals() since if actualFile is null it fails with the error message:
expected "the expected file name" but was NULL
which is quite easy to understand...

What is the best practice when unit testing methods that have assertions instead of exceptions?

Let's say you have a public method like this
public Sprite spriteAt(int x, int y)
{
assert withinBorders(x, y) : "PRE: x and y should be inside the borders";
return tileAt(x, y).topSprite();
}
Versus
public Sprite spriteAt(int x, int y)
{
if (!withinBorders(x, y)) throw new InvalidArgumentException();
return tileAt(x, y).topSprite();
}
In the bottom case, I would usually have a unit test cases that checks if an exception is thrown when invalid value for x and/or y are given like:
#Test(expected = InvalidArgumentException.class)
public void SpriteAt_InvalidX_ThrowsInvalidArgumentException()
{
sut.spriteAt(-100, 0);
}
This test case is to ensure that the argument validating logic is implemented in the method.
However, for the assertion method at the top, I am not sure what I am supposed to do. The assertions are not production code and I think this means I don't have to test the assertions.
On the other hand, I think unit tests should notify the developer by failing when there is a change of logic in a method. If I do not write a test case for checking if there is an assertion that checks invalid arguments (just like how I do for exception method), then I may not realize I have made a mistake when I accidentally remove the assertion line of code.
Therefore, what I want to do is to check whether assertion is in place if junit is ran with assertion enabled and don't do anything when assertion is not enabled. Below code will contain pseudocode.
#Test
public void SpriteAt_InvalidX_AssertionThrowsException()
{
if (assertion is enabled)
{
try
{
sut.spriteAt(-100, 0);
fail();
}
catch (AssertionFailureException e)
{
}
}
}
So back to my point. I want to know whether unit tests are supposed to test assertions. If so, am I going in the right direction? If not, how do you prevent from accidentally removing assertion code without unit tests?

Interesting question.
In my opinion if you want to test the case where xor yare not within range explicitly than you should throw the IllegalArgumentException and not use assert. On the other hand you have no method comment that tells the caller of this method that xor ymust be within a range. So if you see it from the callers perspective you have no right to assert this at all. :-p
I would use assert for checking pre conditions I have no unit tests for. But on the other hand you simply should have a unit test for that example.
At the end of the day assert throws a RuntimeException as well and - as you told - only during development time when a compiler flag is that. So why not simply use the IllegalArgumentException which seems perfect for that case. And please give the caller of the method the hint in which case it is thrown with the #throws comment.
My conclusion: assert is rather useless as long as you have good unit test coverage. It's only good for manual testing during development to check up pre conditions where you don't want or can write explicit unit tests for.

You can always test the assert functionality if you turn it on by using -ea java command argument.
assert throws java.lang.AssertionError.
Unit tests should test the behavior, not the implementation. So if the asert behavior is not explicitly defined in the contract (JavaDoc) you should not test it.
assert is pretty useless and is almost never used in real systems.

How to write a unit test in situations where it is obvious by "looking" that the test passed?

Sometimes, I encounter situations where all I need to test is whether the program's execution reaches a certain point without any exceptions being thrown or the program being interrupted or getting caught in an infinite loop or something.
What I don't understand is how to write a unit test for that.
For instance, consider the following "unit test" -
#Test
public void testProgramExecution()
{
Program program = new Program();
program.executeStep1();
program.executeStep2();
program.executeStep3();
// if execution reaches this point, that means the program ran successfully.
// But what is the best practice?
// If I leave it like this, the test will "pass",
// but I am not sure if this is good practice.
}
Usually, at the end of a test, I have a statement like-
assertEquals(expectedString, actualString);
But how to write an assertEquals or other type of test statement for the above case?

Your code looks fine, just remove the comments, but leave this one:
// If execution reaches this point, that means the program ran successfully.
So readers of your code will understand why there are no assertions.
It is worth noting that every method called in your test should have some kind of effect, and that effect should be asserted as having happened correctly, even if you say "you don't care".
If you insist there is no need to check, add a comment to explain why - this will save readers from trawling through your code to find out for themselves why "it doesn't matter", for example:
// No assertions have been made here because the state is unpredictable.
// Any problems with execution will be detected during integration tests.

In situations like that I am just insert
assertTrue(true);
at the end of the function...

If the mere act of an exception not being thrown means the test has passed, you don't need any assertions. Admittedly that suggests that the action has no observable side-effects, which is somewhat rare. A comment would be more useful than an assertion in this case.
Usually I find this only happens when I've got other tests for checking the actual results, and some tests proving invalid input, and a few similar tests proving just about valid input. For example, if I wanted to validate that the input had to be in the range [0, 100] I may well have "full" tests for a few medium values, then invalid values of -1 and 101, then valid values of 0 and 100 which just prove they're valid.

Appropriate use of assert

Can you please help me better understand, what is an appropriate use of “assert” vs “throwing an exception? When is each scenario appropriate?
Scenario 1
CODE
public Context(Algorythm algo) {
if (algo == null) {
throw new IllegalArgumentException("Failed to initialize Context");
}
this.algo = algo;
}
TEST
public void testContext_null() {
try {
context = new Context(null);
fail();
} catch (IllegalArgumentException e) {
assertNotNull(e);
}
}
Scenario 2
CODE
public Context(Algorythm algo) {
assert (algo != null);
this.algo = algo;
}
TEST
public void testContext_null() {
try {
context = new Context(null);
fail();
} catch (AssertionFailedError e) {
assertNotNull(e);
}
}

The main difference with assert is;
the ability to turn on/off selected tests by class/package.
the error thrown.
assert is more approriate for tests which will be turned off in production.
If you want a test which is checked every time, esp if validating data from an input, you should use the check which runs every time.

Assert is a macro (in C/C++, or a function in other languages) that validates a given expression as true or false, and throw an exception in case of false values.
Assert is something to use when ddebugging an application, like when you must check if a math expression really gives you an appropriate value, or if an object/structure member is not null or missing something important, and things like that.
An Exception throwing is more of a real error treatment. Exceptions are errors too and can stop your application, but they are used as the (let's say) "retail version" error treatment of the application. That's because Exceptions can be caught and taken differently to the user, with a little non-technical message instead of symbols and memory addresses, while you can just serialize that into an app log, for example.
On the other hand, asserts will just stop the running process and give you a message like "Assertion failed on source_file.ext, line X. The process will be terminated." And that's not user-friendly :)

The assert keyword should be used when failure to meet a condition violates the integrity of the program. These are intended to be non-recoverable error situations.
Exceptions, on the other hand, alert calling methods to the presence and location of an error but can be handled or ignored at the programmer's discretion.
When testing, you should use the Assert functions when a condition must be met for a test to pass. If you're expecting an exception in that particular test, JUnit 4 has an annotation to signify that an test should throw a particular Exception:
#Test(expected=MyException.class)

Outside of test code, asserts are generally a bad idea. the reason is that unless there are very strict company guidelines in place, you invariably end up with mixed usage, which is bad. there are basically 2 usage scenarios for assert:
extra, possibly slow tests which will be turned off in production
normal, quick code sanity tests which should never be disabled (like requiring a given method parameter to be non-null)
As long as you always follow one of the scenarios, things are fine. however, if your code base ends up with both scenarios, then you are stuck. you have asserts which follow scenario 2 which you don't want to disable, and you have asserts which follow scenario 1 (and are slowing down your production code) which you want to disable. what to do?
most codebases which i have worked with which used asserts in normal code, never ended up disabling them in the production build for exactly this reason. therefore, my recommendation is always to avoid them outside of test code. use normal exceptions for the normal code, and stick the extra, possibly slow code (with asserts) in separate test code.

TestNG: How to test for mandatory exceptions?

I'd like to write a TestNG test to make sure an exception is thrown under a specific condition, and fail the test if the exception is not thrown. Is there an easy way to do this without having to create an extra boolean variable?
A related blog post on this subject: http://konigsberg.blogspot.com/2007/11/testng-and-expectedexceptions-ive.html

#Test(expectedExceptions) is useful for the most common cases:
You expect a specific exception to be thrown
You need the message of that exception to contain specific words
Per the documentation a test will fail if no expectedException is thrown:
The list of exceptions that a test method is expected to throw. If no exception or a different than one on this list is thrown, this test will be marked a failure.
Here are a few scenarios where #Test(expectedExceptions) is not sufficient:
Your test method has several statements and only one of them is expected to throw
You are throwing your own type of exception and you need to make sure it matches a certain criterion
In such cases, you should just revert to the traditional (pre-TestNG) pattern:
try {
// your statement expected to throw
fail();
}
catch(<the expected exception>) {
// pass
}

Use #Test annotation to check expected exceptions.
#Test(
expectedExceptions = AnyClassThatExtendsException.class,
expectedExceptionsMessageRegExp = "Exception message regexp"
)
Or if you don't want to check for exception message, only below is enough
#Test(expectedExceptions = AnyClassThatExtendsException.class)
In that way, you don't need to use ugly try catch block, just invoke you exception-thrower method inside the test.

If you are using Java 7 and TestNG, the below example can be used. For Java 8 you can also use Lambda Expressions.
class A implements ThrowingRunnable {
#Override
public void run() throws AuthenticationFailedException{
spy.processAuthenticationResponse(mockRequest, mockResponse, authenticationContext);
}
}
assertThrows(AuthenticationFailedException.class, new A());

I have to disagree with the the article on the nature of the testing techniques employed. The solution employs a gate, to verify if the test should succeed or fail in an intermediate stage.
In my opinion, it is better to employ Guard Assertions, especially for such tests (assuming that the test does not turn out to be long-winded and complex, which is an anti-pattern in itself). Using guard-assertions forces you to design the SUT in either of the following ways:
design the method itself to provide enough information in the result on whether the invocation passed or succeeded. Sometimes, this cannot be done as the intention of the designer is to not return a result, and instead throw an exception (this can be handled in the second case).
design the SUT so that it's state can be verified after each significant method invocation.
But before we consider the above possibilities, have a look at the following snippet again:
plane.bookAllSeats();
plane.bookPlane(createValidItinerary(), null);
If the intention is to test bookPlane() and verify for execution of that method, it is better to have bookAllSeats() in a fixture. In my understanding, invoking bookAllSeats() is equivalent to setting up the SUT to ensure that the invocation of bookPlane() fails, and hence having a fixture to do the same would make for a more readable test. If the intention are different, I would recommend testing the state after every transition (as I normally would do in functional tests), to help pinpoint the original cause of failure.

Why don't you use the try/fail/catch pattern mentioned in the blog post you linked to?

catch-exception provides probably everything you need to test for expected exceptions.

I created a custom Stack data structure which is backed by an array. The push() method throws a custom exception when the stack is full and you still try to push() data into the stack. You could handle it like this :
public class TestStackDataStructure {
//All test methods use this variable.
public Stack<String> stack;//This Stack class is NOT from Java.
#BeforeMethod
public void beforeMethod(){
//Don't want to repeat this code inside each test, especially if we have several lines for setup.
stack = new Stack<>(5);
}
#Test
public void pushItemIntoAFullStack(){
//I know this code won't throw exceptions, but what if we have some code that does ?
IntStream.rangeClosed(1,5).mapToObj(i -> i + "").forEach(stack::push);
try{
stack.push("6");
Assert.fail("Exception expected.");
}catch (StackIsFullException ex) {
// do nothing;
}
}
//Other tests here.
}
Alternately, you could change your api as suggested here :
#Test
public void pushItemIntoAFullStack(){
IntStream.rangeClosed(1,5).mapToObj(i -> i + "").forEach(stack::push);
Assert.assertFalse( stack.push("6"), "Expected push to fail." );
}
I updated the push method to return true or false if the operation passed or failed, instead of returning void. The Java Stack.push(item) returns the element you tried to insert instead of void. I don't know why. But, it also inherits a similar method addElement(item) from Vector which returns void.
One minor downside I see to making push(item) return a boolean or void is that you are stuck with those return types. If you return Stack instead then you can write code conveniently like this stack.push(1).push(2).push(3).pop(). But, I don't know how often one would have to write code like that.
Similarly, my pop() method used to return a generic type "T" and used to throw an exception if the stack was empty. I updated it to return Optional<T> instead.
#Test
public void popEmptyStack(){
Assert.assertTrue(stack.pop().isEmpty());
}
I guess I am now free of the clunky try-catch blocks and TestNg expectedExceptions. Hopefully, my design is good now.