I know you have fail() to mark that a test has failed under specific condition if used with if condition. However, it's my understanding that fail() is used to mark the failure of the assumption of the test, not the code of the test itself. For example, say I have expectedFile and actualFile, but if we don't have expectedFile, I want to abort the test case. I could:
if(expectedFile == null){
fail()
}
But the logging would imply that it's what we are testing that's failing, not the test case itself. I was wondering if there is a separate way of indicating that the failure comes from the test case itself rather than from the unit that we are testing.
You could use the Assume.assumeNotNull(...) method which would throw an AssumptionViolatedException for you. Something like:
import static org.junit.Assume.assumeNotNull;
// later in the code
void someMethod() {
assumeNotNull(expectedFile, actualFile);
// Rest of code goes here
}
An alternative might be to use Objects.requireNonNull(...), which will throw a NullPointerException if argument is null.
I wonder why you don't simply use assertEquals() since if actualFile is null it fails with the error message:
expected "the expected file name" but was NULL
which is quite easy to understand...
Related
I have an object as the result of an API call and I want to assert the values of a member variable.
This value can be one of the two expected values depending on what the API call "sees" first and sets first. So if the assertion on one value fails, I want to assert for another value before declaring the test as a failure.
What is the best way of doing this? What I have right now is:
try {
assertEquals("message", someObject.getValue1(), expectedValue1);
} catch(AssertionError ae) {
assertEquals("message", someObject.getValue1(), expectedValue2);
}
I am just not sure if this is an acceptable practice. Please comment.
Using exceptions as a sort of glorified goto statement is generally not a good practice. Or at least you will run into people in your career who take a dim view of using exceptions for program flow control.
How about:
Assert.assertTrue((someObject.getValue1().equals(expectedValue1) || (someObject.getValue2().equals(expectedValue2));
Depends on the purpose, automated functional testing or unit testing. I sometimes do this for the former:
try {
assertTrue(boolean condition from earlier in test method to check);
}
catch(AssertionError uhOh) {
Logger.err("condition X failed: detailed info msg"); // broken item #1
}
try {
assertTrue(another condition in same method to check);
}
catch(AssertionError yuck) {
Logger.err("condition X failed: detailed info msg"); // another broken item
fail(); // now blow up as we've checked everything
}
Of course that's using logback Logger and JUnit's Assert.fail() which fails the test method. This way I know of all failures for this method rather than blowing up after the first. In my case, I'm testing a rich-content web app (dialogs and pages that take a lot of user input).
Downside of "fail-fast" (using no catches) is finding one problem, fixing it, running again and finding a new one ("rinse and repeat"), but if used for unit testing this is an asset due to the modularity of the tests (ideally you are only testing one aspect of an item per test).
I'll agree with Aquilon regarding this not being good practice.
However, can you use mocking or some other mechanism to "force" the API "see" one item before the other? That way your tests can reflect the conditions that lead to one assertion being true in one test, and the other assertion being true in another test.
I'm writing a Unit test for the behavior of a method, and want to try to reduce my code duplication - I'm about to double the test cases again, and the file is getting a bit hard to look at.
If anyone's really interested, the code is here (warning to the future - link may not be permanent).
Right now, I do the following, repeated several times in slightly different ways:
#Test
public void testAdd64OfMax64() {
// Set up container
TileEntityChest nmsinv = new TileEntityChest();
Inventory container = new CraftInventory(nmsinv);
// Call method
HashMap<Integer,ItemStack> leftover = container.addItem(new ItemStack(Foo, 64));
// Set up expected result
ItemStack[] expected = empty.clone();
expected[0] = new ItemStack(Foo, 64);
// Verify result
assertThat(container.getContents(), is(expected));
assertThat(leftover, is(Collections.EMPTY_MAP));
}
(Note: assertThat comes from org.hamcrest.Matchers.)
I know that I can put the "Set up container" and cloning the empty array into a #Before method.
My question is, can I put the assertThat method into an #After method and have the test still fail (when it should fail)?
No. The After method should be used to release external resources after a test run. Also, note that it is guaranteed to run even if your Test method throws an exception which may not be desirable in your case.
Also, think about the people who will have to maintain your code in the future. You are violating the PLA, by validating the results of your tests in the After method. Nobody expects that!
It would be better if you created a helper method e.g. checkResults which was called at the end of each of your Test methods to validate your results.
Sometimes, I encounter situations where all I need to test is whether the program's execution reaches a certain point without any exceptions being thrown or the program being interrupted or getting caught in an infinite loop or something.
What I don't understand is how to write a unit test for that.
For instance, consider the following "unit test" -
#Test
public void testProgramExecution()
{
Program program = new Program();
program.executeStep1();
program.executeStep2();
program.executeStep3();
// if execution reaches this point, that means the program ran successfully.
// But what is the best practice?
// If I leave it like this, the test will "pass",
// but I am not sure if this is good practice.
}
Usually, at the end of a test, I have a statement like-
assertEquals(expectedString, actualString);
But how to write an assertEquals or other type of test statement for the above case?
Your code looks fine, just remove the comments, but leave this one:
// If execution reaches this point, that means the program ran successfully.
So readers of your code will understand why there are no assertions.
It is worth noting that every method called in your test should have some kind of effect, and that effect should be asserted as having happened correctly, even if you say "you don't care".
If you insist there is no need to check, add a comment to explain why - this will save readers from trawling through your code to find out for themselves why "it doesn't matter", for example:
// No assertions have been made here because the state is unpredictable.
// Any problems with execution will be detected during integration tests.
In situations like that I am just insert
assertTrue(true);
at the end of the function...
If the mere act of an exception not being thrown means the test has passed, you don't need any assertions. Admittedly that suggests that the action has no observable side-effects, which is somewhat rare. A comment would be more useful than an assertion in this case.
Usually I find this only happens when I've got other tests for checking the actual results, and some tests proving invalid input, and a few similar tests proving just about valid input. For example, if I wanted to validate that the input had to be in the range [0, 100] I may well have "full" tests for a few medium values, then invalid values of -1 and 101, then valid values of 0 and 100 which just prove they're valid.
Can you please help me better understand, what is an appropriate use of “assert” vs “throwing an exception? When is each scenario appropriate?
Scenario 1
CODE
public Context(Algorythm algo) {
if (algo == null) {
throw new IllegalArgumentException("Failed to initialize Context");
}
this.algo = algo;
}
TEST
public void testContext_null() {
try {
context = new Context(null);
fail();
} catch (IllegalArgumentException e) {
assertNotNull(e);
}
}
Scenario 2
CODE
public Context(Algorythm algo) {
assert (algo != null);
this.algo = algo;
}
TEST
public void testContext_null() {
try {
context = new Context(null);
fail();
} catch (AssertionFailedError e) {
assertNotNull(e);
}
}
The main difference with assert is;
the ability to turn on/off selected tests by class/package.
the error thrown.
assert is more approriate for tests which will be turned off in production.
If you want a test which is checked every time, esp if validating data from an input, you should use the check which runs every time.
Assert is a macro (in C/C++, or a function in other languages) that validates a given expression as true or false, and throw an exception in case of false values.
Assert is something to use when ddebugging an application, like when you must check if a math expression really gives you an appropriate value, or if an object/structure member is not null or missing something important, and things like that.
An Exception throwing is more of a real error treatment. Exceptions are errors too and can stop your application, but they are used as the (let's say) "retail version" error treatment of the application. That's because Exceptions can be caught and taken differently to the user, with a little non-technical message instead of symbols and memory addresses, while you can just serialize that into an app log, for example.
On the other hand, asserts will just stop the running process and give you a message like "Assertion failed on source_file.ext, line X. The process will be terminated." And that's not user-friendly :)
The assert keyword should be used when failure to meet a condition violates the integrity of the program. These are intended to be non-recoverable error situations.
Exceptions, on the other hand, alert calling methods to the presence and location of an error but can be handled or ignored at the programmer's discretion.
When testing, you should use the Assert functions when a condition must be met for a test to pass. If you're expecting an exception in that particular test, JUnit 4 has an annotation to signify that an test should throw a particular Exception:
#Test(expected=MyException.class)
Outside of test code, asserts are generally a bad idea. the reason is that unless there are very strict company guidelines in place, you invariably end up with mixed usage, which is bad. there are basically 2 usage scenarios for assert:
extra, possibly slow tests which will be turned off in production
normal, quick code sanity tests which should never be disabled (like requiring a given method parameter to be non-null)
As long as you always follow one of the scenarios, things are fine. however, if your code base ends up with both scenarios, then you are stuck. you have asserts which follow scenario 2 which you don't want to disable, and you have asserts which follow scenario 1 (and are slowing down your production code) which you want to disable. what to do?
most codebases which i have worked with which used asserts in normal code, never ended up disabling them in the production build for exactly this reason. therefore, my recommendation is always to avoid them outside of test code. use normal exceptions for the normal code, and stick the extra, possibly slow code (with asserts) in separate test code.
I'd like to write a TestNG test to make sure an exception is thrown under a specific condition, and fail the test if the exception is not thrown. Is there an easy way to do this without having to create an extra boolean variable?
A related blog post on this subject: http://konigsberg.blogspot.com/2007/11/testng-and-expectedexceptions-ive.html
#Test(expectedExceptions) is useful for the most common cases:
You expect a specific exception to be thrown
You need the message of that exception to contain specific words
Per the documentation a test will fail if no expectedException is thrown:
The list of exceptions that a test method is expected to throw. If no exception or a different than one on this list is thrown, this test will be marked a failure.
Here are a few scenarios where #Test(expectedExceptions) is not sufficient:
Your test method has several statements and only one of them is expected to throw
You are throwing your own type of exception and you need to make sure it matches a certain criterion
In such cases, you should just revert to the traditional (pre-TestNG) pattern:
try {
// your statement expected to throw
fail();
}
catch(<the expected exception>) {
// pass
}
Use #Test annotation to check expected exceptions.
#Test(
expectedExceptions = AnyClassThatExtendsException.class,
expectedExceptionsMessageRegExp = "Exception message regexp"
)
Or if you don't want to check for exception message, only below is enough
#Test(expectedExceptions = AnyClassThatExtendsException.class)
In that way, you don't need to use ugly try catch block, just invoke you exception-thrower method inside the test.
If you are using Java 7 and TestNG, the below example can be used. For Java 8 you can also use Lambda Expressions.
class A implements ThrowingRunnable {
#Override
public void run() throws AuthenticationFailedException{
spy.processAuthenticationResponse(mockRequest, mockResponse, authenticationContext);
}
}
assertThrows(AuthenticationFailedException.class, new A());
I have to disagree with the the article on the nature of the testing techniques employed. The solution employs a gate, to verify if the test should succeed or fail in an intermediate stage.
In my opinion, it is better to employ Guard Assertions, especially for such tests (assuming that the test does not turn out to be long-winded and complex, which is an anti-pattern in itself). Using guard-assertions forces you to design the SUT in either of the following ways:
design the method itself to provide enough information in the result on whether the invocation passed or succeeded. Sometimes, this cannot be done as the intention of the designer is to not return a result, and instead throw an exception (this can be handled in the second case).
design the SUT so that it's state can be verified after each significant method invocation.
But before we consider the above possibilities, have a look at the following snippet again:
plane.bookAllSeats();
plane.bookPlane(createValidItinerary(), null);
If the intention is to test bookPlane() and verify for execution of that method, it is better to have bookAllSeats() in a fixture. In my understanding, invoking bookAllSeats() is equivalent to setting up the SUT to ensure that the invocation of bookPlane() fails, and hence having a fixture to do the same would make for a more readable test. If the intention are different, I would recommend testing the state after every transition (as I normally would do in functional tests), to help pinpoint the original cause of failure.
Why don't you use the try/fail/catch pattern mentioned in the blog post you linked to?
catch-exception provides probably everything you need to test for expected exceptions.
I created a custom Stack data structure which is backed by an array. The push() method throws a custom exception when the stack is full and you still try to push() data into the stack. You could handle it like this :
public class TestStackDataStructure {
//All test methods use this variable.
public Stack<String> stack;//This Stack class is NOT from Java.
#BeforeMethod
public void beforeMethod(){
//Don't want to repeat this code inside each test, especially if we have several lines for setup.
stack = new Stack<>(5);
}
#Test
public void pushItemIntoAFullStack(){
//I know this code won't throw exceptions, but what if we have some code that does ?
IntStream.rangeClosed(1,5).mapToObj(i -> i + "").forEach(stack::push);
try{
stack.push("6");
Assert.fail("Exception expected.");
}catch (StackIsFullException ex) {
// do nothing;
}
}
//Other tests here.
}
Alternately, you could change your api as suggested here :
#Test
public void pushItemIntoAFullStack(){
IntStream.rangeClosed(1,5).mapToObj(i -> i + "").forEach(stack::push);
Assert.assertFalse( stack.push("6"), "Expected push to fail." );
}
I updated the push method to return true or false if the operation passed or failed, instead of returning void. The Java Stack.push(item) returns the element you tried to insert instead of void. I don't know why. But, it also inherits a similar method addElement(item) from Vector which returns void.
One minor downside I see to making push(item) return a boolean or void is that you are stuck with those return types. If you return Stack instead then you can write code conveniently like this stack.push(1).push(2).push(3).pop(). But, I don't know how often one would have to write code like that.
Similarly, my pop() method used to return a generic type "T" and used to throw an exception if the stack was empty. I updated it to return Optional<T> instead.
#Test
public void popEmptyStack(){
Assert.assertTrue(stack.pop().isEmpty());
}
I guess I am now free of the clunky try-catch blocks and TestNg expectedExceptions. Hopefully, my design is good now.