Mark unit test as an expected failure in JUnit - java

How can I mark a test as an expected failure in JUnit 4?
In this case I want to continue to run this test until something is patched upstream. Ignoring the test goes a little too far, as then I might forget about it. I may be able to add an #expected annotation and catch the exception thrown by assertThat, but that also seems to lie about the expected behavior.
Here's what my current test looks like:
#Test
public void unmarshalledDocumentHasExpectedValue()
{
doc = unmarshaller.unmarshal(getResourceAsStream("mydoc.xml"));
final ST title = doc.getTitle();
assertThat(doc.getTitle().toStringContent(), equalTo("Expected"));
}
That assert should succeed, but because of an upstream bug it doesn't. Yet, that test is correct; it should succeed. Virtually all the alternatives that I've found are misleading. Right now I think #Ignore("This test should pass once fixed upstream") is my best bet, but I still have to remember to come back to it. I'd prefer that the test run.
In Python I can use the expectedFailure decorator:
class ExpectedFailureTestCase(unittest.TestCase):
#unittest.expectedFailure
def test_fail(self):
self.assertEqual(1, 0, "broken")
With Qt's QTestLib in C++, you can use QEXPECT_FAIL:
QEXPECT_FAIL("", "Will be fixed next version", Continue);
QCOMPARE(i, 42);
In both cases above, the unit test runs which is what I'm hoping to have happen. Am I missing something in JUnit?

I'm not quite getting the specifics of your scenario, but here's how I generally test for expected failure:
The slick new way:
#Test(expected=NullPointerException.class)
public void expectedFailure() {
Object o = null;
o.toString();
}
for older versions of JUnit:
public void testExpectedFailure() {
try {
Object o = null;
o.toString();
fail("shouldn't get here");
}
catch (NullPointerException e) {
// expected
}
}
If you have a bunch of things that you want to ensure throw an exception, you may also want to use this second technique inside a loop rather than creating a separate test method for each case. If you were just to loop through a bunch of cases in a single method using expected, the first one to throw an exception would end the test, and the subsequent cases wouldn't get checked.

What about explicitly expecting an AssertionError?
#Test(expected = AssertionError.class)
public void unmarshalledDocumentHasExpectedValue() {
// ...
}
If you're reasonably confident that only the JUnit machinery within the test would raise AssertionError, this seems as self-documenting as anything.
You'd still run the risk of forgetting about such a test. I wouldn't let such tests into version control for long, if ever.

I'm assuming here that you want the test to pass if your assert fails, but if the assert succeeds, then the test should pass as well.
The easiest way to do this is to use a TestRule. TestRule gives the opportunity to execute code before and after a test method is run. Here is an example:
public class ExpectedFailureTest {
public class ExpectedFailure implements TestRule {
public Statement apply(Statement base, Description description) {
return statement(base, description);
}
private Statement statement(final Statement base, final Description description) {
return new Statement() {
#Override
public void evaluate() throws Throwable {
try {
base.evaluate();
} catch (Throwable e) {
if (description.getAnnotation(Deprecated.class) != null) {
// you can do whatever you like here.
System.err.println("test failed, but that's ok:");
} else {
throw e;
}
}
}
};
}
}
#Rule public ExpectedFailure expectedFailure = new ExpectedFailure();
// actually fails, but we catch the exception and make the test pass.
#Deprecated
#Test public void testExpectedFailure() {
Object o = null;
o.equals("foo");
}
// fails
#Test public void testExpectedFailure2() {
Object o = null;
o.equals("foo");
}
}
First, note that the first method is marked as #Deprecated. I'm using this as a marker for the method for which I want to ignore any assertion failures. You can do whatever you like to identify the methods, this is just an example.
Next, in the ExpectedFailure#apply(), when I do the base.evaluate(), I'm catching any Throwable (which includes AssertionError) and if the method is marked with the annotation #Deprecated, I ignore the error. You can perform whatever logic you like to decide whether you should ignore the error or not, based on version number, some text, etc. You can also pass a dynamically determined flag into ExpectedFailure to allow it to fail for certain version numbers:
public void unmarshalledDocumentHasExpectedValue() {
doc = unmarshaller.unmarshal(getResourceAsStream("mydoc.xml"));
expectedFailure.setExpectedFailure(doc.getVersionNumber() < 3000);
final ST title = doc.getTitle();
assertThat(doc.getTitle().toStringContent(), equalTo("Expected"));
}
For further examples, see ExternalResource, and ExpectedException
Ignoring an expected failure test rather than passing it
If you want to mark you tests as Ignored rather than Success, it becomes a bit more complex, because tests are ignored before they are executed, so you have to retrospectively mark a test as ignored, which would involve constructing your own Runner. To give you a start, see my answer to How to define JUnit method rule in a suite?. Or ask another question.

One option is mark the test as #Ignore and put text in there that is a bug perhaps and awaiting a fix. That way it won't run. It will then become skipped. You could also make use of the extensions to suit your need in a potentially different way.

I've taken Matthew's answer a step further and actually implemented an #Optional annotation you could use instead of the #Deprecated marker annotation he mentions in his answer. Although simple, I'll share the code with you, maybe it's of help for someone:
#Target(ElementType.METHOD)
#Retention(RetentionPolicy.RUNTIME)
#Documented
public #interface Optional {
/**
* Specify a Throwable, to cause a test method to succeed even if an exception
* of the specified class is thrown by the method.
*/
Class<? extends Throwable>[] exception();
}
With a simple alteration of Matt's ExpectedFailure class:
public class ExpectedFailure implements TestRule {
#Override
public Statement apply(final Statement base, final Description description) {
return statement(base, description);
}
private Statement statement(final Statement base, final Description description) {
return new Statement() {
#Override
public void evaluate() throws Throwable {
try {
base.evaluate();
} catch (Throwable e) {
// check for certain exception types
Optional annon = description.getAnnotation(Optional.class);
if (annon != null && ArrayUtils.contains(annon.exception(), e.getClass())) {
// ok
} else {
throw e;
}
}
}
};
}
}
You can now annotate your test method with #Optional and it will not fail, even if the given type of exception is raised (provide one or more types you would like the test method to pass):
public class ExpectedFailureTest {
#Rule public ExpectedFailure expectedFailure = new ExpectedFailure();
// actually fails, but we catch the exception and make the test pass.
#Optional(exception = NullPointerException.class)
#Test public void testExpectedFailure() {
Object o = null;
o.equals("foo");
}
}
[UPDATE]
You could also rewrite your tests using JUnit's org.junit.Assume instead of the tradtional org.junit.Assert, if you want your tests to pass even if the assumption does not hold.
From Assume's JavaDoc:
A set of methods useful for stating assumptions about the conditions in which a test is meaningful.A failed assumption does not mean the code is broken, but that the test provides no useful information. The default JUnit runner treats tests with failing assumptions as ignored.
Assume is available since JUnit 4.4

Use mocked upstream class if possible. Stub it with correct result. Optionally, replace mock with real object after bug is fixed.

Related

Mocking Java Constant variables (public static) using Mockito/JUnit

I need to mock the constant variable in order to test one of my method. How can I do it with Mokito and Junit.
#Component( "mybean" )
#org.springframework.context.annotation.Scope( value="session" )
public class MyBean {
Public void methodToBeTested() {
if (!AppConst.SOME_CONST.equals(obj.getCostCode())) {
// some logic
}
}
}
AppConst class
#Configuration
public class AppConst
{
public static String SOME_CONST;
public static String HOST_URL;
#PostConstruct
public void postConstruct()
{
SOME_CONST = "My Code";
HOST_URL = "Some URL";
}
}
So, from my junit test class, how can I mock the AppConst and it's variables? Now, when I run it, i hit a nullpointer error.
Can this be done with powermock? if yes please give some sample
Mockito version I use.
compile "org.mockito:mockito-all:1.9.5"
compile "org.powermock:powermock-mockito-release-full:1.6.1"
Instead of mocking there would be another solution to be able to test it:
public void methodToBeTested(SomeObject obj) {
performLogic(AppConst.SOME_CONST, obj);
}
boolean performLogic(String check, SomeObject testObj) {
if (!check.equals(obj.getCostCode())) {
// some logic
return true;
}
return false;
}
That way you can test two things, both combined show you that your code works as intended:
public void testMethodToBeTested() {
MyBean mb = new MyBean() {
#Override
void performLogic(String check, SomeObject testObj) {
assertSame("check constant is passed", AppConst.SOME_CONST, check);
}
}
mb.methodToBeTested(new SomeObject());
mb = new MyBean();
SomeObject so = createSomeTestObject("My Code"); // not the actual constant but an equal String
assertFalse("check some logic not occurred", mb.performLogic("My Code", so));
so = createSomeTestObject("Not the constant");
assertFalse("check some logic not occurred", mb.performLogic("Not the constant", so));
assertTrue("check some logic occurred", mb.performLogic("My Code", so));
// additional tests covering the actual logic
}
Another solution could be putting the condition of the if-statement into its own method, e.g. shouldLogicOccurr(String check) and test that method individually.
In other words: Sometimes it's necessary to refactor your code to make tests easier or sometimes even possible at all. A good side effect is the next time you implement something you already have testability in mind and create your code suitable for this in the first place.
Mocking is a good way to get things under test that use third party libraries that can't be changed and have too many dependencies to be set up but if you end up using that for your own code you've got a design issue.

JUnit4 - Test if method does nothing

How can I test if a method does nothing. For example I have a static method that throws an exception if the given string-argument is null or empty (it's meant for argument-validation). Right now my tests look like this:
#Test
public void notNullOrEmpty_doesNothingIfValueIsNotNullOrEmpty() {
Require.notNullOrEmpty(Generate.randomString());
assertTrue(true); // <- this looks very ugly
}
#Test(expected = IllegalArgumentException.class)
public void notNullOrEmpty_throwsExceptionIfValueIsNull() {
Require.notNullOrEmpty(null);
}
#Test(expected = IllegalArgumentException.class)
public void notNullOrEmpty_throwsExceptionIfValueIsEmpty() {
Require.notNullOrEmpty("");
}
How can I make the first test to pass without calling assertTrue(true), there is a Assert.fail() is there something like an Assert.pass()?
EDIT:
Added missing (expected = IllegalArgumentException.class) to 3rd test
You have just to remove the assert in the first method.
#Test
public void notNullOrEmpty_doesNothingIfValueIsNotNullOrEmpty() {
Require.notNullOrEmpty(Generate.randomString());
// Test has passed
}
If the test method runs completely then it means it pass with success. Look at Eclipse junit output:
Update: as an additional comment, if you use Mockito framework you can leverage verify method to verify that a method was called X times. For instance, I used something like this:
verify(cmAlertDao, times(5)).save(any(CMAlert.class));
In your case, since you are testing static methods, then you might find useful using PowerMock which allows you to verify static methods (since Mockito doesn't). And you can use verifyStatic(...).
You should add #Test(expected = YourException.class) annotation.
Try to add to the first test:
#Test
public void notNullOrEmpty_doesNothingIfValueIsNotNullOrEmpty() {
String str = Generate.randomString();
Require.notNullOrEmpty(str);
assertNotNull(str);
}
and probably to you have better to rename it to notNullOrEmpty_doesNothingIfValueIsNotNullOrNotEmpty because you are testing it for not empty value.
A unit test must assert which is expected in the behavior of the method.
If in your specification, when your call notNullOrEmpty()no exception must be thrown when the data is valid and an exception must be thrown when the data is not valid so in your unit test you must do no assertion when the data is valid since if it doesn't success, a exception will be thrown and the test will so be in failure.
#Test
public void notNullOrEmpty_doesNothingIfValueIsNotNullOrEmpty() {
Require.notNullOrEmpty(Generate.randomString());
}

PowerMock UnfinishedStubbingException error on final class

Trying to get Mockito and PowerMock to behave, but I'm getting an UnfinishedStubbingException when trying to run this code:
#RunWith(PowerMockRunner.class)
#PrepareForTest(FileIOHelper.class)
public class FileIOHelperTest {
#Test
public void testIOExceptionOnWrite() {
PowerMockito.mockStatic(FileIOHelper.class);
PowerMockito.doThrow(new IOException()).when(FileIOHelper.class);
PowerMockito.verifyStatic();
FileIOHelper.write(Mockito.anyString(), Mockito.anyString());
}
#After
public void validate() {
Mockito.validateMockitoUsage();
}
}
and this IO class
public final class FileIOHelper {
public static void write(final String file, String message, final boolean appendNewLine) {
if(checkArgs(file, message)) {
final Path path = Paths.get(file);
StandardOpenOption mode = StandardOpenOption.APPEND;
if(Files.notExists(path)) {
mode = StandardOpenOption.CREATE_NEW;
}
if(appendNewLine) {
message += System.getProperty("line.separator");
}
try {
Files.write(path, message.getBytes(), mode);
} catch(IOException e) {
handleException(e, "Problem writing to " + file);
}
}
}
private static boolean checkArgs(final String... args) {
if(args != null && args.length > 0) {
for(final String arg : args) {
if(arg == null || arg.isEmpty()) {
return false;
}
}
}
return true;
}
private static void handleException(final IOException e, final String errorMsg) {
handleException(e, errorMsg, true);
}
private static void handleException(final IOException e, final String errorMsg, final boolean printStace) {
checkArgs(errorMsg);
System.err.println(errorMsg);
System.err.println(e.getMessage());
if(printStace) {
e.printStackTrace();
}
}
}
What I want to do is somehow trigger the IOException so handleException can be tested. Why one might ask? I'm looking at my Jacoco report and I see this:
I've looked at:
How to mock a void static method to throw exception with Powermock?
Powermock/mockito does not throw exception when told to
PowerMockito mock static method which throws exception
https://github.com/jayway/powermock/wiki/MockitoUsage#how-to-stub-void-static-method-to-throw-exception
http://www.johnmullins.co/blog/2015/02/15/beginners-guide-to-using-mockito-and-powermockito-to-unit-test-java/
and I'm still completely lost. I have no idea if I need to trigger the IOException or if I can somehow verify the output of handleException without doThrow. Someone, help!
Error log:
org.mockito.exceptions.misusing.UnfinishedStubbingException:
Unfinished stubbing detected here:
-> at FileIOHelperTest.testIOExceptionOnWrite(FileIOHelperTest.java:8) // doThrow line
E.g. thenReturn() may be missing.
Examples of correct stubbing:
when(mock.isOk()).thenReturn(true);
when(mock.isOk()).thenThrow(exception);
doThrow(exception).when(mock).someVoidMethod();
Hints:
1. missing thenReturn()
2. you are trying to stub a final method, you naughty developer!
3: you are stubbing the behaviour of another mock inside before 'thenReturn' instruction if completed
My recommendation: forget about using PowerMock.
If you have to mock static methods, then build your own little wrapper class around that. Then, for testing, your wrapper can return something you control; and for production usage; your wrapper just calls the static method.
PowerMock looks like the solution to many problems; but rather sooner than later, it will be the cause of much more problems. It breaks coverage, it makes it harder to change the underlying JVM, and so on.
Seriously: if your design can only be tested with PowerMock, this is very often a clear indication that your design is bad. So: focus on reworking your code under test; instead of investing time into a tool like PowerMock that does more harm than good.
I have spent countless hours trying to resolve PowerMock problems; and since I started to instead write "better to test" production code ... I have written hundreds or thousands of tests without ever needing PowerMock again.
In your case: start by avoiding static all over the place. Basically you achieve that by (worst case) pulling little wrapper classes around the static calls you have to make. For testing, you can mock the wrapper object; and in production code, you use dependency injection to provide a (singleton/enum) wrapper object that simply makes the static call.
First of all, IOException is checked exception - it should be declared with throws in the method signature. But your method FileIOHelper.write does not have such. This may be the reason of the UnsutisfiedStubbingException.
I do not understand, what your are trying to test: if the FileIOHelper is a mock - handleException will be never called, since it is called by the real write method, not by mocked.
First, you have to mock the class 'Files', not 'FileIOHelper'. FileIOHelper is the tested class. Second, you didn't specified which method should throw IOException. The unit test method should be as follows (supposing the tested method catches and manage the IOException):
#RunWith(PowerMockRunner.class)
#PrepareForTest(Files.class)
public class FileIOHelperTest {
#Test
public void testIOExceptionOnWrite() {
PowerMockito.mockStatic(Files.class);
PowerMockito.doThrow(new IOException()).when(Files.class);
Files.write("path", "message", true);
FileIOHelper.write("path", "message", true);
PowerMockito.verifyStatic();
Files.write("path", "message", true);
}
}

Why does this JUnit assertion fail if method executed on same line?

I have a simple JUnit test. It fails, unless I have called the relevant function previously.
FAILS:
#Test public void isTotalSuccess(){
writer.calculateStats();
Assert.assertFalse(writer.isTotalSuccess());
}
PASSES and displays "TotalSuccess is false":
#Test public void isTotalSuccess(){
writer.calculateStats();
// One additional line here
System.out.println("TotalSuccess is " + writer.isTotalSuccess());
Assert.assertFalse(writer.isTotalSuccess());
}
I don't understand why this is happening, since usually nested methods/functions are evaluated first, before their enclosing function.
Please explain why the test fails unless the method being tested has already been called once. Is this behaviour specific to JUnit, or have I missed something really obvious?
More information:
These tests are part of WriterTest, which is testing Writer
writer is declared as a field of WriterTest with private static Writer writer = new Writer();
writer.isTotalSuccess() is a very simple method:
public boolean isTotalSuccess() {
if (total == success){
return true;
}
return false;
}

How to mock a String using mockito?

I need to simulate a test scenario in which I call the getBytes() method of a String object and I get an UnsupportedEncodingException.
I have tried to achieve that using the following code:
String nonEncodedString = mock(String.class);
when(nonEncodedString.getBytes(anyString())).thenThrow(new UnsupportedEncodingException("Parsing error."));
The problem is that when I run my test case I get a MockitoException that says that I can't mock a java.lang.String class.
Is there a way to mock a String object using mockito or, alternatively, a way to make my String object throw an UnsupportedEncodingException when I call the getBytes method?
Here are more details to illustrate the problem:
This is the class that I want to test:
public final class A {
public static String f(String str){
try {
return new String(str.getBytes("UTF-8"));
} catch (UnsupportedEncodingException e) {
// This is the catch block that I want to exercise.
...
}
}
}
This is my testing class (I'm using JUnit 4 and mockito):
public class TestA {
#Test(expected=UnsupportedEncodingException.class)
public void test(){
String aString = mock(String.class);
when(nonEncodedString.getBytes(anyString())).thenThrow(new UnsupportedEncodingException("Parsing error."));
A.f(aString);
}
}
The problem is the String class in Java is marked as final, so you cannot mock is using traditional mocking frameworks. According to the Mockito FAQ, this is a limitation of that framework as well.
How about just creating a String with a bad encoding name? See
public String(byte bytes[], int offset, int length, String charsetName)
Mocking String is almost certainly a bad idea.
If all you are going to do in your catch block is throw a runtime exception then you can save yourself some typing by just using a Charset object to specify your character set name.
public final class A{
public static String f(String str){
return new String(str.getBytes(Charset.forName("UTF-8")));
}
}
This way you aren't catching an exception that will never happen just because the compiler tells you to.
As others have indicated, you can't use Mockito to mock a final class. However, the more important point is that the test isn't especially useful because it's just demonstrating that String.getBytes() can throw an exception, which it can obviously do. If you feel strongly about testing this functionality, I guess you could add a parameter for the encoding to f() and send a bad value into the test.
Also, you are causing the same problem for the caller of A.f() because A is final and f() is static.
This article might be useful in convincing your coworkers to be less dogmatic about 100% code coverage: How to fail with 100% test coverage.
From its documentation, JDave can't remove "final" modifiers from classes loaded by the bootstrap classloader. That includes all JRE classes (from java.lang, java.util, etc.).
A tool that does let you mock anything is JMockit.
With JMockit, your test can be written as:
import java.io.*;
import org.junit.*;
import mockit.*;
public final class ATest
{
#Test(expected = UnsupportedOperationException.class)
public void test() throws Exception
{
new Expectations()
{
#Mocked("getBytes")
String aString;
{
aString.getBytes(anyString);
result = new UnsupportedEncodingException("Parsing error.");
}
};
A.f("test");
}
}
assuming that the complete "A" class is:
import java.io.*;
public final class A
{
public static String f(String str)
{
try {
return new String(str.getBytes("UTF-8"));
}
catch (UnsupportedEncodingException e) {
throw new UnsupportedOperationException(e);
}
}
}
I actually executed this test in my machine. (Notice I wrapped the original checked exception in a runtime exception.)
I used partial mocking through #Mocked("getBytes") to prevent JMockit from mocking everything in the java.lang.String class (just imagine what that could cause).
Now, this test really is unnecessary, because "UTF-8" is a standard charset, required to be supported in all JREs. Therefore, in a production environment the catch block will never be executed.
The "need" or desire to cover the catch block is still valid, though. So, how to get rid of the test without reducing the coverage percentage? Here is my idea: insert a line with assert false; as the first statement inside the catch block, and have the Code Coverage tool ignore the whole catch block when reporting coverage measures. This is one of my "TODO items" for JMockit Coverage. 8^)
Mockito can't mock final classes. JMock, combined with a library from JDave can. Here are instructions.
JMock doesn't do anything special for final classes other than rely on the JDave library to unfinalize everything in the JVM, so you could experiment with using JDave's unfinalizer and see if Mockito will then mock it.
You can also use PowerMock's Mockito extension to mock final classes/methods even in system classes such as String. However I would also advice against mocking getBytes in this case and rather try to setup your expectation so that a real is String populated with the expected data is used instead.
You will be testing code that can never be executed. UTF-8 support is required to be in every Java VM, see http://java.sun.com/javase/6/docs/api/java/nio/charset/Charset.html
It is a project requirement that the unit tests coverage percentage must but higher than a given value. To achieve such percentage of coverage the tests must cover the catch block relative to the UnsupportedEncodingException.
What is that given coverage target? Some people would say that shooting for 100% coverage isn't always a good idea.
Besides, that's no way to test whether or not a catch block was exercised. The right way is to write a method that causes the exception to be thrown and make observation of the exception being thrown the success criterion. You do this with JUnit's #Test annotation by adding the "expected" value:
#Test(expected=IndexOutOfBoundsException.class) public void outOfBounds() {
new ArrayList<Object>().get(1);
}
Have you tried passing an invalid charsetName to getBytes(String)?
You could implement a helper method to get the charsetName, and override that method within your test to a nonsense value.
Perhaps A.f(String) should be A.f(CharSequence) instead. You can mock a CharSequence.
If you can use JMockit, look at Rogério answer.
If and only if your goal is to get code coverage but not actually simulate what missing UTF-8 would look like at runtime you can do the following (and that you can't or don't want to use JMockit):
public static String f(String str){
return f(str, "UTF-8");
}
// package private for example
static String f(String str, String charsetName){
try {
return new String(str.getBytes(charsetName));
} catch (UnsupportedEncodingException e) {
throw new IllegalArgumentException("Unsupported encoding: " + charsetName, e);
}
}
public class TestA {
#Test(expected=IllegalArgumentException.class)
public void testInvalid(){
A.f(str, "This is not the encoding you are looking for!");
}
#Test
public void testNormal(){
// TODO do the normal tests with the method taking only 1 parameter
}
}
You can change your method to take the interface CharSequence:
public final class A {
public static String f(CharSequence str){
try {
return new String(str.getBytes("UTF-8"));
} catch (UnsupportedEncodingException e) {
// This is the catch block that I want to exercise.
...
}
}
}
That way, you can still pass in String, but you can mock any way you like.
instead of mocking string inject string value at setup method as below
public void setup() throws IllegalAccessException {
FieldUtils.writeField(yourTestClass, "stringVariableName", "value", true);
}
If you have a block of code that can never actually be run, and a managerial requirement to have 100% test coverage, then something's going to have to change.
What you could do is make the character encoding a member variable, and add a package-private constructor to your class that lets you pass it in. In your unit test, you could call the new constructor, with a nonsense value for the character encoding.

Categories