I have to write some unit tests and for now i have some but i decide to post a theme because i do not know if the test are correct although they run in Eclipse and EclEmma shows that the tests covered tested code (Highlight green). In one sentence are the unit test correct.
Method which i will test
#Service
#Configurable
public class UserManager {
#Autowired
private UserManagerDao umDao;
public long getUserId(String email) throws Exception {
String[] partsOfEmail = email.split("#");
return umDao.getUserId(partsOfEmail [0], partsOfEmail [1]);
}
}
Dao class
public class UserManagerDao extends JdbcDaoSupport {
#Autowired
private MessageSourceAccessor msa;
public long getUserId(String userName, String domain) throws Exception {
String sql = msa.getMessage("sql.select.user_id");
Object[] params = new Object[] { userName, domain };
List<Long> result = getJdbcTemplate().queryForList(sql, params, Long.class);
if (result.size() > 0) {
return result.get(0);
}
return 0;
}
The test
#RunWith(MockitoJUnitRunner.class)
public class UserManagerTest {
#Mock
UserManagerDao umDao;
private String email = "email#email.com";
#Test
public void testGetPassportUserId() throws Exception {
um.getPassportUserId(email);
}
}
The quick answer is 'not quite'.
Your code doesn't seem to be complete - for example, in your test you call um.getPassportUserId(email), but um is not declared (I assume it's UserManager and injected somehow?), and I see no method called getPassportUserId. Ignoring those points, the biggest issue I can see is that you have no assertion in your test; you're not checking that the method returns what you expect.
A common pattern for tests is 'Given, When, Then', meaning that your test should start with a 'given' section - that is, it sets up the mock(s) to return whatever you need, then the 'when' should be your method call and 'then' checks the result of the method call.
So for your example, in Pseudocode, I'd expect something like:
#Test
public void testGetPassportUserId() throws Exception {
// Given...
String username = "foo";
String domain = "bar";
long expectedId = 1L;
// Use Mockito to tell the mocked umDao to return expectedId when you request "foo#bar"
// When...
long actualId = um.getPassportUserId(username + "#" + domain);
// Then...
// Use an assertion to ensure that actualId = expectedId
}
I've deliberately left this quite open-ended, because there are many correct answers to how you set up the mock (check the Mockito documentation) and even more correct ways to do the assertion (check JUnit documentation, as well as Mockito and Hamcrest).
As an aside, this is a good example of a potential pitfall associated with code coverage tools. Emma is doing exactly the right thing in telling you that all the code has been executed, because it has, BUT you have no way of knowing that the execution was correct, without performing some assertions in your test, so it has given you a false sense of security.
Related
I am trying to test the following method with JUnit tests, using Mockito:
#Override public List<Adoption> search(String username, Integer id) {
List<Adoption> emptySearchResult = new ArrayList<>();
if(id != null && !username.equals("") ) {
if(!this.petRepository.findById(id).isPresent()){
return emptySearchResult;
}
if(!this.appUserRepository.findByUsername(username).isPresent()){
return emptySearchResult;
}
Pet pet = this.petRepository.findById(id).orElseThrow( () -> new PetNotFoundException(id));
AppUser user = this.appUserRepository.findByUsername(username).orElseThrow( () -> new UsernameNotFoundException(username));
return this.adoptionRepository.findAllByUserAndPet(user, pet);
}
else if(id != null && username.equals("")){
if(!this.petRepository.findById(id).isPresent()){
return emptySearchResult;
}
Pet pet = this.petRepository.findById(id).orElseThrow( () -> new PetNotFoundException(id));
return this.adoptionRepository.findAllByPet(pet);
}
else if(id == null && !username.equals("")) {
if(!this.appUserRepository.findByUsername(username).isPresent()){
return emptySearchResult;
}
AppUser user = this.appUserRepository.findByUsername(username).orElseThrow( () -> new UsernameNotFoundException(username));
return this.adoptionRepository.findAllByUser(user);
}
else {
return this.adoptionRepository.findAll();
}
}
However, I run into a problem with the following part:
if(!this.petRepository.findById(id).isPresent())
Even though I have mocked this.petRepository.findById(id), for some reason isPresent() is returning false. This is my initialization for the tests:
#Mock
private AdoptionRepository adoptionRepository;
#Mock
private PetRepository petRepository;
#Mock
private AppUserRepository appUserRepository;
private AdoptionServiceImpl service;
private Adoption adoption1;
private Adoption adoption2;
private Adoption adoption3;
private AppUser user;
private AppUser user2;
private Pet pet;
private Pet petAlteadyAdopted;
List<Adoption> allAdoptions = new ArrayList<>();
List<Adoption> userFilteredAdoptions = new ArrayList<>();
List<Adoption> petFilteredAdoptions = new ArrayList<>();
#Before
public void init() {
MockitoAnnotations.initMocks(this);
user = new AppUser("username","name","lastname","email#gmail.com","pass",ZonedDateTime.now(), Role.ROLE_USER, City.Skopje);
user2 = new AppUser("username1","name","lastname","email#gmail.com","pass",ZonedDateTime.now(), Role.ROLE_USER, City.Skopje);
Center center = new Center("a", City.Bitola,"url");
pet = new Pet("p", Type.DOG,"b", Gender.FEMALE,"d",center, ZonedDateTime.now(),"url",null,false,ZonedDateTime.now());
petAlteadyAdopted = new Pet("p", Type.DOG,"b", Gender.FEMALE,"d",center, ZonedDateTime.now(),"url",null,true,ZonedDateTime.now());
pet.setId(0);
petAlteadyAdopted.setId(1);
adoption1 = new Adoption(ZonedDateTime.now(),ZonedDateTime.now(),Status.ACTIVE,user,pet);
adoption2 = new Adoption(ZonedDateTime.now(),ZonedDateTime.now(),Status.CLOSED,user,pet);
adoption3 = new Adoption(ZonedDateTime.now(),ZonedDateTime.now(),Status.CLOSED,user2,new Pet());
allAdoptions.add(adoption1);
allAdoptions.add(adoption2);
allAdoptions.add(adoption3);
petFilteredAdoptions.add(adoption2);
petFilteredAdoptions.add(adoption1);
userFilteredAdoptions.add(adoption2);
userFilteredAdoptions.add(adoption1);
Mockito.when(this.adoptionRepository.findById(0)).thenReturn(java.util.Optional.of(adoption1));
Mockito.when(this.adoptionRepository.findById(1)).thenReturn(java.util.Optional.of(adoption2));
Mockito.when(this.petRepository.findById(0)).thenReturn(java.util.Optional.of(pet));
Mockito.when(this.petRepository.findById(1)).thenReturn(java.util.Optional.of(petAlteadyAdopted));
Mockito.when(this.appUserRepository.findByUsername("username")).thenReturn(java.util.Optional.of(user));
Mockito.when(this.appUserRepository.findByUsername("username1")).thenReturn(java.util.Optional.of(user2));
Mockito.when(this.adoptionRepository.findAll()).thenReturn(allAdoptions);
Mockito.when(this.adoptionRepository.findAllByPet(pet)).thenReturn(petFilteredAdoptions);
Mockito.when(this.adoptionRepository.findAllByUser(user)).thenReturn(userFilteredAdoptions);
Mockito.when(this.adoptionRepository.findAllByUserAndPet(user,pet)).thenReturn(userFilteredAdoptions);
Mockito.when(this.adoptionRepository.save(Mockito.any(Adoption.class))).thenReturn(adoption1);
this.service = Mockito.spy(new AdoptionServiceImpl(this.adoptionRepository, this.petRepository,this.appUserRepository));
}
As a result, the following test fails, even though it should pass:
#Test
public void searchTest2() {
List<Adoption> adoptionList = this.service.search("",0);
Assert.assertEquals(petFilteredAdoptions,adoptionList);
}
In order to solve this I tried mocking the isPresent() method:
Mockito.when(this.petRepository.findById(0).isPresent()).thenReturn(true);
But I'm getting the following exception:
org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
Boolean cannot be returned by findById() findById() should return
Optional***
If you're unsure why you're getting above error read on. Due to the
nature of the syntax above problem might occur because:
This exception might occur in wrongly written multi-threaded tests. Please refer to Mockito FAQ on limitations of concurrency
testing.
A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub spies -
with doReturn|Throw() family of methods. More in javadocs for Mockito.spy() method.
I also tried the following variation:
Mockito.doReturn(true).when(this.petRepository.findById(0)).isPresent();
But then I got the following exception:
org.mockito.exceptions.misusing.UnfinishedStubbingException:
Unfinished stubbing detected here:
-> at mk.finki.ukim.milenichinja.ServiceTests.AdoptionServiceFilterTests.init(AdoptionServiceFilterTests.java:87)
E.g. thenReturn() may be missing. Examples of correct stubbing:
when(mock.isOk()).thenReturn(true);
when(mock.isOk()).thenThrow(exception);
doThrow(exception).when(mock).someVoidMethod(); Hints:
missing thenReturn()
you are trying to stub a final method, which is not supported
you are stubbing the behaviour of another mock inside before 'thenReturn' instruction is completed
Any ideas how to solve this problem?
In the init method, you are stubbing findById on the mock instance this.petRepository to return a non-mock Optional, which is good. In your new test, you are trying to set a return value for isPresent, which you can't do because Optional is not a mock. If you want to override the behavior per-test, you'll need to stub findById to return an Optional of a different instance. Therefore, this is right, though it appears exactly as it does in init and consequently it can't tell you why your test is failing.
Mockito.when(this.petRepository.findById(0))
.thenReturn(java.util.Optional.of(pet));
Mockito works by creating a mock object that subclasses a class and overrides every method. The overridden method is what interacts with a static (ThreadLocal) infrastructure, allowing you to use when syntax. The important thing here is that when ignores its argument, and instead tries to mock the last interaction that you made with a mock. You can find out more in the SO questions How does mockito when() invocation work? and How do Mockito matchers work?.
When you see this call:
Mockito.when(this.petRepository.findById(0))
.thenReturn(java.util.Optional.of(pet));
Then it works as you've intended:
petRepository is a mock, findById is presumably an overridable method, Mockito records the fact that you've called it with the argument 0.
findById doesn't have any behavior stubbed yet, so it does its default, returning null.
when doesn't care that it just received null, because the null doesn't tell it anything about what methods were called to get the null. Instead it looks back at its most recent record (findById(0)) and returns an object with the thenVerb methods you expect.
You call thenReturn, so Mockito sets up petRepository to return the Optional instance you created and passed in.
But when you try this call:
Mockito.when(this.petRepository.findById(0).isPresent()).thenReturn(true);
Then the most recent interaction isn't isPresent, it's findById, so Mockito assumes you want findById(0) to thenReturn(true) and throws WrongTypeOfReturnValue. Optional is not a mock, so interacting with it doesn't let Mockito record its interaction or replay your behavior. For what it's worth, I also wouldn't advise mocking it: Optional is a final class, and though Mockito has recently added some support for mocking final types, Optional is simple and straightforward enough that it makes more sense to just return the Optional instance you want rather than trying to mock it.
With all that said, your code looks right; as long as PetRepository is an interface I don't see anything about the way your method looks or the way your mocks look that would cause this.petRepository.findById(0) to return an absent Optional. In fact, I don't even see where you would create an absent Optional for it to return, so I can only guess that you are using more real objects in your test than you think you are.
I am trying to test the following method with JUnit tests, using Mockito:
#Override public List<Adoption> search(String username, Integer id) {
List<Adoption> emptySearchResult = new ArrayList<>();
if(id != null && !username.equals("") ) {
if(!this.petRepository.findById(id).isPresent()){
return emptySearchResult;
}
if(!this.appUserRepository.findByUsername(username).isPresent()){
return emptySearchResult;
}
Pet pet = this.petRepository.findById(id).orElseThrow( () -> new PetNotFoundException(id));
AppUser user = this.appUserRepository.findByUsername(username).orElseThrow( () -> new UsernameNotFoundException(username));
return this.adoptionRepository.findAllByUserAndPet(user, pet);
}
else if(id != null && username.equals("")){
if(!this.petRepository.findById(id).isPresent()){
return emptySearchResult;
}
Pet pet = this.petRepository.findById(id).orElseThrow( () -> new PetNotFoundException(id));
return this.adoptionRepository.findAllByPet(pet);
}
else if(id == null && !username.equals("")) {
if(!this.appUserRepository.findByUsername(username).isPresent()){
return emptySearchResult;
}
AppUser user = this.appUserRepository.findByUsername(username).orElseThrow( () -> new UsernameNotFoundException(username));
return this.adoptionRepository.findAllByUser(user);
}
else {
return this.adoptionRepository.findAll();
}
}
However, I run into a problem with the following part:
if(!this.petRepository.findById(id).isPresent())
Even though I have mocked this.petRepository.findById(id), for some reason isPresent() is returning false. This is my initialization for the tests:
#Mock
private AdoptionRepository adoptionRepository;
#Mock
private PetRepository petRepository;
#Mock
private AppUserRepository appUserRepository;
private AdoptionServiceImpl service;
private Adoption adoption1;
private Adoption adoption2;
private Adoption adoption3;
private AppUser user;
private AppUser user2;
private Pet pet;
private Pet petAlteadyAdopted;
List<Adoption> allAdoptions = new ArrayList<>();
List<Adoption> userFilteredAdoptions = new ArrayList<>();
List<Adoption> petFilteredAdoptions = new ArrayList<>();
#Before
public void init() {
MockitoAnnotations.initMocks(this);
user = new AppUser("username","name","lastname","email#gmail.com","pass",ZonedDateTime.now(), Role.ROLE_USER, City.Skopje);
user2 = new AppUser("username1","name","lastname","email#gmail.com","pass",ZonedDateTime.now(), Role.ROLE_USER, City.Skopje);
Center center = new Center("a", City.Bitola,"url");
pet = new Pet("p", Type.DOG,"b", Gender.FEMALE,"d",center, ZonedDateTime.now(),"url",null,false,ZonedDateTime.now());
petAlteadyAdopted = new Pet("p", Type.DOG,"b", Gender.FEMALE,"d",center, ZonedDateTime.now(),"url",null,true,ZonedDateTime.now());
pet.setId(0);
petAlteadyAdopted.setId(1);
adoption1 = new Adoption(ZonedDateTime.now(),ZonedDateTime.now(),Status.ACTIVE,user,pet);
adoption2 = new Adoption(ZonedDateTime.now(),ZonedDateTime.now(),Status.CLOSED,user,pet);
adoption3 = new Adoption(ZonedDateTime.now(),ZonedDateTime.now(),Status.CLOSED,user2,new Pet());
allAdoptions.add(adoption1);
allAdoptions.add(adoption2);
allAdoptions.add(adoption3);
petFilteredAdoptions.add(adoption2);
petFilteredAdoptions.add(adoption1);
userFilteredAdoptions.add(adoption2);
userFilteredAdoptions.add(adoption1);
Mockito.when(this.adoptionRepository.findById(0)).thenReturn(java.util.Optional.of(adoption1));
Mockito.when(this.adoptionRepository.findById(1)).thenReturn(java.util.Optional.of(adoption2));
Mockito.when(this.petRepository.findById(0)).thenReturn(java.util.Optional.of(pet));
Mockito.when(this.petRepository.findById(1)).thenReturn(java.util.Optional.of(petAlteadyAdopted));
Mockito.when(this.appUserRepository.findByUsername("username")).thenReturn(java.util.Optional.of(user));
Mockito.when(this.appUserRepository.findByUsername("username1")).thenReturn(java.util.Optional.of(user2));
Mockito.when(this.adoptionRepository.findAll()).thenReturn(allAdoptions);
Mockito.when(this.adoptionRepository.findAllByPet(pet)).thenReturn(petFilteredAdoptions);
Mockito.when(this.adoptionRepository.findAllByUser(user)).thenReturn(userFilteredAdoptions);
Mockito.when(this.adoptionRepository.findAllByUserAndPet(user,pet)).thenReturn(userFilteredAdoptions);
Mockito.when(this.adoptionRepository.save(Mockito.any(Adoption.class))).thenReturn(adoption1);
this.service = Mockito.spy(new AdoptionServiceImpl(this.adoptionRepository, this.petRepository,this.appUserRepository));
}
As a result, the following test fails, even though it should pass:
#Test
public void searchTest2() {
List<Adoption> adoptionList = this.service.search("",0);
Assert.assertEquals(petFilteredAdoptions,adoptionList);
}
In order to solve this I tried mocking the isPresent() method:
Mockito.when(this.petRepository.findById(0).isPresent()).thenReturn(true);
But I'm getting the following exception:
org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
Boolean cannot be returned by findById() findById() should return
Optional***
If you're unsure why you're getting above error read on. Due to the
nature of the syntax above problem might occur because:
This exception might occur in wrongly written multi-threaded tests. Please refer to Mockito FAQ on limitations of concurrency
testing.
A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub spies -
with doReturn|Throw() family of methods. More in javadocs for Mockito.spy() method.
I also tried the following variation:
Mockito.doReturn(true).when(this.petRepository.findById(0)).isPresent();
But then I got the following exception:
org.mockito.exceptions.misusing.UnfinishedStubbingException:
Unfinished stubbing detected here:
-> at mk.finki.ukim.milenichinja.ServiceTests.AdoptionServiceFilterTests.init(AdoptionServiceFilterTests.java:87)
E.g. thenReturn() may be missing. Examples of correct stubbing:
when(mock.isOk()).thenReturn(true);
when(mock.isOk()).thenThrow(exception);
doThrow(exception).when(mock).someVoidMethod(); Hints:
missing thenReturn()
you are trying to stub a final method, which is not supported
you are stubbing the behaviour of another mock inside before 'thenReturn' instruction is completed
Any ideas how to solve this problem?
In the init method, you are stubbing findById on the mock instance this.petRepository to return a non-mock Optional, which is good. In your new test, you are trying to set a return value for isPresent, which you can't do because Optional is not a mock. If you want to override the behavior per-test, you'll need to stub findById to return an Optional of a different instance. Therefore, this is right, though it appears exactly as it does in init and consequently it can't tell you why your test is failing.
Mockito.when(this.petRepository.findById(0))
.thenReturn(java.util.Optional.of(pet));
Mockito works by creating a mock object that subclasses a class and overrides every method. The overridden method is what interacts with a static (ThreadLocal) infrastructure, allowing you to use when syntax. The important thing here is that when ignores its argument, and instead tries to mock the last interaction that you made with a mock. You can find out more in the SO questions How does mockito when() invocation work? and How do Mockito matchers work?.
When you see this call:
Mockito.when(this.petRepository.findById(0))
.thenReturn(java.util.Optional.of(pet));
Then it works as you've intended:
petRepository is a mock, findById is presumably an overridable method, Mockito records the fact that you've called it with the argument 0.
findById doesn't have any behavior stubbed yet, so it does its default, returning null.
when doesn't care that it just received null, because the null doesn't tell it anything about what methods were called to get the null. Instead it looks back at its most recent record (findById(0)) and returns an object with the thenVerb methods you expect.
You call thenReturn, so Mockito sets up petRepository to return the Optional instance you created and passed in.
But when you try this call:
Mockito.when(this.petRepository.findById(0).isPresent()).thenReturn(true);
Then the most recent interaction isn't isPresent, it's findById, so Mockito assumes you want findById(0) to thenReturn(true) and throws WrongTypeOfReturnValue. Optional is not a mock, so interacting with it doesn't let Mockito record its interaction or replay your behavior. For what it's worth, I also wouldn't advise mocking it: Optional is a final class, and though Mockito has recently added some support for mocking final types, Optional is simple and straightforward enough that it makes more sense to just return the Optional instance you want rather than trying to mock it.
With all that said, your code looks right; as long as PetRepository is an interface I don't see anything about the way your method looks or the way your mocks look that would cause this.petRepository.findById(0) to return an absent Optional. In fact, I don't even see where you would create an absent Optional for it to return, so I can only guess that you are using more real objects in your test than you think you are.
Suppose I have a code like below
public boolean checkForecastIfCityRaining(String name){
result = WeatherAPICallToSomeVendor(name)
if(result = rain)return true; else return false;
}
How would I unit test if the result data will change depending
on what the API vendor is providing?
Would i mock a fixed result of every scenario and then unit test
it like so?
A UNIT test should really only test a single method at a time (Isolate other functionality that might be invoked by that method). My current group might achieve that by writing our function like this:
public class WeatherCheck
{
private ForecastService fs;
public WeatherCheck(ForecastService fs)
{
forecastService = fs;
}
public boolean checkForecastIfCityRaining(String name){
result = forecastService.weatherAPICallToSomeVendor(name)
if(result = rain)return true; else return false;
}
This would allow us to pass a mock forecast service into the constructor. Dependency Injection would be better, but we don't do that yet.
Note: We differentiate between a Unit test and an Integration test. We still might write our Integration tests with Junit, but they have more of a tendency to go out and actually poke the service--this can give you advance warning of a failure. So you might write an integration test for ForecastService that simply calls the weatherAPICallToSomeVendor method of ForecastService and ensures a non-error result (Perhaps no exceptions or doesn't return null...).
I think the function needs to be rewritten as this:
public boolean checkForecastInCityCondition(String city, String condition){
result = WeatherAPICallToSomeVendor(city)
return result == condition;
}
Now you gain the advantage of exposing clients to care about arbitrary conditions and you can enhance with a new API as needed. From a testing perspective you can now safely write tests like this:
public void testRainingInLancaster() throws Exception{
//your code here
}
public void testSnowInRedding() throws Exception{
//your code here
}
And you can determine which pieces need to be mocked for testing.
I am new to writing tests in java, and seem to be unable to test if a method of a class is called.
I am sending metrics to datadog, and want to test in the code if a function of another class was called.
It says I need to mock first, but I couldn't get it to work.
MetricRecorder.java
import com.timgroup.statsd.StatsDClient;
import com.timgroup.statsd.NonBlockingStatsDClient;
import com.google.common.base.Preconditions;
public class MetricRecorder {
private final String namespace;
private final static StatsDClient metrics = new NonBlockingStatsDClient(
"my.prefix",
"localhost",
8125,
new String[] {"tag:value"}
);
public MetricRecorder(String namespace) {
Preconditions.checkNotNull(namespace);
this.namespace = namespace;
}
public void inc(String metricName) {
this.inc(metricName, 1);
}
public void inc(final String metricName, final long value) {
Preconditions.checkNotNull(metricName);
try {
metrics.recordHistogramValue(MetricRecorder.name(namespace, metricName), value);
} catch (Exception e) {
logger.warn("Unable to record metric {} due to :", metricName, e);
}
}
...
}
MetricRecorderTest.java
public class MetricsRecorderTest {
#Test
public void metricsRecorderTest() {
MetricRecorder recorder = new MetricRecorder("dev");
recorder.inc("foo", 1);
verify(recorder.metrics, times(1)).recordHistogramValue(eq("dev.foo"), 1);
}
}
When I run the test I get this => org.mockito.exceptions.misusing.NotAMockException:
Argument passed to verify() is of type NonBlockingStatsDClient and is not a mock!
Any idea of how I should be testing if recordHistogramValue was called, and if so with what arguments?
Since it looks like StatsDClient is an interface of some kind, it would make your testing effort easier to simply inject this dependency into your object. Even if you're not using an IoC container like Spring or Guice, you can still somewhat control this simply by passing an instance of it in through the constructor.
public MetricRecorder(String namespace, StatsDClient client) {
Preconditions.checkNotNull(namespace);
Preconditions.checkNotNull(client);
this.namespace = namespace;
this.client = client;
}
This will make your testing simpler since all you realistically need to do is mock the object passed in during test.
Right now, the reason it's failing is because you're newing up the instance, and Mockito (in this current configuration) isn't equipped to mock the newed instance. In all honesty, this set up will make testing simpler to conduct, and you should only need your client configured in one area.
#RunWith(MockitoJUnitRunner.class)
public class MetricsRecorderTest {
#Test
public void metricsRecorderTest() {
StatsDClient dClientMock = Mockito.mock(StatsDClient.class);
MetricRecorder recorder = new MetricRecorder("dev", dClientMock);
recorder.inc("foo", 1);
verify(recorder.metrics).recordHistogramValue(eq("dev.foo"), 1);
}
}
You are getting things wrong here. You don't use a mocking framework to test your "class under test".
You use the mocking framework to create mocked objects; which you then pass to your "class under test" within a test case. Then your "code under test" calls methods on the mocked object; and by controlling returned values (or by verifying what happens to your mock); that is how you write your testcases.
So, your testcase for a MetricRecorder doesn't mock a MetricRecorder; it should mock the StatsDClient class; and as Makoto suggests; use dependency injection to put an object of that class into MetricRecorder.
Besides: basically writing "test-able" code is something that needs to be practiced. I wholeheartedly recommend you to watch these videos if you are serious about getting in this business. All of them; really (worth each second!).
How can I mark a test as an expected failure in JUnit 4?
In this case I want to continue to run this test until something is patched upstream. Ignoring the test goes a little too far, as then I might forget about it. I may be able to add an #expected annotation and catch the exception thrown by assertThat, but that also seems to lie about the expected behavior.
Here's what my current test looks like:
#Test
public void unmarshalledDocumentHasExpectedValue()
{
doc = unmarshaller.unmarshal(getResourceAsStream("mydoc.xml"));
final ST title = doc.getTitle();
assertThat(doc.getTitle().toStringContent(), equalTo("Expected"));
}
That assert should succeed, but because of an upstream bug it doesn't. Yet, that test is correct; it should succeed. Virtually all the alternatives that I've found are misleading. Right now I think #Ignore("This test should pass once fixed upstream") is my best bet, but I still have to remember to come back to it. I'd prefer that the test run.
In Python I can use the expectedFailure decorator:
class ExpectedFailureTestCase(unittest.TestCase):
#unittest.expectedFailure
def test_fail(self):
self.assertEqual(1, 0, "broken")
With Qt's QTestLib in C++, you can use QEXPECT_FAIL:
QEXPECT_FAIL("", "Will be fixed next version", Continue);
QCOMPARE(i, 42);
In both cases above, the unit test runs which is what I'm hoping to have happen. Am I missing something in JUnit?
I'm not quite getting the specifics of your scenario, but here's how I generally test for expected failure:
The slick new way:
#Test(expected=NullPointerException.class)
public void expectedFailure() {
Object o = null;
o.toString();
}
for older versions of JUnit:
public void testExpectedFailure() {
try {
Object o = null;
o.toString();
fail("shouldn't get here");
}
catch (NullPointerException e) {
// expected
}
}
If you have a bunch of things that you want to ensure throw an exception, you may also want to use this second technique inside a loop rather than creating a separate test method for each case. If you were just to loop through a bunch of cases in a single method using expected, the first one to throw an exception would end the test, and the subsequent cases wouldn't get checked.
What about explicitly expecting an AssertionError?
#Test(expected = AssertionError.class)
public void unmarshalledDocumentHasExpectedValue() {
// ...
}
If you're reasonably confident that only the JUnit machinery within the test would raise AssertionError, this seems as self-documenting as anything.
You'd still run the risk of forgetting about such a test. I wouldn't let such tests into version control for long, if ever.
I'm assuming here that you want the test to pass if your assert fails, but if the assert succeeds, then the test should pass as well.
The easiest way to do this is to use a TestRule. TestRule gives the opportunity to execute code before and after a test method is run. Here is an example:
public class ExpectedFailureTest {
public class ExpectedFailure implements TestRule {
public Statement apply(Statement base, Description description) {
return statement(base, description);
}
private Statement statement(final Statement base, final Description description) {
return new Statement() {
#Override
public void evaluate() throws Throwable {
try {
base.evaluate();
} catch (Throwable e) {
if (description.getAnnotation(Deprecated.class) != null) {
// you can do whatever you like here.
System.err.println("test failed, but that's ok:");
} else {
throw e;
}
}
}
};
}
}
#Rule public ExpectedFailure expectedFailure = new ExpectedFailure();
// actually fails, but we catch the exception and make the test pass.
#Deprecated
#Test public void testExpectedFailure() {
Object o = null;
o.equals("foo");
}
// fails
#Test public void testExpectedFailure2() {
Object o = null;
o.equals("foo");
}
}
First, note that the first method is marked as #Deprecated. I'm using this as a marker for the method for which I want to ignore any assertion failures. You can do whatever you like to identify the methods, this is just an example.
Next, in the ExpectedFailure#apply(), when I do the base.evaluate(), I'm catching any Throwable (which includes AssertionError) and if the method is marked with the annotation #Deprecated, I ignore the error. You can perform whatever logic you like to decide whether you should ignore the error or not, based on version number, some text, etc. You can also pass a dynamically determined flag into ExpectedFailure to allow it to fail for certain version numbers:
public void unmarshalledDocumentHasExpectedValue() {
doc = unmarshaller.unmarshal(getResourceAsStream("mydoc.xml"));
expectedFailure.setExpectedFailure(doc.getVersionNumber() < 3000);
final ST title = doc.getTitle();
assertThat(doc.getTitle().toStringContent(), equalTo("Expected"));
}
For further examples, see ExternalResource, and ExpectedException
Ignoring an expected failure test rather than passing it
If you want to mark you tests as Ignored rather than Success, it becomes a bit more complex, because tests are ignored before they are executed, so you have to retrospectively mark a test as ignored, which would involve constructing your own Runner. To give you a start, see my answer to How to define JUnit method rule in a suite?. Or ask another question.
One option is mark the test as #Ignore and put text in there that is a bug perhaps and awaiting a fix. That way it won't run. It will then become skipped. You could also make use of the extensions to suit your need in a potentially different way.
I've taken Matthew's answer a step further and actually implemented an #Optional annotation you could use instead of the #Deprecated marker annotation he mentions in his answer. Although simple, I'll share the code with you, maybe it's of help for someone:
#Target(ElementType.METHOD)
#Retention(RetentionPolicy.RUNTIME)
#Documented
public #interface Optional {
/**
* Specify a Throwable, to cause a test method to succeed even if an exception
* of the specified class is thrown by the method.
*/
Class<? extends Throwable>[] exception();
}
With a simple alteration of Matt's ExpectedFailure class:
public class ExpectedFailure implements TestRule {
#Override
public Statement apply(final Statement base, final Description description) {
return statement(base, description);
}
private Statement statement(final Statement base, final Description description) {
return new Statement() {
#Override
public void evaluate() throws Throwable {
try {
base.evaluate();
} catch (Throwable e) {
// check for certain exception types
Optional annon = description.getAnnotation(Optional.class);
if (annon != null && ArrayUtils.contains(annon.exception(), e.getClass())) {
// ok
} else {
throw e;
}
}
}
};
}
}
You can now annotate your test method with #Optional and it will not fail, even if the given type of exception is raised (provide one or more types you would like the test method to pass):
public class ExpectedFailureTest {
#Rule public ExpectedFailure expectedFailure = new ExpectedFailure();
// actually fails, but we catch the exception and make the test pass.
#Optional(exception = NullPointerException.class)
#Test public void testExpectedFailure() {
Object o = null;
o.equals("foo");
}
}
[UPDATE]
You could also rewrite your tests using JUnit's org.junit.Assume instead of the tradtional org.junit.Assert, if you want your tests to pass even if the assumption does not hold.
From Assume's JavaDoc:
A set of methods useful for stating assumptions about the conditions in which a test is meaningful.A failed assumption does not mean the code is broken, but that the test provides no useful information. The default JUnit runner treats tests with failing assumptions as ignored.
Assume is available since JUnit 4.4
Use mocked upstream class if possible. Stub it with correct result. Optionally, replace mock with real object after bug is fixed.