We Keep Coding

Using SoftAssert with Parallel Execution in TestNG produces unexpected results

I am using the rest-assured library to test our REST api that deals with data on sports. In short I have 2 different #Test methods to call per sport, one #Test method to make multiple GET requests to gather all athlete image urls and store in a static ArrayList, and the other method to instantiate a SoftAssert object and actually call all of the url's in a for loop and soft assert a 200 response code. I then do a assertAll() at the end of the 2nd test method.
For example - I have a #Test getSoccerAthletes() which gathers all the urls from the response, the method will repeat until all athlete urls are gathered as the response is limited to 250 athletes at a time. After this method finishes, then the 2nd #Test method for Soccer will execute, it is named testSoccerAthletes() and you can see that it uses dependsOnMethods. Below is the setup.
#Test(priority = 1)
public static void getSoccerAthletes() {
baseURI = "https://XXXXX";
basePath = "XXXXX";
Response res = given().queryParam("apikey", "XXXXXXXX")
.queryParam("top", "250")
.queryParam("skip", soccerSkip)
.when().get();
Assert.assertEquals(res.statusCode(), 200);
JsonPath jPath = res.jsonPath();
System.out.println("Soccer: currentPageStart/totalCount " + jPath.getInt("currentPageStart")
+ "/" + jPath.getInt("totalCount"));
if (soccerSkip == 0) {
allSoccerAthletes = jPath.get("page.links.headshots.full");
} else {
ArrayList<String> athletes = jPath.get("page.links.headshots.full");
allSoccerAthletes.addAll(athletes);
}
if (jPath.getInt("currentPageStart") + 250 < jPath.getInt("totalCount")) {
soccerSkip += 250;
getSoccerAthletes();
}
}
#Test(priority = 2, dependsOnMethods = {"getSoccerAthletes"})
public static void testSoccerAthletes() {
SoftAssert softAssert = new SoftAssert();
for (int i = 0; i < allSoccerAthletes.size(); i++) {
String url = allSoccerAthletes.get(i);
System.out.println("Soccer Athlete: " + i + "/" + allSoccerAthletes.size());
Response res = when().head(url);
softAssert.assertEquals(res.statusCode(), 200, "Failed url: " + url);
if (i == allSoccerAthletes.size() - 1)
allSoccerAthletes.clear();
}
softAssert.assertAll();
}
I am seeing varying results, some of which have failures from the #Test testXXXXAthletes methods mixed up. This first suggestion online was to instantiate a SoftAssert object in each #Test method (which I currently am for every testXXXXAthletes method) so that can't be it.
I am starting to lean toward there are thread safety issues, but I am not really sure how to move forward with a solution. Reasons I believe there are thread safety issues are from some articles I have looked into, but do not fully understand. Articles -->
https://learn2automate.wordpress.com/2017/07/13/parallel-testng-soft-assertions-the-right-way/,
Retrieve test name on TestNG,
https://github.com/rest-assured/rest-assured/issues/1420
Any help in resolving my issues would be greatly appreciated! Btw I have also looked into using the #DataProvider annotations and that brings me to other questions about the structure of these tests. The getXXXAthletes methods are acting as data providers to the testXXXAthletes methods, but they have the dependsOnMethods attribute. Should I be using both DataProvider and dependsOnMethods, or one over the other?

Further research confirmed my theory about the thread-safety issues. I was able to conclude that the issue has nothing to do with SoftAssert and everything to do with rest-assured not being thread-safe. That information can be found here --> https://github.com/rest-assured/rest-assured/pull/851
I was able to find a thread-local branch in which someone kindly did the work to make rest-assured thread-safe. I downloaded 2 files RestAssuredThreadLocal and RestAssuredThreadLocalImpl which can be seen here --> https://github.com/rest-assured/rest-assured/commit/3307ba6c79c5547e88cea286d38e5c8a6d679229
After downloading those 2 files, there were some errors that needed resolved and some deprecated methods needed replaced. After that I was able to successfully run my rest-assured tests in parallel with TestNG with the correct results.

How to test that a method is called by Guava Cache only once

I'm trying to test that a cache I created is working properly, and duplicate calls to a method aren't actually called multiple times, but loaded from my cache.
I'm using LoadingCache from Google Guava to accomplish this.
So my naive approach was to create a spy, and verify that the method was called once. I then figured out that because spy() is a decorator, I can only see if the method was called on that object. Since the method is being called by LoadingCache, my spy cannot verify it.
How do I best test that my cache is being used properly?
(note: I could make a LoadingCache dependency, and check that the correct method is called, but then I don't know if my cache is working. Maybe the hash is calculated in a way I didn't anticipate, and so it actually is calling the method every time. I want to actually see the effect)
private final LoadingCache<TaskDetails, byte[]> cache;
...
cache = CacheBuilder.newBuilder()
.maximumSize(cacheSize)
.expireAfterAccess(Duration.ofMinutes(cacheDurationInMinutes))
.build(CacheLoader.from(this::doTask));
public byte[] taskCaller(...) {
...
return cache.getUnchecked();
}
public byte[] doTask(...) {
if(something.doSomething() ... ) {
...
}
First test attempt doesn't work, Mockito says: Wanted but not invoked (the method was "never called", since the spy doesn't know the cache called it)
#Test
public void test() {
// given
TaskService spy = spy(
new TaskService(...)
);
// when
for (int i = 0; i < 3; i++) {
spy.taskCaller(...);
}
// then
//if caching is working, we should only do the real work once
verify(spy, times(1)).doTask(any(TaskDetails.class));
}
Second attempt: NullPointerException on first statement in the method ( something.doSomething() ), I'm not completely sure why, but I'm convinced for the same reason as before
#Test
public void test() {
// given
TaskService spy = spy(
new TaskService(...)
);
final int[] counter = {0};
when(spy.doTask(any())).thenAnswer(invocation -> {
counter[0]++;
return new byte[]{0,0,0,0};
});
// when
int run = 3;
for (int i = 0; i < run; i++) {
spy.taskCaller(...);
}
// then
//if caching is working, we should only do the real work once
assertThat(run).isEqualTo(counter[0]);
verify(spy, times(1)).doTask(any(TaskDetails.class));
}

Two ideas:
Enable cache stats recording, interact with the cache, get the cache stats, then verify that the load count (and possibly other stats) are what you expect.
Make your CacheLoader a separate named class rather than a method reference, an instance of which is passed to CacheBuilder.build(), and arrange for that instance to be a mock in your test. You can then verify method call counts and arguments on your mock loader.

I'm using LoadingCache from Google Guava to accomplish this.
That's the point. If you trust Guava enough, why don't you trust that its tests work?
Did you write the cache? No, you didn't. Then don't test it as part of unit tests: only as part of integration tests, with the rest of the whole application.
Just test that the cache-loading method is called when you invoke your cache-using method.
If you want to test that you have a cache, perform two calls on the general method, and verify that the cache-loading method is called only once.

How do I write a Mockito test case?

Hi I was writing a test case for
#Override
public SnapDocument updateFlights(SnapDocument snapDocument, FlightListingResponse flightsResponse) {
HashMap<String, FaresInfo> outboundFlights = new HashMap<>();
HashMap<String, FaresInfo> inboundFlights = new HashMap<>();
Objects.requireNonNull(flightsResponse.getOutboundFlights()).getFlights().forEach(faresInfo -> outboundFlights.put(faresInfo.getJourneySellKey(), faresInfo));
Objects.requireNonNull(flightsResponse.getReturnFlights()).getFlights().forEach(faresInfo -> inboundFlights.put(faresInfo.getJourneySellKey(), faresInfo));
snapDocument.setOutboundFlights(outboundFlights);
snapDocument.setInboundFlights(inboundFlights);
return snapDocument;
}
This is the test case I have written
public void updateFilghtsSuccess(){
when(snapService.getSnapDocument(any())).thenReturn(snapDocument);
when(snapService.updateFlights(snapDocument, flightListingResponse)).thenReturn(snapDocument);
verify(snapService).updateFlights(snapDocument, flightListingResponse);
}
The build is failing, How to write the test cases for this so that I can improve the coverage of my code?

when, given and verify should only be used when dealing with mocks.
As your code works with plain old java objects and no other external components are called, you do not need them.
Just construct the pojos to pass to the method and assert that the pojo that comes back out contains the correct values. You don't need mockito for that, as the test does not require mocks.
#Test
void testUpdateFlights() {
// Arrange
SnapService snapService = new SnapServiceImpl();
SnapDocument sd = new SnapDocument();
FlightListingResponse response = new FlightListingResponse();
response.setOutboundFlights(...);
response.setReturnFlights(...);
// Act
SnapDocument result = snapService.updateFlights(sd, response);
// Assert
assertThat(result.getOutboundFlights()).contains(...);
}
You do not need them now, but when would you need a mock? If your code calls a component that is not under test and you don't want that the actual implementation of it is executed. Unit tests are to test 1 unit of work. All other external components should be mocked out.

EasyMock : java.lang.IllegalStateException: 1 matchers expected, 2 recorded

I am having a problem with EasyMock 2.5.2 and JUnit 4.8.2 (running through Eclipse). I have read all the similar posts here but have not found an answer. I have a class containing two tests which test the same method. I am using matchers.
Each test passes when run alone.
The first test always passes - this is true if I switch the order of the tests in the file.
Here is a simplified version of the test code:
private Xthing mockXthing;
private MainThing mainThing;
#Before
public void setUp() {
mockXthing = EasyMock.createMock(Xthing.class);
mainThing = new MainThing();
mainThing.setxThing(mockXthing);
}
#After
public void cleanUp() {
EasyMock.reset(mockXthing);
}
#Test
public void testTwo() {
String abc = "abc";
EasyMock.expect(mockXthing.doXthing((String) EasyMock.anyObject())).andReturn(abc);
EasyMock.replay(mockXthing);
String testResult = mainThing.testCallingXthing((Long) EasyMock.anyObject());
assertEquals("abc", testResult);
EasyMock.verify(mockXthing);
}
#Test
public void testOne() {
String xyz = "xyz";
EasyMock.expect(mockXthing.doXthing((String) EasyMock.anyObject())).andReturn(xyz);
EasyMock.replay(mockXthing);
String testResult = mainThing.testCallingXthing((Long) EasyMock.anyObject());
assertEquals("xyz", testResult);
EasyMock.verify(mockXthing);
}
The second (or last) test always fails with the following error:
java.lang.IllegalStateException: 1 matchers expected, 2 recorded
Any insight to this would be greatly appreciated.
Thanks,
Anne

I haven't looked meticulously closely yet, but this looks suspect:
String testResult = mainThing.testCallingXthing((Long) EasyMock.anyObject());
anyObject() is a matcher and you're calling it after the replay. It's not used to produce any object. It's used to instruct EasyMock to allow any object. EasyMock is detecting that extra matcher but it is not harmful until the second test. At that point, the number of matchers that EasyMock has recorded but hasn't yet used (2) doesn't line up with the number of parameters expected for the second doXthing call (1).
You should be passing in real parameters to testCallingXthing (or a mock that is in replay mode). Try passing in null directly, or a real value like 2.

for me this failure (in my case 2 matchers expected, 4 recorded.) meant "you are mixing easymock and mockito in the same unit test, so accidentally calling easymock's notNull() method for a mockito argument. Which causes the failure but only if the tests are run in a certain order.

Try:
String testResult = mainThing.testCallingXthing(eq(EasyMock.anyLong()));
There are more refined matchers than anyObject(). These allow you to make type-based assertions about collaborators.
From the EasyMock documentation:
eq(X value)
Matches if the actual value is equals the expected value. Available for all primitive types and for objects.
anyBoolean(), anyByte(), anyChar(), anyDouble(), anyFloat(), anyInt(), anyLong(), anyObject(), anyShort()

You should reset mock after each test method to get rid of this problem. Adding below code will solve this problem.
#After
public void after(){
EasyMock.reset(mockXthing)
}

PowerMock, mockito, verify static method

I'm trying to get PowerMock to work with mockito, and I'm following the documentation here: http://code.google.com/p/powermock/wiki/MockitoUsage13.
To simplify a bit, lets say that I have a static method:
StaticObj.put(String key, String val) { ... }
And the class to be tested does something like this:
public class ClassToTest {
public void doSomething(Params p) {
if (StringUtils.isNotBlank(p.getK()) StaticObj.put("k1", p.getK());
if (StringUtils.isNotBlank(p.getX()) StaticObj.put("x1", p.getX());
}
}
In my unit test I'd like to verify that StaticObj.put is called for K and X when they are not blank or null, so I do something like this:
public void testNormalCase() {
// assume that mocking setup for statics already happened in some #Before function..
Params params = new Params("k", "x");
ClassToTest classToTest = new ClassToTest();
classToTest.doSomething(params);
// now I want to verify:
PowerMockito.verifyStatic(times(1));
StaticObj.put("k1", "k1");
PowerMockito.verifyStatic(times(1));
StaticObj.put("x1", "x");
}
This works, and it's what I'd expect. What doesn't work, is if I comment out the verification for K, then the verification for X fails! The error message indicates that ("x1", "x") is expected but got ("k1", "k"). Why is this? Am I not coding this correctly?
Also it leads me to believe that the following type of test, which passes, might pass for the wrong reason entirely:
public void testOtherCase() {
// assume that mocking setup for statics already happened in some #Before function..
Params params = new Params("k", null);
ClassToTest classToTest = new ClassToTest();
classToTest.doSomething();
// now I want to verify:
PowerMockito.verifyStatic(never());
StaticObj.put(eq("x1"), anyString());
}
E.g. I wonder if powermock sees "k1", decides that "x1" was never called, and passes. (?)
To state it generally, I have a static method that is called N times (where N changes depending on the input params). And I want to verify that it was called in the correct cases (which can be determined by input params). It seems like powermock doesn't handle this well, unless I misunderstand.
Thanks for any ideas!

I read this question and the issue carefully but not sure if I understood them clearly - From my understanding, it's correct that powermock raise the exception when you pass k and x but only verify k.
Because you are mocking the static method StaticObj.put, when you pass parameter k and x and verify it with
PowerMockito.verifyStatic(times(1));
StaticObj.put("k1", "k1");
PowerMockito.verifyStatic(times(1));
StaticObj.put("x1", "x");
This should work. And when you verify parameter k and x with verification for k is commented out.
// PowerMockito.verifyStatic(times(1));
// StaticObj.put("k1", "k1");
PowerMockito.verifyStatic(times(1));
StaticObj.put("x1", "x");
Powermock will get the invocation with put("k1"...) first apparently, so the verification of x will raise an error. Your verification process is sequenced.

I don't know as of which version, but PowerMockito.verifyStatic(VerificationMode) is deprecated. Just wanted to point that out to anyone else finding this years after the last post.