We Keep Coding

TestNG reporting issues due to multi-threading

I have a testNG framework integrated with testrails. Because of the restrictions around the testrail api, the framework was designed to collate test result in bulk and upload them all at once when the test run is complete.
To do so, I've created a BaseTest class that provides a variable id that each test method can set itself to match a corresponding test case in test rails. Once the test method assigns this variable, we pack it in to the result object:
public abstract class BaseTest {
protected static final ThreadLocal<Integer> testrailIds = new ThreadLocal();
protected int testRailCaseId = -1;
//TODO: set down a clear and strong process for handling environment/domain. Once that is in place, report it
// loudly so it is clear where the tests ran
protected String baseUrl = "";
protected static final Logger LOG = Logger.getLogger(BaseTest.class);
/**
* records the case ID stored at the individual test level inside the result set for the
* test so that we can access it later when reporting our results
* #param result the result context object
*/
#AfterMethod(alwaysRun = true)
public void afterMethod(ITestResult result) {
if(testRailCaseId == -1) {
LOG.warn("NO CASE ID HAS BEEN SET FOR THIS TEST");
}
result.setAttribute("case_id", testrailIds.get());
}
Once all tests have executed we build a request object in the afterSuite method and pipe both the test cases and test results to testrail.
for(ISuiteResult suiteResult: suite.getResults().values()) {
ctx = suiteResult.getTestContext();
for (ITestResult result : ctx.getPassedTests().getAllResults()) {
cases.add((int) result.getAttribute("case_id"));
JSONObject resultJson = new JSONObject();
resultJson.put("case_id", result.getAttribute("case_id"));
resultJson.put("status_id", 1);
payload.add(resultJson);
}
for (ITestResult result : ctx.getFailedTests().getAllResults()) {
cases.add((int) result.getAttribute("case_id"));
JSONObject resultJson = new JSONObject();
resultJson.put("case_id", result.getAttribute("case_id"));
resultJson.put("status_id", 5);
payload.add(resultJson);
}
}
// get a clean instance of the api
TestRailApi tr = new TestRailApi();
//now dump that arraylist into a json param and add it to the test run
tr.updateRun(runId, cases);
//once the test run has been created, clean up again and build the results request
tr = new TestRailApi();
tr.addResultsForCases(runId, payload);
the testRailCaseId is set at the beginning of each test method with a simple assignment
this.testRailCaseId = 491;
or
testrailIds.set(489);
This worked fine until we started using multi-threading. Now, the value of testRaidCaseId is being overwritten by parallel tests, resulting in smaller result sets than expected.
I've been attempting to manage the threads through a ThreadLocal (as seen in the code above), but have been unsuccessful so far -- values I try to set in the before method or in the tests are coming up empty in the after methods.
The test methods themselves are fine, my only struggle is with shared content being passed into them from the parent.
Anyone have any guidance for how to manage my variables across the baseTest through the test methods to ensure my various ids don't clobber each other?
Sample test case:
#Test
#Parameters({ "domain", "username", "password" })
public void logInAuthEmptyToken(#Optional("http://REDACTED.com") String domain, String username, String password) {
this.testRailCaseId = 385;
Map<String, String> loginInfo = BaseAuthTests.login(domain, username, password);
AuthServiceApi auth = new AuthServiceApi(domain);
auth.addAuthTokens("", loginInfo.get("arrival_key"), loginInfo.get("profile_key"));
auth.executeLogin(400);
}

Mockito mocking a new instance call with parameters

For school purposes I am creating an application that's working with a stock API.
I am trying to write a test for a method that gets all the stock data of the last 10 years. Instead of actually getting all that data, I want to throw an exception.
The method I Want to test:
#Override
public List<StockData> getAllTeslaStockData() throws AlphaVantageException {
List<StockData> stockData;
AlphaVantageConnector apiConnector = new AlphaVantageConnector(APIKEY, TIMEOUT);
TimeSeries stockTimeSeries = new TimeSeries(apiConnector);
try {
Daily responseDaily = stockTimeSeries.daily("TSLA", OutputSize.FULL);
stockData = responseDaily.getStockData();
} catch (AlphaVantageException e) {
LOGGER.log(Level.SEVERE, "something went wrong: ", e);
throw e;
}
return stockData;
}
The stockTimeSeries.daily(....) call can throw the AlphaVantageException.
I've mocked the TimeSeries class like this:
TimeSeries stockTimeSeries = mock(TimeSeries.class);
In my test class I want to mock this call, and return an exception instead of actual data.
when(stockTimeSeries.daily("TSLA", OutputSize.FULL)).thenThrow(new AlphaVantageException("No stock data available"));
Regardless of how I am trying to mock this bit of code, it'll never throw the exception. It will always just execute the code, and return valid stock data, instead of throwing the exception like i've tried to do.
How can I mock this bit of code, so that itll throw the exception I am expecting for my tests.
The AlphaVantageConnector, TimeSeries and Daily classes are part of a library used to access the stock API, so I can't change these classes.
I am using JUnit 4.12 and Mockito to try and achieve this.

You can use thenThrow() method. Below is the example
#Test(expected = NullPointerException.class)
public void whenConfigNonVoidRetunMethodToThrowEx_thenExIsThrown() {
MyDictionary dictMock = mock(MyDictionary.class);
when(dictMock.getMeaning(anyString()))
.thenThrow(NullPointerException.class);
dictMock.getMeaning("word");

The TimeSeries object is created in the method itself, so you can't mock it - mocking is intended to mock members.
What you can do is to do something like
class YourClass {
private Supplier<TimeSeries> seriesCreator = () -> {
return new TimeSeries(new AlphaVantageConnector(APIKEY, TIMEOUT));
}
which you use to create the series in your method
#Override
public List<StockData> getAllTeslaStockData() throws AlphaVantageException {
TimeSeries stockTimeSeries = seriesCreator.get();
Now you can mock that Supplier.
#Mock Supplier<TimeSeries> seriesCreatorMock;
#InjectMocks MyClass sut;
and in your test
#Test(expected = AlphaVantageException.class)
void testException() {
when(seriesCreatorMock.get()).thenThrow(new AlphaVantageException());
sut.getAllTeslaStockData()
}
EDIT: as suggested by Angkur in the comments, the clean way would be to
class SeriesCreator implements Supplier<TimeSeries> {
public TimeSeries get() {
return new TimeSeries(new AlphaVantageConnector(APIKEY, TIMEOUT));
}
}
class YourClass {
private Supplier<TimeSeries> seriesCreator = new SeriesCreator();
// ...

The code in the main class is creating a new instance of TimeSeries which it will use every time this method is called, so the mocked TimeSeries object is not getting used at all.
TimeSeries stockTimeSeries = new TimeSeries(apiConnector); // --> This is not getting mocked
try {
Daily responseDaily = stockTimeSeries.daily("TSLA", OutputSize.FULL);
stockData = responseDaily.getStockData();
}
You should create another method in your class (or even a separate class if it better satisfies the SOLID principles) which returns you the TimeSeries object. Something like:-
<access modifier> TimeSeries getTimeSeries(...) {
}
and then this method should be mocked in the Junit, and when mocked, it should return the Mocked TimeSeries reference (which is created in TimeSeries stockTimeSeries = mock(TimeSeries.class); ). You would need to use .spy() on the main class (unless you are using a different class to create TimeSeries object) in order to be able to mock the specific method getTimeSeries() but not the others.
MainClass mainObject = Mockito.spy(new MainClass());
Mockito.when(mainObject.getTimeSeries()).thenReturn(stockTimeSeries);
Then, the method call stockTimeSeries.daily() will get actually mocked by your existing code :
when(stockTimeSeries.daily("TSLA", OutputSize.FULL)).thenThrow(new AlphaVantageException("No stock data available"));
NOTE: you should also consider using .anyString() style methods provided by Mockito API while mocking.

Mockito - Test if a method of a class is called

I am new to writing tests in java, and seem to be unable to test if a method of a class is called.
I am sending metrics to datadog, and want to test in the code if a function of another class was called.
It says I need to mock first, but I couldn't get it to work.
MetricRecorder.java
import com.timgroup.statsd.StatsDClient;
import com.timgroup.statsd.NonBlockingStatsDClient;
import com.google.common.base.Preconditions;
public class MetricRecorder {
private final String namespace;
private final static StatsDClient metrics = new NonBlockingStatsDClient(
"my.prefix",
"localhost",
8125,
new String[] {"tag:value"}
);
public MetricRecorder(String namespace) {
Preconditions.checkNotNull(namespace);
this.namespace = namespace;
}
public void inc(String metricName) {
this.inc(metricName, 1);
}
public void inc(final String metricName, final long value) {
Preconditions.checkNotNull(metricName);
try {
metrics.recordHistogramValue(MetricRecorder.name(namespace, metricName), value);
} catch (Exception e) {
logger.warn("Unable to record metric {} due to :", metricName, e);
}
}
...
}
MetricRecorderTest.java
public class MetricsRecorderTest {
#Test
public void metricsRecorderTest() {
MetricRecorder recorder = new MetricRecorder("dev");
recorder.inc("foo", 1);
verify(recorder.metrics, times(1)).recordHistogramValue(eq("dev.foo"), 1);
}
}
When I run the test I get this => org.mockito.exceptions.misusing.NotAMockException:
Argument passed to verify() is of type NonBlockingStatsDClient and is not a mock!
Any idea of how I should be testing if recordHistogramValue was called, and if so with what arguments?

Since it looks like StatsDClient is an interface of some kind, it would make your testing effort easier to simply inject this dependency into your object. Even if you're not using an IoC container like Spring or Guice, you can still somewhat control this simply by passing an instance of it in through the constructor.
public MetricRecorder(String namespace, StatsDClient client) {
Preconditions.checkNotNull(namespace);
Preconditions.checkNotNull(client);
this.namespace = namespace;
this.client = client;
}
This will make your testing simpler since all you realistically need to do is mock the object passed in during test.
Right now, the reason it's failing is because you're newing up the instance, and Mockito (in this current configuration) isn't equipped to mock the newed instance. In all honesty, this set up will make testing simpler to conduct, and you should only need your client configured in one area.
#RunWith(MockitoJUnitRunner.class)
public class MetricsRecorderTest {
#Test
public void metricsRecorderTest() {
StatsDClient dClientMock = Mockito.mock(StatsDClient.class);
MetricRecorder recorder = new MetricRecorder("dev", dClientMock);
recorder.inc("foo", 1);
verify(recorder.metrics).recordHistogramValue(eq("dev.foo"), 1);
}
}

You are getting things wrong here. You don't use a mocking framework to test your "class under test".
You use the mocking framework to create mocked objects; which you then pass to your "class under test" within a test case. Then your "code under test" calls methods on the mocked object; and by controlling returned values (or by verifying what happens to your mock); that is how you write your testcases.
So, your testcase for a MetricRecorder doesn't mock a MetricRecorder; it should mock the StatsDClient class; and as Makoto suggests; use dependency injection to put an object of that class into MetricRecorder.
Besides: basically writing "test-able" code is something that needs to be practiced. I wholeheartedly recommend you to watch these videos if you are serious about getting in this business. All of them; really (worth each second!).

Mockito mocking won't work for this method. Am I doing something wrong?

I'm current using mockito 1.8.4 in this spring mvc application. Here is the code for the class/method I am trying to test.
public class CompleteTaskController implements IController {
public static Logger log = Logger.getLogger(CompleteTaskController.class);
#Override
public void handle() {
GUIFactory gf = new GUIFactory();
IDatabasePullListOfUsers pull = new OraclePullListOfUsers();
IDatabaseUserManagement manage = OracleUserManagement.getInstance();
gf.makeGUI("completeTask", pull.pullAssignedRequests(GUIFactory.userLoggedIn));
manage.completeTask(gf.getRequestID(), GUIFactory.userLoggedIn);
gf.makeCustomGUI("Task has been completed");
log.fatal(GUIFactory.userLoggedIn + " has completed the task of request id " + gf.getRequestID());
gf.makeGUI("adminpanel");
}
}
so far, all the tests pass BUT the last one - which I just can't seem to understand why. Here is the code for my tests:
public class CompleteTaskControllerTest {
#Test
public void testHandleCallsMakeGUIAndPassesItPullAssignedRequestsAndAString(){
CompleteTaskController mockCtc = mock(CompleteTaskController.class);
GUIFactory mockGf = mock(GUIFactory.class);
IDatabasePullListOfUsers mockPull = mock(OraclePullListOfUsers.class);
mockCtc.handle();
verify(mockGf).makeGUI("test", mockPull.pullAssignedRequests("test"));
}
#Test
public void testHandleCallsCompleteTaskAndPassesItGetRequestIDAndAString(){
CompleteTaskController mockCtc = mock(CompleteTaskController.class);
IDatabaseUserManagement mockManage = mock(OracleUserManagement.class);
GUIFactory mockGf = mock(GUIFactory.class);
mockCtc.handle();
when(mockGf.getRequestID()).thenReturn(1);
verify(mockManage).completeTask(mockGf.getRequestID(),"Test");
}
#Test
public void testHandleCallsMakeCustomGUIAndPassesItAString(){
CompleteTaskController mockCtc = mock(CompleteTaskController.class);
GUIFactory mockGf = mock(GUIFactory.class);
mockCtc.handle();
verify(mockGf).makeCustomGUI("test");
}
}
Sorry for throwing all this code at you - the only test I am having issues with is the third test - which is trying to mock the gf.makeCustomGUI("Task has been completed") method!

With the statement
verify(mockGf).makeCustomGUI("test");
are you trying to verify whether the "makeCustomerGUI" has been called with argument "test". And if that is your requirement, you can do that as below:
GUIFactory mockGf = mock(GUIFactory.class);
mockGf.makeCustomerGUI("test");
verify(mockGf).makeCustomGUI("test");
Once your mock object is created, Mockito will remember all invocations on it. So you can selectively verify the invocations.

How do I create a mock object for Spring's WebServiceTemplate?

I have a class which calls out to an existing web service. My class properly handles valid results as well as fault strings generated by the web service. The basic call to the web service looks something like this (although this is simplified).
public String callWebService(final String inputXml)
{
String result = null;
try
{
StreamSource input = new StreamSource(new StringReader(inputXml));
StringWriter output = new StringWriter();
_webServiceTemplate.sendSourceAndReceiveToResult(_serviceUri, input, new StreamResult(output));
result = output.toString();
}
catch (SoapFaultClientException ex)
{
result = ex.getFaultStringOrReason();
}
return result;
}
Now I need to create some unit tests which test all of the success and failure conditions. It cannot call the actual web service, so I was hoping there were mock objects available for the client side of Spring-WS. Does anyone know of an mock objects available for the WebServiceTemplate or any related classes? Should I just attempt to write my own and modify my class to use the WebServiceOperations interface vs. WebServiceTemplate?

Michael's answer is very close, but here is the example that works.
I already use Mockito for my unit tests, so I am familiar with the library. However, unlike my previous experience with Mockito, simply mocking the return result does not help. I need to do two things to test all of the use cases:
Modify the value stored in the StreamResult.
Throw a SoapFaultClientException.
First, I needed to realize that I cannot mock WebServiceTemplate with Mockito since it is a concrete class (you need to use EasyMock if this is essential). Luckily, the call to the web service, sendSourceAndReceiveToResult, is part of the WebServiceOperations interface. This required a change to my code to expect a WebServiceOperations vs a WebServiceTemplate.
The following code supports the first use case where a result is returned in the StreamResult parameter:
private WebServiceOperations getMockWebServiceOperations(final String resultXml)
{
WebServiceOperations mockObj = Mockito.mock(WebServiceOperations.class);
doAnswer(new Answer()
{
public Object answer(InvocationOnMock invocation)
{
try
{
Object[] args = invocation.getArguments();
StreamResult result = (StreamResult)args[2];
Writer output = result.getWriter();
output.write(resultXml);
}
catch (IOException e)
{
e.printStackTrace();
}
return null;
}
}).when(mockObj).sendSourceAndReceiveToResult(anyString(), any(StreamSource.class), any(StreamResult.class));
return mockObj;
}
The support for the second use case is similar, but requires the throwing of an exception. The following code creates a SoapFaultClientException which contains the faultString. The faultCode is used by the code I am testing which handles the web service request:
private WebServiceOperations getMockWebServiceOperations(final String faultString)
{
WebServiceOperations mockObj = Mockito.mock(WebServiceOperations.class);
SoapFault soapFault = Mockito.mock(SoapFault.class);
when(soapFault.getFaultStringOrReason()).thenReturn(faultString);
SoapBody soapBody = Mockito.mock(SoapBody.class);
when(soapBody.getFault()).thenReturn(soapFault);
SoapMessage soapMsg = Mockito.mock(SoapMessage.class);
when(soapMsg.getSoapBody()).thenReturn(soapBody);
doThrow(new SoapFaultClientException(soapMsg)).when(mockObj).sendSourceAndReceiveToResult(anyString(), any(StreamSource.class), any(StreamResult.class));
return mockObj;
}
More code may be required for both of these use cases, but they work for my purposes.

actually i don't know if there exist preconfigured Mock Objects, but i doubt there are configured for all your "failure Conditions", so you can create a special Spring ApplicationContext for your JUnit Test with a substitute or work with a mock Framework, it's not that hard :-)
i used the Mockito Mock Framework for the example (and typed it quickly), but EasyMock or your preferred mock framework should do it as well
package org.foo.bar
import java.util.ArrayList;
import java.util.List;
import org.junit.Before;
import org.junit.Test;
import static org.mockito.Mockito.*;
import static org.junit.Assert.*;
public class WebserviceTemplateMockTest {
private WhateverTheInterfaceIs webServiceTemplate;
private TestClassInterface testClass;
private final String inputXml = "bar";
#Test
public void testClient(){
//
assertTrue("foo".equals(testClass.callWebService(inputXml));
}
/**
* Create Webservice Mock.
*/
#Before
public void createMock() {
// create Mock
webServiceTemplate = mock(WhateverTheInterfaceIs.class);
// like inputXml you need to create testData for Uri etc.
// 'result' should be the needed result data to produce the
// real result of testClass.callWebService(...)
when(webServiceTemplate.sendSourceAndReceiveToResult(Uri, inputXml, new StreamResult(output))).thenReturn(result);
// or return other things, e.g.
// .thenThrow(new FoobarException());
// see mockito documentation for more possibilities
// Setup Testclass
TestClassImpl temp = new TestClassImpl();
temp.setWebServiceTemplate(generatedClient);
testClass = temp;
}
}