TestNG reporting issues due to multi-threading - java

I have a testNG framework integrated with testrails. Because of the restrictions around the testrail api, the framework was designed to collate test result in bulk and upload them all at once when the test run is complete.
To do so, I've created a BaseTest class that provides a variable id that each test method can set itself to match a corresponding test case in test rails. Once the test method assigns this variable, we pack it in to the result object:
public abstract class BaseTest {
protected static final ThreadLocal<Integer> testrailIds = new ThreadLocal();
protected int testRailCaseId = -1;
//TODO: set down a clear and strong process for handling environment/domain. Once that is in place, report it
// loudly so it is clear where the tests ran
protected String baseUrl = "";
protected static final Logger LOG = Logger.getLogger(BaseTest.class);
/**
* records the case ID stored at the individual test level inside the result set for the
* test so that we can access it later when reporting our results
* #param result the result context object
*/
#AfterMethod(alwaysRun = true)
public void afterMethod(ITestResult result) {
if(testRailCaseId == -1) {
LOG.warn("NO CASE ID HAS BEEN SET FOR THIS TEST");
}
result.setAttribute("case_id", testrailIds.get());
}
Once all tests have executed we build a request object in the afterSuite method and pipe both the test cases and test results to testrail.
for(ISuiteResult suiteResult: suite.getResults().values()) {
ctx = suiteResult.getTestContext();
for (ITestResult result : ctx.getPassedTests().getAllResults()) {
cases.add((int) result.getAttribute("case_id"));
JSONObject resultJson = new JSONObject();
resultJson.put("case_id", result.getAttribute("case_id"));
resultJson.put("status_id", 1);
payload.add(resultJson);
}
for (ITestResult result : ctx.getFailedTests().getAllResults()) {
cases.add((int) result.getAttribute("case_id"));
JSONObject resultJson = new JSONObject();
resultJson.put("case_id", result.getAttribute("case_id"));
resultJson.put("status_id", 5);
payload.add(resultJson);
}
}
// get a clean instance of the api
TestRailApi tr = new TestRailApi();
//now dump that arraylist into a json param and add it to the test run
tr.updateRun(runId, cases);
//once the test run has been created, clean up again and build the results request
tr = new TestRailApi();
tr.addResultsForCases(runId, payload);
the testRailCaseId is set at the beginning of each test method with a simple assignment
this.testRailCaseId = 491;
or
testrailIds.set(489);
This worked fine until we started using multi-threading. Now, the value of testRaidCaseId is being overwritten by parallel tests, resulting in smaller result sets than expected.
I've been attempting to manage the threads through a ThreadLocal (as seen in the code above), but have been unsuccessful so far -- values I try to set in the before method or in the tests are coming up empty in the after methods.
The test methods themselves are fine, my only struggle is with shared content being passed into them from the parent.
Anyone have any guidance for how to manage my variables across the baseTest through the test methods to ensure my various ids don't clobber each other?
Sample test case:
#Test
#Parameters({ "domain", "username", "password" })
public void logInAuthEmptyToken(#Optional("http://REDACTED.com") String domain, String username, String password) {
this.testRailCaseId = 385;
Map<String, String> loginInfo = BaseAuthTests.login(domain, username, password);
AuthServiceApi auth = new AuthServiceApi(domain);
auth.addAuthTokens("", loginInfo.get("arrival_key"), loginInfo.get("profile_key"));
auth.executeLogin(400);
}

Related

Create unique user for each test class run executing automation test

I have implemented parallel testing using Selenium and TestNG for a web application, however, my webapp got a restriction where only one session can be handled at the time (it is not supporting multi-sessions, when you login with the same user it will disconnect your older session), the solution of which I have thought to solve it is to create a unique user for each test (using API), so I have implemented it but when I am running the tests in parallel (using testng.xml file with thread-count="2") I am creating the same user twice! with the same credentials, I want to be able to create a unique user where it will create one unique user for each run.
This is my code:
public class BaseApiTest extends BaseTest {
protected String token;
protected String CREATED_ADMIN_TEST_USER;
protected String CREATED_ADMIN_TEST_PASSWORD;
private static final AtomicReference<String> ACCESS_TOKEN = new AtomicReference<>();
#BeforeClass(alwaysRun=true)
public void baseApiSetup() throws InterruptedException, ApiException {
generateToken();
createAdminUser();
}
private void generateToken() {
........
..........
...........
token = "Bearer " + ACCESS_TOKEN.get();
context.setAttribute("api-key", token);
context.setAttribute("HOST", HOST);
T_Logger.info("Host url address is: =[{}]", HOST);
T_Logger.info("new api key token =[{}]", token);
}
private void createAdminUser() throws ApiException, InterruptedException {
UsersAPIUnitTest usersAPITU = new UsersAPIUnitTest(context);
usersAPITU.createUser();
CREATED_ADMIN_TEST_USER = UsersAPIUnitTest.getEmail();
CREATED_ADMIN_TEST_PASSWORD = UsersAPIUnitTest.getPassword();
}
}
and this is used by the login page with the newly created user:
protected void adminSignIn() {
loginPage.login(CREATED_ADMIN_TEST_USER, CREATED_ADMIN_TEST_PASSWORD, true);
writeToLoggerSignIn(CREATED_ADMIN_TEST_USER);
}
and then I am starting to run my tests.
Expected: each test class will contain its own unique user
Actual: all the users that are being created are the same user with the same credentials
========================EDIT===============================
This is how I create new user:
public String createUserCms(String name, String email, String phone, String password) throws ApiException {
NewUserPayload body = new NewUserPayload();
body.setStatus(true);
body.setName(name);
body.setEmail(email);
body.setPhone(phone);
body.setPassword(password);
body.setPasswordConfirmation(password);
printBody(body);
return usersApi.createUser(token, body);
}
I have been working on a generic solution for this problem so that all automation teams can use it, still work in progress. I'm sure you would have explored other options, but creating users every time you run a test is not a good idea, rather you have to create the users once and use them otherwise you end up adding more users into your system (if you are cleaning up the users #afterMethod that's fine)
But for now, in your case, you have to make sure that you pass a unique name/email when creating the user and return that username/email address.
If you can share your createUser method snippet, we can help you more.

Code stuck when querying CosmosDB in java integration test

I'm writing integration test for an API.
In DAO class, when I try to fetch data from Cosmos DB using AsyncDocumentClient instance queryDocuments method, program has stuck indefinitely.
This is how my function call looks like:
Iterator< FeedResponse<Document>> getDataFromCosmos = asyncDocumentClientInstance.queryDocuments(collectionLink, querySpec, feedOptions).toBlocking().getIterator();
This is happening only when I run the test cases (in test mode).
But the code and API is working fine when I run in normal mode.
This is how my test class looks like:
(I'm doing integration testing, not unit testing)
import org.junit.runner.RunWith;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
#RunWith(SpringJUnit4ClassRunner.class)
public class ApiTest extends TestngInitializer
{
#Autowired
ServiceImpl serviceImpl;
#Test
public void apiTest()
{
RequestClass requestObject = new requestObject();
// hardcoding requestObject data variables using setters
ResponseClass response = serviceImpl.apiMethod(requestObject,HttpHeaders);
//Assertions on response
}
}
My TestngInitializer class looks like (Setting path to get all spring beans)
#ContextConfiguration(locations = {"classpath:/META-INF/biz-context.xml","classpath:/META-INF/service-context.xml"})
public class TestngInitializer
{
}
When I try to debug line by line, when control has reached BlockingObservable.java, control is waiting indefinitely for some thread to finish which is not the case when I run the code in normal mode.
BlockingUtils.awaitForComplete(latch, subscription);
If anyone can help me on this, that would be great.
Here is the small reproducible code I tried as well:
#Test
public void testDBLocally()
{
AsyncDocumentClient asyncDocumentClient = readDocumentClient(); //calling below function
FeedOptions feedOptions = new FeedOptions();
feedOptions.setEnableCrossPartitionQuery(true);
String collectionLink = "/dbs/link_to_collection";
SqlQuerySpec querySpec = new SqlQuerySpec("select * from some_document c ");
//here I tried two approaches
//first approach using iterator
Iterator<FeedResponse<Document>> iterator = asyncDocumentClient.queryDocuments(collectionLink,querySpec,feedOptions).toBlocking().getIterator();
while(iterator.hasNext())
{
//control stuck indefinitely at iterator.hasNext()
//control never reached inside while loop
}
//second approach using last()
FeedResponse<Document> response = capsReadDocumentClient.queryDocuments(collectionLink, querySpec, feedOptions).toBlocking().last(); //control stuck here
documentList = response.getResults(); //control never reached here
}
public AsyncDocumentClient readDocumentClient() {
String HOST = "https://azure-cosmosdb.documents.azure.com:443/";
String MASTER_KEY = "SOME_MASTER_KEY";
return new AsyncDocumentClient.Builder()
.withServiceEndpoint(HOST)
.withMasterKeyOrResourceToken(MASTER_KEY)
.withConnectionPolicy(ConnectionPolicy.GetDefault())
.withConsistencyLevel(ConsistencyLevel.Session)
.build();
}

Can't mock the methods inside the constructor

Here is my first class where the constructor has an object calling methods in other class.
Class Search{
public Search(String username, JSONObject accounts) throws Exception {
Credentials credentials = new Credentials(username);
String Uid = credentials.getUserName();
String Pwd = new String(credentials.getCredentials().getPassword());
}
public getDOB(){
--------------------
-------------
}
}
Class Credentaials:
import javax.resource.spi.security.PasswordCredential;
Public class Credentials{
public Credentials(String name){
}
public PasswordCredential getCredentials(){
return passwordCredential;
}
public String getUserName(){
PasswordCredential localCredential = getCredentials();
return localCredential.getUsername();
}
}
Class test:
#RunWith(PowerMockRunner.class)
#PrepareForTest({Search.class, Credentials.class})
public class JunitTest {
#Test
public void playTest() {
PasswordCredential pwdCreds = new PasswordCredential();
pwdCreds.setPassword("test");
Credentials credentials = new Credentials("user");
Credentials credentials = Mockito.spy(credentials);
Mockito.doReturn(pwdCreds).when(credentials).getCredentials();
Mockito.doReturn("cmApptest").when(credentials).getUserName();
Search search = new Search("username", jsonobject);
search.getDOB();
}
}
Whenever I debug the test class, it is executing the getCredentials and getUserName methods even after I mocked them. I was expecting the actual methods not to execute, instead it should return the values as I mentioned in the JunitTest class.
You aren't replacing the real version of Credentials that is being used in your Search class with a mock. Rather, you're clearly creating and using a real Credentials object inside of your Search object's constructor. For mocking to work, you have to actually replace the credentials object in your Search object with a mock. Just creating a mock of the same type somewhere in your code doesn't cause it to replace instances of the real object somewhere else in your code.
Often, dependency injection is used to introduce mocking, like with Spring. Here's a simple way to do what you want. Redefine your Search constructor like this:
class Search {
Search(String username, JSONObject accounts, Credentials creds) throws Exception {
Credentials credentials = creds? creds : new Credentials(username);
String Uid = credentials.getUserName();
String Pwd = new String(credentials.getCredentials().getPassword());
}
Search(String username, JSONObject accounts) throws Exception {
this(username, accounts, null);
}
}
The behavior of your production code will not be affected, but you can optionally construct Search with a mock.
Credentials credentials = new Credentials("user");
Credentials credentials1 = Mockito.spy(credentials);
Mockito.doReturn(pwdCreds).when(credentials1).getCredentials();
Mockito.doReturn("cmApptest").when(credentials1).getUserName();
Search search = new Search("username", jsonobject, credentials1);
search.getDOB();
There's no magic in terms of your code using a mock rather than the real object. Mocking frameworks just let you easily create stand-in objects that act in very specific ways for testing. You still have to cause those objects to be used by your code.
Also, you don't really need/want a spy here. You really want a mock, because you're defining the behavior of all of the methods in Credentials. With a mock, you wouldn't need to instantiate your Credentials object at all. So the first lines of the test code I gave above could could be:
Credentials credentials1 = Mockito.mock(Credentials.class);
(or something like this. I'm not actually trying this code)

Drools - Retrieve Output object

I am new to Kie Workbench and Execution Server. I am using Java Rest calls to run rules in kie workbench. Please find the code below:
private String kieServerUrl;
private String kieServerContainerId;
private String KieServerUsername;
private String kieServerPassword;
private RuleServicesClient ruleClient;
private static final String INPUT_OUT_IDENTIFIER = "Input";
private static final String SESSION_OBJECTS = "SessionObjects";
private static final String RUN_ALL_RULES = "RunAllRules";
public void init() {
final KieServicesConfiguration config = KieServicesFactory.newRestConfiguration(kieServerUrl, KieServerUsername, kieServerPassword);
config.setMarshallingFormat(MarshallingFormat.XSTREAM);
KieServicesClient kieServicesClient = KieServicesFactory.newKieServicesClient(config);
ruleClient = kieServicesClient.getServicesClient(RuleServicesClient.class);
}
#Override
public Output process(final Input input) {
Output output = null;
logger.debug("Running rules ..");
BatchExecutionCommandImpl executionCommand = new BatchExecutionCommandImpl();
executionCommand.getCommands().add(new InsertObjectCommand(input, INPUT_OUT_IDENTIFIER));
executionCommand.getCommands().add(new FireAllRulesCommand(RUN_ALL_RULES));
executionCommand.getCommands().add(new GetObjectsCommand(null, SESSION_OBJECTS));
logger.debug("Sending commands to the server");
ServiceResponse<ExecutionResults> response = ruleClient.executeCommandsWithResults(kieServerContainerId, executionCommand);
if(response.getType().equals(ServiceResponse.ResponseType.SUCCESS)){
logger.debug("Commands executed with success! Response: ");
final ExecutionResultImpl result = (ExecutionResultImpl) response.getResult();
ArrayList<Object> values = (ArrayList<Object>)result.getValue(SESSION_OBJECTS);
}else{
logger.error("Error executing rules. Message: {}", response.getMsg());
}
logger.debug("...finished running rules.");
return output;
}
The rules are correctly executed and the Output Object are instancied during the rules. One problem is when I again call this method to execute rules for the second time, I receive two Output object and for each subsequent calls I get one additional object. It seems that the objects in the session are stored and not getting cleared for each call. How can I achieve that for every call I will get only one Output object in return?
Since you are new to Drools, you may not know that Drools has two session types, stateless and stateful. Verify the KIE Execution Server session configuration is stateless, as stateful keeps the facts from prior requests processing.
Verify it is stateless by its settings in the project editor:
Open Project Editor -> Knowledge bases and sessions
Review the existing one or create one with:
Add Knowledge Sessions -> and set the State to Stateless

How to reset variable values to null on completion of a TestNG Test Class

I have created some automated tests to test an API using TestNG.
public class EndToEndTest {
private ResponseParser response;
private static String sessionId;
private static float cardCharge;
#BeforeClass
public void setUp() {
baseURI = getUri();
response = new ResponseParser();
#Test
#Parameters({"login", "username", "password"})
public void testOne(){
//send some test data into API using specific test data parameters
sessionId = response.getSessionId();
cardCharge = response.getCardCharge();
//assertion etc...
}
#Test
public void testTwo()
Request req = new Request.RequestBuider()
.withSession(sessionId)
.withCardCharge(cardCharge)
.with....etc...
}
This test is a template against which I have 10 or so individual testng xml files providing individual test data - essentially 10 tests. I am running the 10 testng xml files using 1 testng suite xml file.
When one 'test' finishes, I want the variables - 'sessionId' etc...to be reset to null. With the way I am running them, the variable values persist between the #Tests (desirable) but also at the class level, presumably because the way I am running with a suite, they remain in scope and not eligible for GC.
Is there a more elegant/robust solution than simply resetting each to null, for example in an #AfterClass method?
#AfterClass
public void tearDown(){
sessionId = null;
.....
}
yes, there is another way to perform this. You can override the #Test class and manipulate its behavior where u can set a default value for every variable being used.
please refer ;
http://www.javatpoint.com/custom-annotation
Did you have a look to the #BeforeTest and #AfterTestannotations?
See [TestNg documentation][1]
http://testng.org/doc/documentation-main.html

Categories