Karate tests pass or fail inconsistently - java

I'm using Karate with Gradle, and have 8 feature files that test read / GET functionality for my Spring Boot API.
I'm seeing these tests fail in a way that feels quite random.
The failures are related to Authorisation somehow, but I can't see anything that's wrong on the face of it.
Here's an example,
This fails
Feature: Get Objectives
Background:
* url baseUrl
* def objectiveEndpoint = '/objectives'
* header Authorization = token
Scenario: Get an Objective that exists
Given path objectiveEndpoint + '/37376564-3139-6232-2d66-6631392d3466'
When method GET
Then status 200
And match response contains { objectiveId: '37376564-3139-6232-2d66-6631392d3466' }
And this passes
Feature: Get Assessments
Background:
* url baseUrl
* def assessmentEndpoint = '/assessments'
* header Authorization = token
Scenario: Get an assessment that exists
Given path assessmentEndpoint + '/2900b695-d344-4bec-b25d-524f6b22a93a'
When method GET
Then status 200
And match response contains { odiAssessmentId: '2900b695-d344-4bec-b25d-524f6b22a93a' }
The objective test fails due to a 401 with the following message:
com.intuit.karate.exception.KarateException: objectives-read.feature:12 - status code was: 401, expected: 200, response time: 74, url: http://localhost:8080/objectives/37376564-3139-6232-2d66-6631392d3466, response: {"path":"/objectives/37376564-3139-6232-2d66-6631392d3466","error":"Unauthorized","message":"Unauthorized","timestamp":"2020-06-02T08:04:57.231+0000","status":401}
The assessments test passes.
I'm getting a token by running an Authorisation feature, and storing the result from that into the token variable.
The Auth feature is:
Feature: Log In
Background:
* url 'https://$URL/oauth/token'
* def localSecret = secret
* def localClientId = clientId
Scenario: Get token
Given url 'https://$URL/oauth/token'
And form field grant_type = 'client_credentials'
And form field client_id = localClientId
And form field client_secret = localSecret
And form field audience = 'http://localhost:8080'
When method post
Then status 200
And match response contains { access_token: '#string' }
And def access_token = $.access_token
I then add the token to config, and pass it into each test like this:
var result = karate.callSingle('classpath:login.feature', config);
config.token = 'Bearer ' + result.access_token;
I've confirmed that the token used in the above two features is valid. I've manually tested my API with the token printed in the output of the failed tests, and both of the above tests work fine when I recreate them in Postman. This doesn't feel like a problem with my API because if I rerun the test suite, the tests that fail differ, and on my CI I have a green build with these tests.
I'm experiencing the problem both when running test suites individually like this:
#Karate.Test
Karate testAssessments() {
return Karate.run().relativeTo(AssessmentsRunner.class);
}
and when running all of my tests in parallel like this:
public class SpecTestParallel {
public static void generateReport(String karateOutputPath) {
Collection<File> jsonFiles = FileUtils.listFiles(new File(karateOutputPath), new String[]{"json"}, true);
List<String> jsonPaths = new ArrayList(jsonFiles.size());
jsonFiles.forEach(file -> jsonPaths.add(file.getAbsolutePath()));
Configuration config = new Configuration(new File("target"), "Test API");
ReportBuilder reportBuilder = new ReportBuilder(jsonPaths, config);
reportBuilder.generateReports();
}
#Test
void testParallel() {
Results results = Runner.path("classpath:specTests").tags("~#ignore").parallel(5);
generateReport(results.getReportDir());
assertEquals(0, results.getFailCount(), results.getErrorMessages());
}
}
Has anyone had a similar issue before?

As it turns out, this was unexpected behaviour in IntelliJ.
So Karate appears to have a retry policy. If a test failes, it will retry a few times.
When I run tests using the test runner function in IntelliJ, each time a test fails, IntelliJ logs that in the test runner window as it should for a failed test, but Karate keeps running, retries and the test passes. I can see that in the reports now but the IntelliJ test runner doesn't update.
This leads to a confusing situation where tests pass, but appear to fail locally, but the tests pass on CI.
Here is an example of local tests:
and the same commit passing in CI:
I'm not really sure what the fix is here, if there is one. It would be nice if IntelliJ was aware of this behaviour, or maybe Karate could only report the result of a test after all retries have been processed - right now it looks like Karate reports as soon as the first test run is processed.
It's been a confusing few days.

Related

TestNG java.lang.AssertionError: Expected: <1> Actual: <>

I am new to TestNG and my task is to test a SportBetting apps API endpoints. To access the endpoints I also should authenticate the user, and somehow keep the session/ cookie to keep the user logged in if I want to test other api endpoints too.
My method looks like this right now:
#Test
public void returnPlayerById(){
RestAssured.baseURI ="http://localhost:8080/sportsbetting-web/players/1";
given().get("http://localhost:8080/sportsbetting-web/players/1").then().statusCode(200).body("player.id[0]", equalTo(1));
}
However it gives the following error:
java.lang.AssertionError: 1 expectation failed.
XML path player.id[0] doesn't match.
Expected: <1>
Actual: <>
I understand, that somehow my method finds nothing. But why? Is it because the code itself is bad, or is it because the authentication is missing? Im very new to this topic. How should I proceed?

Rest Assured Proxy Setting issue (java.net.ConnectException: Connection timed out exception)

One of the API (Say XYZ API) call alone requires the proxy to be set, When I run a suite where this API is called before and after every test script in the suite there is java.net.ConnectException seen randomly.
Only for XYZ API, I set and reset the proxy using the line:
RestAssured.proxy(String, int)
and
RestAssured.reset()
Please help me resolve this issue. Could this be a bug?
Note: This API call fails randomly just saying first 10 times it would have passed may be the 11th it fails with exception rest all scripts are then skipped in execution.
Code snapshot:
public void GetAuthenticationToken(String userid, String password)
{
RestAssured.proxy(config.getProperty("ProxyHost"),Integer.parseInt(config.getProperty("ProxyPort"))); //Authentication API is outside network and requires proxy
String APIUrl_Aut = config.getProperty("APIUrl_Aut");
String APIBody_Aut = "grant_type=password&username="+userid+"&password="+password;
//Making post request with authentication
Response response = RestAssured.given().log().all().headers("Accept", "application/json","Content-Type","application/x-www-form-urlencoded").body(APIBody_Aut).
when().post(APIUrl_Aut).then().contentType(ContentType.JSON).extract().response();
if (response.getStatusCode() == 200)
{
WritePropertyToTemp("AUTH_TOKEN", "Bearer "+response.body().jsonPath().get("access_token").toString());
WritePropertyToTemp("AUTH_TOKEN_OnlyToken",response.body().jsonPath().get("access_token").toString());
log.info("Authentication token generated successfully");
}
else
log.info("Authentication token failed to generate");
RestAssured.reset(); //Resetting proxy
}

Failure rule for cucumber-jvm

I am writing test suite for Rest APIs using cucumber-java. I was trying to use #Rule, however I found it was not supported by cucumber-jvm. Is there any option to write a rule on step failure ?
I am need to print the request-response on assertion failure (I'm asserting on the http response status), so that I can get an understanding on what went wrong?
Any help on this is much appreciated
Cucumber-jvm has the #After annotation and you could put something in there but why not simply include the response in the assertion statement failure text?
I use rest-assured to manage my HTTP request-responses. My responses end up in objects of class Response. To get the entire response, I might code this:
response.getBody().prettyPrint()
EDIT
You specified request as well. Given requestSpec is the instance of class RequestSpecification that we initialized with the request values, I use
requestSpec.given().log().all()
to get the request into string form.
So using JUnit assertTrue
assertTrue("REST service x Status Code =" + response.getStatusCode() +
", for request = " + requestSpec.given().log().all() + ",\n response = " +
response.getBody().prettyPrint(), response.getStatusCode() == 200);

restAssured - cannot master post method

fellow stackoverflowians :)
I've been for quit time to make a Post call using Gmail API.
Been trying to use createDraft and createLabel.
Now I guess I've found how to do this correctly (mostly) but I get this error:
java.lang.AssertionError: 1 expectation failed.
Expected status code <200> but was <400>.
I realise that this error occurs because I make incorrect request.
Could You, guys, help me with this?
Here's my code:
import io.restassured.RestAssured.*
import io.restassured.http.ContentType
import io.restassured.matcher.RestAssuredMatchers.*
import org.hamcrest.Matchers.*
import org.testng.annotations.Test
class RestAPIAutoTestPost {
#Test
fun createLabelInGoogleMail() {
RestAssured.baseURI = "https://www.googleapis.com/gmail/v1/users/me"
val accessToken = "ya29.Glw7BEv6***"
val jsonAsMap = HashMap<String, Any>()
jsonAsMap.put("id", "labelAPITestNameID")
jsonAsMap.put("labelListVisibility", "labelShow")
jsonAsMap.put("messageListVisibility", "show")
jsonAsMap.put("messagesTotal", "0")
jsonAsMap.put("messagesUnread", "0")
jsonAsMap.put("name", "labelAPITestName")
jsonAsMap.put("threadsTotal", "0")
jsonAsMap.put("threadsUnread", "0")
jsonAsMap.put("type", "user")
given().
contentType(ContentType.JSON).
body(jsonAsMap).
`when`()
post("/labels?access_token=$accessToken").
then().
statusCode(200)
}
}
I suppose I use HashMap incorrectly or I use some incorrect body fields.
I've only started to learn restAssured so I beg my pardons for newby question.
Thanks!
P.S. I'd really appreciate for any help with Post methods and puting data into body
I think your use of RestAssured and HashMap is correct. I think you are getting a 400 from this API because you are specifying the id property. By playing with this in Google's API Explorer, I was able to generate 400 errors by doing that. According to the documentation, the only things you need to specify for a POST/Create are: labelListVisibility, messageListVisibility, and name. The id is returned to you as part of the response.
A good feature in RestAssured is that you can have it log what it sends or receives when there is an error or all the time.
Log all requests:
given().log().all()
Log all responses:
`when`().log().all()
Or just when validations fail:
`when`().log().ifValidationFails()
Using that will give you a more precise reason why your interaction with the API is failing because it will show whatever Google is sending back. So we can see for sure if I'm right about the id.
And since you seem to be using Kotlin for this, you might want to take advantage of its great multiline string capabilities and just create the JSON payload manually:
val body = """
{
"labelListVisibility": "labelShow",
"messageListVisibility": "show",
"name": "ThisIsATest"
}
"""

Iterating through SOAP requests from a properties file

I am writing a java plugin that I plan to use to test a number of web services. These SOAPs for the web services are located in a properties file, and are grouped under which WSDL they fall under (Subscriber, Network, User, etc...). Also, there are some regexs associated with each web service to test the response against.
Properties Example
#Web services to be tested and regexes to test responses
#List of web service groups used (WSDLs)
webservice.list = SubscriberMgmt,NetworkMgmt
# < -- SubscriberMgmt -- >
#getSubscriberDevices
webservice.subscriber = <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ws="http://ws.blah.blah.com"><soapenv:Header/><soapenv:Body><ws:getSubscriberDevices><PhoneNumber></PhoneNumber><LastName></LastName><MACAddress></MACAddress><ExternalId></ExternalId><AccountExternalId>john</AccountExternalId><IPAddress></IPAddress></ws:getSubscriberDevices></soapenv:Body></soapenv:Envelope>
webservice.SubscriberMgmt.regex = subscriberId="(.+?)"
webservice.SubscriberMgmt.regex.1 = externalId="(.+?)"
#getMpegResultsById
webservice.subscriber.1 = <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ws="http://ws.blah.blah.com"><soapenv:Header/><soapenv:Body><ws:getMpegResultsById><SubscriberId>100016</SubscriberId><Duration>2880</Duration></ws:getMpegResultsById></soapenv:Body></soapenv:Envelope>
webservice.SubscriberMgmt.1.regex = id="(.+?)"
webservice.SubscriberMgmt.1.regex.1 = externalId="(.+?)"
I currently have code to connect using each WSDL from the properties file, so say when the 'webservicegroup' variable is SubscriberMgmt, I'd like to test the .subscriber web service(s) and check the responses if it contains the corresponding regex(es). (the 'data' variable only corresponds to one SOAP request from the property file at the moment)
//Soap Request
try
{
for(String webservicegroup : webserviceList)
{
URL url = new URL("http://" + server + "/webservices/" + webservicegroup);
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
conn.setRequestProperty("Content-type", "text/xml; charset=utf-8");
conn.setRequestProperty("SOAPAction", "\"\"");
String loginEnc = new BASE64Encoder().encodeBuffer((username + ":" + password).getBytes());
loginEnc = loginEnc.replaceAll("\n", "");
conn.setRequestProperty("Authorization", "Basic " + loginEnc);
conn.setConnectTimeout(timeout);
conn.setReadTimeout(timeout);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
//Send request
wr.write(data);
wr.flush();
wr.close();
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
//Save response
String line;
while ((line = in.readLine()) != null)
{
response += line;
}
in.close();
}
}
Any ideas on the best way of doing this? Any help is greatly appreciated
Assuming your connection and POST/GET code is working:
Step 1: Get the entire response as a string:
String response = new String(ByteStreams.toByteArray(inputStream), "UTF-8");
In the above line of code, the ByteStreams class is part of google's guava library. Similar code can be found in apache commons-io, if you prefer that.
Step 2: Test Regular expressions:
if( response.matches(regex) ) { ... }
Don't reinvent the wheel, building a custom testing software mini empire from scratch.
Use SOAPUI open source version to test web services. It allows you to:
generate SOAP tests from web service endpoint WSDL, saving manual labour
avoid processing "pre-canned" SOAP strings and parsing files
automatically generate mocks of your service endpoints, including ability to programmatically control custom responses.
implement test step logic including loops, branches, invoking other test steps, running scripts, and reading/writing parameters from/to property files and data sources (although you must programatically navigate & modify XmlHolder instances via scripts if you wish to "data-drive" your tests)
Execute SOAP, REST, security & load testing of web services, and also JDBC and HTTP test calls.
Integrates with the common build and continuous integration tools for test automation (TDD & continuous delivery).
Use SOAPUI operations within your IDE, via plugins.
It's considered fairly standard "best practice" for testing web services.
To checks SOAP response messages for valid content, using the open source version of SOAPUI:
you can use XPath or XQuery expressions to validate the XML.
you can use script assertions
E.g. if your SOAP response is:
<soap:Body>
<GetEnumResponse xmlns="http://www.xyz.com/">
<GetEnumResult>
<ErrorCode>0</ErrorCode>
<StatusId>0</StatusId>
</GetEnumResult>
<enumsInformation>
<EnumInformation>
<TransactionId>0</TransactionId>
<ConstraintId>5000006</ConstraintId>
<EnumValue>xyz</EnumValue>
<Index>10</Index>
</EnumInformation>
</enumsInformation>
</GetEnumResponse>
</soap:Body>
You can script:
import com.eviware.soapui.support.XmlHolder
def holder = new XmlHolder(messageExchange.responseContentAsXml)
holder.namespaces["tal"]="http://www.xyz.com/"
def node = holder.getNodeValue("//tal:ConstraintId[1]");
log.info(node);
assert node == "5000006";
You can even use the maximum power of standard java regex processing.
Create java classes that do the regex processing and put them into a jar file and place in soapUIinstallation/bin/ext as explained here.
Or wrap your SOAPUI Test inside a JUnit test method, and add standard java code at end to check regexs. This also eases test automation & allows any non-web service tests to be executed as well. This approach works with SOAPUI open source version, whereas the alternative of using SOAPUI assertion steps requires the Pro version.
Steps:
If you choose, install the SOAPUI plugin in your IDE
Use SOAPUI to create a test suite
Use SOAPUI to create test case(s) within the test suite
Use SOAPUI to create test step(s) within the test suite. This is the core of using SOAPUI.
Create a Java project in your IDE. Within this project, add a JUnit test case.
Add all JARs from SoapUI bin and lib directories to Java Build Path.
Within the Junit Test case, add code to execute a SOAPUI test step
Obtain the MessageExchange object, get the response from it, and then get headers, content or attachments. Run a regex check on result.
The following is indicative only. Not intended to be a working example.
package com.example;
import org.junit.Test;
import com.eviware.soapui.tools.SoapUITestCaseRunner;
public class SoapUIProject {
// runs an entire SOAPUI test suite
#Test
public void soapTest1() throws Exception {
SoapUITestCaseRunner runner = new SoapUITestCaseRunner();
runner.setProjectFile("/path/to/your/W3Schools-Tutorial-soapui-project.xml");
runner.run();
}
// runs a single SOAPUI test step - and checks response matches a regex
#Test
public void soapTest2() throws Exception {
WsdlProject project = new WsdlProject( "src/dist/sample-soapui-project.xml" );
TestSuite testSuite = project.getTestSuiteByName("My Test Suite");
TestCase testCase = testSuite.getTestCaseByName("My Test Case");
TestCaseRunner testCaseRunner = new WsdlTestCaseRunner(testCase,null);
// Must have test step setup as WsdlMessageExchange for cast to work
WsdlMessageExchangeTestStepResult testStepResult = (WsdlMessageExchangeTestStepResult)testStep.runTestStepByName("My Test Step");
// TestStep testStep = testCase.getTestStepByName("My Test Step");
// TestCaseRunContext testCaseRunContext = new WsdlTestRunContext(testStep);
// testStep.prepare(testCaseRunner, testCaseRunContext);
// WsdlMessageExchangeTestStepResult testStepResult = (WsdlMessageExchangeTestStepResult)testStep.run(testCaseRunner, testCaseRunContext);
MessageExchange[] messageExchanges = testStepResult.getMessageExchanges();
for (MessageExchange me : messageExchanges) {
String response = me.getResponseContentAsXML();
// do any desired regex processing
// can use any desired assertions
}
assertEquals( Status.FINISHED, runner.getStatus() );
}
}
Further refs: http://www.soapui.org/Scripting-Properties/tips-a-tricks.html#3-xml-nodes
http://books.google.com.au/books?id=DkWx7xZ263gC&printsec=frontcover#v=onepage&q&f=false

Categories