We Keep Coding

ChromeDevTools in selenium, waiting for response bodies

I need to work on ajax response, that is one of responses received upon visiting a page. I use selenium dev tools and java. I create a listener, that intercepts a specific request and then I want to work on response it brings. However I need to setup static wait, or else selenium don't have time to save RequestId. I read Chrome Dev Tools documentation, but it's a new thing for me. I wonder if there is a method that would allow me to wait for this call to be completed, other than the static wait.
Here is my code:
#Test(groups = "test")
public void x() throws InterruptedException, JsonProcessingException {
User user = User.builder();
ManageAccountStep manageAccountStep = new ManageAccountStep(getDriver());
DashboardPO dashboardPO = new DashboardPO(getDriver());
manageAccountStep.login(user);
DevTools devTools = ((HasDevTools) getDriver()).maybeGetDevTools().orElseThrow();
devTools.createSessionIfThereIsNotOne();
devTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty()));
// end of boilerplate
final RequestId[] id = new RequestId[1];
devTools.addListener(Network.responseReceived(), response -> {
log.info(response.getResponse().getUrl());
if (response.getResponse().getUrl().contains(DESIRED_URL)){
id[0] = response.getRequestId();
}
});
dashboardPO
.clickLink(); // here is when my DESIRED_URL happens
Utils.sleep(5000); // Something like Thread.sleep(5000)
String responseBody = devTools.send(Network.getResponseBody(id[0])).getBody();
// some operations on responseBody
devTools.clearListeners();
devTools.disconnectSession();
}
If I don't use 5 seconds wait id variable gets never assigned and I null pointer exception requestId is required. During these 5 seconds log.info prints all api calls that are happening and it almost always finds my id. I would like to refrain from static wait though. I am thinking about something similiar to maybe jQuery.active()==0, but my page doesn't use jQuery.

You may try custom function Explicit Wait. Something like this:
public String getResponseBody(WebDriver driver, DevTools devTools) {
return new WebDriverWait(driver,5)
.ignoring(NullPointerException.class)
.until(driver ->
devTools.send(Network.getResponseBody(id[0])).getBody());
}
So, it won't wait for all 5 seconds. The moment it got the data, it would come of out of the until method. Also add whichever Exception that was coming up here.
Has put these lines of code as separate method because, devTools object is locally defined. In order to use them inside this anonymous inner function, it has to be final or effectively final.

I seem to run into this issue when running tests in parallel (and headless) and trying to capture the requests and responses, I get:
{"No data found for resource with given identifier"},"sessionId" ...
However, now .until seems to only take ExpectedCondition
So a similar solution (to the accepted answer), but without using "WebDriverWait.until" that I use is:
public static String getResponseBody(DevTools devTools, RequestId id) {
String requestPostData = "";
LocalDateTime then = LocalDateTime.now();
String err = "";
Integer it = 0;
while (true) {
err = "";
try{requestPostData = devTools.send(Network.getResponseBody(id)).getBody();} catch( Exception e){err = e.getMessage();};
if (requestPostData != null && !requestPostData.equals("")) {break;}
if (err.equals("")) {break;} // if we don't have an error message, its quite possible the responseBody really is an empty string
long timeTaken = ChronoUnit.SECONDS.between(then, LocalDateTime.now());
if (timeTaken >= 5) {requestPostData = err + ", timeTaken:" + timeTaken; break;}
if(it > 0) {TimeUnit.SECONDS.sleep(it);} // I prefer waiting longer and longer, avoiding stack overflows
it++;
}
return requestPostData;
}
It just loops until it doesn't error, and returns the string back as soon as it can (but I actually set timeTaken >= 60 due to many parallel requests)

Downloading attachments from unseen messages

I work on university project in java. I have to download attachments from new emails using GMAIL API.
I successfully connected to gmail account using OAuth 2.0 authorization.
private static final List<String> SCOPES = Collections.singletonList(GmailScopes.GMAIL_READONLY);
I tried to get unseen mails using
ListMessagesResponse listMessageResponse = service.users().messages().list(user).setQ("is:unseen").execute();
listMessageResponse is not null but when I call method .getResultSizeEstimate() it returns 0
also I tried to convert listMessageResponse to List < Message > (I guess this is more usable) using
List<Message> list = listMessageResponse.getMessages();
But list launches NullPointerException
Then tried to get each attachment with
for(Message m : list) {
List<MessagePart> part = m.getPayload().getParts();
for(MessagePart p: part) {
if(p.getFilename()!=null && p.getFilename().length()>0) {
System.out.println(p.getFilename()); // Just to check attachment filename
}
}
}
Is my approach correct (if not how to fix it) and how should I download those attachments.
EDIT 1:
Fixed q parameter, I mistakenly wrote is:unseen instead of is:unread.
Now app reaches unread mails successfully.
(For example there was two unread mails and both successfully reached, I can get theirs IDs easy).
Now this part trows NullPointerException
List<MessagePart> part = m.getPayload().getParts();
Both messages have attachments and m is not null (I get ID with .getID())
Any ideas how to overcome this and download attachment?
EDIT 2:
Attachments Downloading part
for(MessagePart p : parts) {
if ((p.getFilename() != null && p.getFilename().length() > 0)) {
String filename = p.getFilename();
String attId = p.getBody().getAttachmentId();
MessagePartBody attachPart;
FileOutputStream fileOutFile = null;
try {
attachPart = service.users().messages().attachments().get("me", p.getPartId(), attId).execute();
byte[] fileByteArray = Base64.decodeBase64(attachPart.getData());
fileOutFile = new FileOutputStream(filename); // Or any other dir
fileOutFile.write(fileByteArray);
fileOutFile.close();
}catch (IOException e) {
System.out.println("IO Exception processing attachment: " + filename);
} finally {
if (fileOutFile != null) {
try {
fileOutFile.close();
} catch (IOException e) {
// probably doesn't matter
}
}
}
}
}
Downloading working like charm, tested app with different type of emails.
Only thing left is to change label of unread message (that was reached by app) to read. Any tips how to do it?
And one tiny question:
I want this app to fetch mails on every 10 minutes using TimerTask abstract class. Is there need for manual "closing" of connection with gmail or that's done automatically after run() method iteration ends?
#Override
public void run(){
// Some fancy code
service.close(); // Something like that if even exists
}

I don't think ListMessagesResponse ever becomes null. Even if there are no messages that match your query, at least resultSizeEstimate will get populated in the resulting response: see Users.messages: list > Response.
I think you are using the correct approach, just that there is no message that matches your query. Actually, I never saw is:unseen before. Did you mean is:unread instead?
Update:
When using Users.messages: list only the id and the threadId of each message is populated, so you cannot access the message payload. In order to get the full message resource, you have to use Users.messages: get instead, as you can see in the referenced link:
Note that each message resource contains only an id and a threadId. Additional message details can be fetched using the messages.get method.
So in this case, after getting the list of messages, you have to iterate through the list, and do the following for each message in the list:
Get the message id via m.getId().
Once you have retrieved the message id, use it to call Gmail.Users.Messages.Get and get the full message resource. The retrieved message should have all fields populated, including payload, and you should be able to access the corresponding attachments.
Code sample:
List<Message> list = listMessageResponse.getMessages();
for(Message m : list) {
Message message = service.users().messages().get(user, m.getId()).execute();
List<MessagePart> part = message.getPayload().getParts();
// Rest of code
}
Reference:
Class ListMessagesResponse
Users.messages: list > Response

java axis web service client setMaintainSession on multiple services (cookies?)

I'm implementing a client to a web service (and the guys maintaining the web service have been a litte unresponsive..) I've used axis and WSDL2Java to generate java classes and I can call their login-method on their authentication-service ok, and get a sessionId back (eg z4zojhiqkw40lj55kgtn1oya). However, it seems that i cannot use this sessionId as a parameter anywhere. Even a call to their hasSession()-method directly after login returned false. I managed to solve this by setting setMaintainSession(true) on the Locator-object for this service. But the problem is, that this first service, the Authentication-service, is only used for authentification. If I then call setMaintainSession(true) on eg ProductServiceLocator, and call some method on it, I will get an error because of unauthenticated session. I have to find a way to share the session between the services on the client side.
Looking on their php code example-it seeems like they are storing the session in a cookie. How can I mimic this behaviour in my java client?
php-code:
$authentication = new SoapClient ( "https://webservices.24sevenoffice.com/authenticate/authenticate.asmx?wsdl", $options );
// log into 24SevenOffice if we don't have any active session. No point doing this more than once.
$login = true;
if (!empty($_SESSION['ASP.NET_SessionId'])){
$authentication->__setCookie("ASP.NET_SessionId", $_SESSION['ASP.NET_SessionId']);
try{
$login = !($authentication->HasSession()->HasSessionResult);
}
catch ( SoapFault $fault ) {
$login = true;
}
}
if( $login ){
$result = ($temp = $authentication->Login($params));
// set the session id for next time we call this page
$_SESSION['ASP.NET_SessionId'] = $result->LoginResult;
// each seperate webservice need the cookie set
$authentication->__setCookie("ASP.NET_SessionId", $_SESSION['ASP.NET_SessionId']);
// throw an error if the login is unsuccessful
if($authentication->HasSession()->HasSessionResult == false)
throw new SoapFault("0", "Invalid credential information.");
}
My code is the following:
AuthenticateLocator al = new AuthenticateLocator();
al.setMaintainSession(true);
Credential c = new Credential(CredentialType.Community,username,password,guid);
AuthenticateSoap s = al.getAuthenticateSoap();
String sessionId = s.login(c);
System.out.println("Session id was: "+sessionId);
System.out.println("Has Session: "+s.hasSession()); //Hooray, now works after setMaintainSession(true)
//And now trying to call another Service
CompanyServiceLocator cl = new CompanyServiceLocator();
cl.setMaintainSession(true);
CompanyServiceSoap css = cl.getCompanyServiceSoap();
css.getCountryList(); //FAILS!
So what can I do to make this work?

Hooray, I finally solved it myself :-D
Thanx a lot to the excellent article at http://www.nsftools.com/stubby/ApacheAxisClientTips.htm
I had to do the following with my code to make it work:
CompanyServiceLocator cl = new CompanyServiceLocator();
cl.setMaintainSession(true);
CompanyServiceSoap css = cl.getCompanyServiceSoap();
((Stub)css)._setProperty(HTTPConstants.HEADER_COOKIE, "ASP.NET_SessionId="+sessionId); //New line that does the magic
css.getCountryList(); //SUCCESS :-D
Operating in the high-level abstraction of the autogenerated classes, it was unknown to me that casting the service classes to Stub would expose more methods and properties that could be set. Good to know for later I guess :-)

Best practice: catching failure points in java.net.URL

New to the JVM, working with Scala and Play 2.0
I'm converting a legacy application over to Play, one that requires payment processing via Authorize.net. Looking through java.net.URL source, there are numerous potential points of failure. Given the interface I've written below, where would you implement try/catch blocks? I'll need to adapt method signatures accordingly, probably returning an Either[Error, Success] to calling client code
import java.net.{URL, URLEncoder}
import java.io.{BufferedReader, DataOutputStream, InputStreamReader}
import javax.net.ssl._
trait Authnet {
private val prodUrl = "https://secure.authorize.net/gateway/transact.dll"
private val testUrl = "https://test.authorize.net/gateway/transact.dll"
protected def authNetProcess(params: Map[String,String]) = {
val(conn, urlParams) = connect(params)
val request = new DataOutputStream( conn.getOutputStream )
request.write(urlParams.getBytes)
request.flush()
request.close()
val response = new BufferedReader(new InputStreamReader(conn.getInputStream))
val results = response.readLine().split("\\|")
response.close()
results.toList
}
private def connect(params: Map[String,String]) = {
val urlParams = (config ++ params) map { case(k,v) =>
URLEncoder.encode(k, "UTF-8") + "=" + URLEncoder.encode(v, "UTF-8")
} mkString("&")
lazy val url = if (isDev) new URL(testUrl) else new URL(prodUrl)
val conn = url.openConnection
conn.setDoOutput(true)
conn.setUseCaches(false)
(conn, urlParams)
}
private val config = Map(
'x_login -> "...",
'x_tran_key -> "...",
...
)
}

Stick to the thumb rule:
Only catch an exception if you must handle it.
There is no sharp definition for "must handle" but it means you should resist the urge to catch an exception because you can just to throw a different exception.
The "must handle" is mainly defined by how your application should work or other dependencies.
If the application requires to display an error to the user instead of aborting with an exception, then it's a must.
In that case catching the excpetion adds also some meaningful processing.
If an API requires to throw a different exception, then it's a must, but the APIs definition is possibly not sound.
I am always questioning the added value of replacing an exception with just another exception.
Applying this to your example:
Would it add some value to catch an exception from connect() in authNetProcess()?
No! There is no way to handle that exception inside of connect(). So its ok to leave that exception to the caller of authNetProcess. There you could provide different handling based on the kind of the exception.

EDIT
Well, if any part of the connection/stream process fails, the transaction is hosed, so silly to only capture error on opening of the connection. I'm just wrapping the whole transaction in a catching (operation) option block, and leaving it at that; I'm not too concerned re: the exact cause of the error (whatever it is gets logged) as it is transient, so catch it, have the user try again; if error persists, contact us...
ORIGINAL
OK, well, given the up votes and lack of commentary to-date, the only conclusion I can draw is...nobody around here knows what they're doing! heh, heh, joking ;-)
Although I'm new to the JVM, try/catch/finally bloat is getting old fast; via the wonders of Scala type inference, I have abstracted away general error handling into concise implementations:
catching ( operation ) option
catching ( operation ) either
Unless I receive feedback otherwise, for now I'm copping out by just catching connection creation (I believe, in this case, the most likely error condition). Here's the new implementation:
protected def authNetProcess(params: Map[String,String]) = {
connect() match {
case Some(conn) =>
val request = new DataOutputStream(conn.getOutputStream)
request.write(getUrlParams(params).getBytes)
request.flush()
request.close()
val response = new BufferedReader(new InputStreamReader(conn.getInputStream))
val results = response.readLine().split("\\|")
response.close()
results.toList
case None => List[String]()
}
}
private def connect() = {
lazy val url = if (isDev) new URL(testUrl) else new URL(prodUrl)
catching ( url.openConnection ) option match {
case Some(conn) =>
conn.setDoOutput(true)
conn.setUseCaches(false)
//conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded")
Some(conn)
case None => None // connection failed
}
}
I suppose a more rigorous approach would be to extract all potential error conditions into maybeWorked Option operations, and then wrap them all up in a for comprehension. That's probably the proper/responsible approach...but only so many hours in a day, will revisit this later
feedback appreciated!

AMF client in Java

I am using BlazeDS java client to get info from this page.
This page has a form in the middle that when you select a type, the location combo on the button gets updated.
I am trying to use BlazeDS to get those values in java.
I have been using Charles web proxy to debug, and this are the screenshots from the request and the response:
My code so far is the following:
// Create the AMF connection.
AMFConnection amfConnection = new AMFConnection();
// Connect to the remote url.
String url = "http://orlandoinfo.com/flex2gateway/";
try
{
amfConnection.connect(url);
}
catch (ClientStatusException cse)
{
System.out.println(cse);
return;
}
// Make a remoting call and retrieve the result.
try
{
// amfConnection.registerAlias("flex.messaging.io.ArrayCollection", "flex.messaging.io.ArrayCollection");
amfConnection.call("ColdFusion.getLocations", new Object[] {"consumer", "attractions", "ATTR"});
}
catch (ClientStatusException cse)
{
System.out.println(cse);
}
catch (ServerStatusException sse)
{
System.out.println(sse);
}
// Close the connection.
amfConnection.close();
When I run it I get a:
ServerStatusException
data: ASObject(15401342){message=Unable to find source to invoke, rootCause=null, details=null, code=Server.Processing}
HttpResponseInfo: HttpResponseInfo
code: 200
message: OK
Can anyone spot what's wrong?
Thanks for reading!

I ended up using Charles Web Proxy. Sniffing AMF parameters and running my code with -Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=8888
I compare both calls and modify to look alike.
The working code looks like this:
String url = "http://www.theGateWayurl.com";
// Generates the connection to the amf gateway.
AMFConnection amfConnection = new AMFConnection();
// Must register the class that this library will use to load the
// AMF object information.
// The library will read AMF object variables and use setters from
// the java bean stated in this line.
AMFConnection.registerAlias("", new LabelData().getClass().getName());
try {
// Do the connection.
amfConnection.connect(url);
// This page requires a certain headers to function.
// The Content-type is used to sniff with Charles Web Proxy.
amfConnection.addHttpRequestHeader("Content-type", "application/x-amf");
// The Referer is used by the webpage to allow gathering information.
amfConnection.addHttpRequestHeader("Referer", "http://orlandoinfo.com/ws/b2c/sitesearch/customtags/comSearch.swf");
// The rest of the HTTP POST sent by this library is wrapped
// inside a RemotingMessage.
// Prepare the msg to send.
RemotingMessage msg = new RemotingMessage();
// The method called in the server.
msg.setOperation("getLocations");
// Where the request came from. Similar to referer.
msg.setSource("ws.b2c.sitesearch.components.myService");
// The destination is a needed parameter.
msg.setDestination("ColdFusion");
// Create the body with the parameters needed to call the
// operation set with setOperation()
msg.setBody(new Object[] {"consumer", "attractions"});
// This is needed but not used.
msg.setMessageId("xxxxxxxxxx");
// Send the msg.
AcknowledgeMessage reply = (AcknowledgeMessage) amfConnection.call("null", msg);
// Parse the reply from the server.
ArrayCollection body = (ArrayCollection) reply.getBody();
for (Object obj : body) {
LabelData location = (LabelData) obj;
// Do something with the info.
}
} catch (ClientStatusException cse) {
// Do something with the exception.
} catch (ServerStatusException sse) {
// Do something with the exception.
} finally {
amfConnection.close();
}
The LabelData is just a java bean with with two vars: Data and Label.
I tried to comment every line for a better understanding.
Take into account what Stu mention in previous comments about crossdomain.xml to see if you have the rights to do this kind of things.