I need to work on ajax response, that is one of responses received upon visiting a page. I use selenium dev tools and java. I create a listener, that intercepts a specific request and then I want to work on response it brings. However I need to setup static wait, or else selenium don't have time to save RequestId. I read Chrome Dev Tools documentation, but it's a new thing for me. I wonder if there is a method that would allow me to wait for this call to be completed, other than the static wait.
Here is my code:
#Test(groups = "test")
public void x() throws InterruptedException, JsonProcessingException {
User user = User.builder();
ManageAccountStep manageAccountStep = new ManageAccountStep(getDriver());
DashboardPO dashboardPO = new DashboardPO(getDriver());
manageAccountStep.login(user);
DevTools devTools = ((HasDevTools) getDriver()).maybeGetDevTools().orElseThrow();
devTools.createSessionIfThereIsNotOne();
devTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty()));
// end of boilerplate
final RequestId[] id = new RequestId[1];
devTools.addListener(Network.responseReceived(), response -> {
log.info(response.getResponse().getUrl());
if (response.getResponse().getUrl().contains(DESIRED_URL)){
id[0] = response.getRequestId();
}
});
dashboardPO
.clickLink(); // here is when my DESIRED_URL happens
Utils.sleep(5000); // Something like Thread.sleep(5000)
String responseBody = devTools.send(Network.getResponseBody(id[0])).getBody();
// some operations on responseBody
devTools.clearListeners();
devTools.disconnectSession();
}
If I don't use 5 seconds wait id variable gets never assigned and I null pointer exception requestId is required. During these 5 seconds log.info prints all api calls that are happening and it almost always finds my id. I would like to refrain from static wait though. I am thinking about something similiar to maybe jQuery.active()==0, but my page doesn't use jQuery.
You may try custom function Explicit Wait. Something like this:
public String getResponseBody(WebDriver driver, DevTools devTools) {
return new WebDriverWait(driver,5)
.ignoring(NullPointerException.class)
.until(driver ->
devTools.send(Network.getResponseBody(id[0])).getBody());
}
So, it won't wait for all 5 seconds. The moment it got the data, it would come of out of the until method. Also add whichever Exception that was coming up here.
Has put these lines of code as separate method because, devTools object is locally defined. In order to use them inside this anonymous inner function, it has to be final or effectively final.
I seem to run into this issue when running tests in parallel (and headless) and trying to capture the requests and responses, I get:
{"No data found for resource with given identifier"},"sessionId" ...
However, now .until seems to only take ExpectedCondition
So a similar solution (to the accepted answer), but without using "WebDriverWait.until" that I use is:
public static String getResponseBody(DevTools devTools, RequestId id) {
String requestPostData = "";
LocalDateTime then = LocalDateTime.now();
String err = "";
Integer it = 0;
while (true) {
err = "";
try{requestPostData = devTools.send(Network.getResponseBody(id)).getBody();} catch( Exception e){err = e.getMessage();};
if (requestPostData != null && !requestPostData.equals("")) {break;}
if (err.equals("")) {break;} // if we don't have an error message, its quite possible the responseBody really is an empty string
long timeTaken = ChronoUnit.SECONDS.between(then, LocalDateTime.now());
if (timeTaken >= 5) {requestPostData = err + ", timeTaken:" + timeTaken; break;}
if(it > 0) {TimeUnit.SECONDS.sleep(it);} // I prefer waiting longer and longer, avoiding stack overflows
it++;
}
return requestPostData;
}
It just loops until it doesn't error, and returns the string back as soon as it can (but I actually set timeTaken >= 60 due to many parallel requests)
Related
I work on university project in java. I have to download attachments from new emails using GMAIL API.
I successfully connected to gmail account using OAuth 2.0 authorization.
private static final List<String> SCOPES = Collections.singletonList(GmailScopes.GMAIL_READONLY);
I tried to get unseen mails using
ListMessagesResponse listMessageResponse = service.users().messages().list(user).setQ("is:unseen").execute();
listMessageResponse is not null but when I call method .getResultSizeEstimate() it returns 0
also I tried to convert listMessageResponse to List < Message > (I guess this is more usable) using
List<Message> list = listMessageResponse.getMessages();
But list launches NullPointerException
Then tried to get each attachment with
for(Message m : list) {
List<MessagePart> part = m.getPayload().getParts();
for(MessagePart p: part) {
if(p.getFilename()!=null && p.getFilename().length()>0) {
System.out.println(p.getFilename()); // Just to check attachment filename
}
}
}
Is my approach correct (if not how to fix it) and how should I download those attachments.
EDIT 1:
Fixed q parameter, I mistakenly wrote is:unseen instead of is:unread.
Now app reaches unread mails successfully.
(For example there was two unread mails and both successfully reached, I can get theirs IDs easy).
Now this part trows NullPointerException
List<MessagePart> part = m.getPayload().getParts();
Both messages have attachments and m is not null (I get ID with .getID())
Any ideas how to overcome this and download attachment?
EDIT 2:
Attachments Downloading part
for(MessagePart p : parts) {
if ((p.getFilename() != null && p.getFilename().length() > 0)) {
String filename = p.getFilename();
String attId = p.getBody().getAttachmentId();
MessagePartBody attachPart;
FileOutputStream fileOutFile = null;
try {
attachPart = service.users().messages().attachments().get("me", p.getPartId(), attId).execute();
byte[] fileByteArray = Base64.decodeBase64(attachPart.getData());
fileOutFile = new FileOutputStream(filename); // Or any other dir
fileOutFile.write(fileByteArray);
fileOutFile.close();
}catch (IOException e) {
System.out.println("IO Exception processing attachment: " + filename);
} finally {
if (fileOutFile != null) {
try {
fileOutFile.close();
} catch (IOException e) {
// probably doesn't matter
}
}
}
}
}
Downloading working like charm, tested app with different type of emails.
Only thing left is to change label of unread message (that was reached by app) to read. Any tips how to do it?
And one tiny question:
I want this app to fetch mails on every 10 minutes using TimerTask abstract class. Is there need for manual "closing" of connection with gmail or that's done automatically after run() method iteration ends?
#Override
public void run(){
// Some fancy code
service.close(); // Something like that if even exists
}
I don't think ListMessagesResponse ever becomes null. Even if there are no messages that match your query, at least resultSizeEstimate will get populated in the resulting response: see Users.messages: list > Response.
I think you are using the correct approach, just that there is no message that matches your query. Actually, I never saw is:unseen before. Did you mean is:unread instead?
Update:
When using Users.messages: list only the id and the threadId of each message is populated, so you cannot access the message payload. In order to get the full message resource, you have to use Users.messages: get instead, as you can see in the referenced link:
Note that each message resource contains only an id and a threadId. Additional message details can be fetched using the messages.get method.
So in this case, after getting the list of messages, you have to iterate through the list, and do the following for each message in the list:
Get the message id via m.getId().
Once you have retrieved the message id, use it to call Gmail.Users.Messages.Get and get the full message resource. The retrieved message should have all fields populated, including payload, and you should be able to access the corresponding attachments.
Code sample:
List<Message> list = listMessageResponse.getMessages();
for(Message m : list) {
Message message = service.users().messages().get(user, m.getId()).execute();
List<MessagePart> part = message.getPayload().getParts();
// Rest of code
}
Reference:
Class ListMessagesResponse
Users.messages: list > Response
I'm listening for connection changes through events pluging ("amq.rabbitmq.event", "connection.#").
It works properly so I'm adding at java side two additional parameters as clientproperties, to get the identity of the user that connects or disconnect.
However at c# side I can only access these properties as a list of byte[], and not sure on how to convert it to a Dictionary or so..
If I print all entries
if (args.BasicProperties.Headers.TryGetValue("client_properties", out object value))
{
var items = value as List<object>;
foreach(var item in items)
{
Console.WriteLine($"{item.GetType().ToString()}");
var bytes = item as byte[];
result.Add(Encoding.UTF8.GetString(bytes));
}
}
I can see this:
{<<"platform">>,longstr,<<"Java">>}
{<<"capabilities">>,table,[{<<"connection.blocked">>,bool,true},{<<"basic.nack">>,bool,true},{<<"exchange_exchange_bindings">>,bool,true},{<<"authentication_failure_close">>,bool,true},{<<"publisher_confirms">>,bool,true},{<<"consumer_cancel_notify">>,bool,true}]}
{<<"groupId">>,longstr,<<"1e6e935f0d4d9ec446d67dadc85cbafd10d1a095">>}
{<<"information">>,longstr,<<"Licensed under the MPL. See http://www.rabbitmq.com/">>}
{<<"version">>,longstr,<<"4.8.1">>}
{<<"copyright">>,longstr,<<"Copyright (c) 2007-2018 Pivotal Software, Inc.">>}
{<<"product">>,longstr,<<"RabbitMQ">>}
What kind of object format is and how can I parse this?:
{<<id>>,type,<<value>>}
Apparently ( as for an answer I got on Rabbit client google group for this questions ), client_properties is something that's not created to being read by the receiving party..
However is a really good way to have something like LWT ( Last Will and Testament ), then I am using it at the minute doing the parse by myself.
if (args.BasicProperties.Headers.TryGetValue("client_properties", out object value))
{
var items = value as List<object>;
foreach (var item in items)
{
var bytes = item as byte[];
//{<<id>>, type, <<value>>}
String itemStr = Encoding.UTF8.GetString(bytes);
var parts = itemStr.Split(",");
var key = CleanErlangString(parts[0]);
var value = CleanErlangString(parts[2]);
// Do things with key/value
}
}
ClearErlangFunction
private static string CleanErlangString(string toClean)
{
return toClean
.Replace("{", "").Replace("}", "")
.Replace("\"", "")
.Replace("<<", "").Replace(">>", "");
}
What I am doing to use it as LWT, is setting a custom property on client side and then obtaining it while reading events at "amq.rabbitmq.event", "connection.#". With that I know who have disconnected and even process something as LWT with my core server.
I hope this helps someone :)
I am trying to get the index status for a very large index using the Java API. In the background, I am updating the same index with large amounts of data
I have tried
IndicesStatsResponse indicesStatsResponse = client.admin().indices()
.prepareStats(index_name).all().execute().actionGet();
but this does not return the correct size because it is a very large index. However using the REST call, I got the correct answer.
GET /index_name/_stats
Perhaps I need to use some kind of listener mechanism, so I tried
IndicesStatsRequest req = new IndicesStatsRequest();
req.all();
client.admin().indices().stats(req, new ActionListener<IndicesStatsResponse>()
{
#Override
public void onResponse(IndicesStatsResponse response)
{
long s = response.getIndex(indexName).getTotal().getStore().getSizeInBytes();
System.out.println(indexName + " " + s);
}
..
});
but this threw org.elasticsearch.common.util.concurrent.EsRejectedExecutionException
I also tried
IndicesStatsRequest req = new IndicesStatsRequest();
req.all();
ActionFuture<IndicesStatsResponse> statsResponseFeature = client.admin().indices().stats(req.clear().flush(true).refresh(true));
IndicesStatsResponse statsResponse = statsResponseFeature.get(1, TimeUnit.MINUTES);
But this did not retrieve any useful information.
Strangely enough, when I run it in debug mode in Eclipse, it works perfectly. So maybe there is some flush mechanism I am missing.
What is the way out?
I am trying to develop a record and playback tool using selenium webdriver like the way Selenium IDE does. I started withCchrome browser, tried different approaches. Few of them are here:
Tried creating a JavaScript with event listeners and tried executing it using JavascriptExecutor. In the JS script I have implicit wait to return some value. Sample code:
var flag = 0;
var elementId;
window.addEventListener("click", function (e) {
elementTagName=e.target.id;
alert(elementTagName);
flag++;
});
var timer = setInterval(function () {
myTimer();
}, 1000);
function myTimer() {
if(flag == 0){
document.getElementById("demo").innerHTML=flag;
} else {
clearInterval(timer);
return elementId; //Returning the element ID which was clicked
}
}
But now the problem is, webdriver code written in java(shown below) is not waiting for the return. Same code works fine when I run it individually.
Object response = ((JavascriptExecutor) driver).executeScript(script);
if (null != response) {
System.out.println((String) response);
}
Any other way I can do it?
Instead of returning the JS value, you can store it in a JS variable, by changing this line:
return elementId;
to this:
retVal = elementId;
Then, whenever you would have accessed response in Java, execute JS to access the JS var on the page instead:
// execute JS functions from your question above
((JavascriptExecutor) driver).executeScript(script);
// wait however long needed for those functions to complete
Thread.sleep(1000);
// get result from page
String response = (String)((JavascriptExecutor) driver).executeScript("return retVal");
New to the JVM, working with Scala and Play 2.0
I'm converting a legacy application over to Play, one that requires payment processing via Authorize.net. Looking through java.net.URL source, there are numerous potential points of failure. Given the interface I've written below, where would you implement try/catch blocks? I'll need to adapt method signatures accordingly, probably returning an Either[Error, Success] to calling client code
import java.net.{URL, URLEncoder}
import java.io.{BufferedReader, DataOutputStream, InputStreamReader}
import javax.net.ssl._
trait Authnet {
private val prodUrl = "https://secure.authorize.net/gateway/transact.dll"
private val testUrl = "https://test.authorize.net/gateway/transact.dll"
protected def authNetProcess(params: Map[String,String]) = {
val(conn, urlParams) = connect(params)
val request = new DataOutputStream( conn.getOutputStream )
request.write(urlParams.getBytes)
request.flush()
request.close()
val response = new BufferedReader(new InputStreamReader(conn.getInputStream))
val results = response.readLine().split("\\|")
response.close()
results.toList
}
private def connect(params: Map[String,String]) = {
val urlParams = (config ++ params) map { case(k,v) =>
URLEncoder.encode(k, "UTF-8") + "=" + URLEncoder.encode(v, "UTF-8")
} mkString("&")
lazy val url = if (isDev) new URL(testUrl) else new URL(prodUrl)
val conn = url.openConnection
conn.setDoOutput(true)
conn.setUseCaches(false)
(conn, urlParams)
}
private val config = Map(
'x_login -> "...",
'x_tran_key -> "...",
...
)
}
Stick to the thumb rule:
Only catch an exception if you must handle it.
There is no sharp definition for "must handle" but it means you should resist the urge to catch an exception because you can just to throw a different exception.
The "must handle" is mainly defined by how your application should work or other dependencies.
If the application requires to display an error to the user instead of aborting with an exception, then it's a must.
In that case catching the excpetion adds also some meaningful processing.
If an API requires to throw a different exception, then it's a must, but the APIs definition is possibly not sound.
I am always questioning the added value of replacing an exception with just another exception.
Applying this to your example:
Would it add some value to catch an exception from connect() in authNetProcess()?
No! There is no way to handle that exception inside of connect(). So its ok to leave that exception to the caller of authNetProcess. There you could provide different handling based on the kind of the exception.
EDIT
Well, if any part of the connection/stream process fails, the transaction is hosed, so silly to only capture error on opening of the connection. I'm just wrapping the whole transaction in a catching (operation) option block, and leaving it at that; I'm not too concerned re: the exact cause of the error (whatever it is gets logged) as it is transient, so catch it, have the user try again; if error persists, contact us...
ORIGINAL
OK, well, given the up votes and lack of commentary to-date, the only conclusion I can draw is...nobody around here knows what they're doing! heh, heh, joking ;-)
Although I'm new to the JVM, try/catch/finally bloat is getting old fast; via the wonders of Scala type inference, I have abstracted away general error handling into concise implementations:
catching ( operation ) option
catching ( operation ) either
Unless I receive feedback otherwise, for now I'm copping out by just catching connection creation (I believe, in this case, the most likely error condition). Here's the new implementation:
protected def authNetProcess(params: Map[String,String]) = {
connect() match {
case Some(conn) =>
val request = new DataOutputStream(conn.getOutputStream)
request.write(getUrlParams(params).getBytes)
request.flush()
request.close()
val response = new BufferedReader(new InputStreamReader(conn.getInputStream))
val results = response.readLine().split("\\|")
response.close()
results.toList
case None => List[String]()
}
}
private def connect() = {
lazy val url = if (isDev) new URL(testUrl) else new URL(prodUrl)
catching ( url.openConnection ) option match {
case Some(conn) =>
conn.setDoOutput(true)
conn.setUseCaches(false)
//conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded")
Some(conn)
case None => None // connection failed
}
}
I suppose a more rigorous approach would be to extract all potential error conditions into maybeWorked Option operations, and then wrap them all up in a for comprehension. That's probably the proper/responsible approach...but only so many hours in a day, will revisit this later
feedback appreciated!