We Keep Coding

Why is my Jsoup Code not Returning the Correct Elements?

I am working on an app in Android Studio and am having some trouble web-scraping with JSoup. I have successfully connected to the webpage and returned some basic elements to test the library, but now I cannot actually get the elements I need for my app.
I am trying to get a number of elements with the "data-at" attribute. The weird thing is, a few elements with the "data-at" attribute are returned, but not the ones I am looking for. For whatever reason my code is not extracting all of the elements that share the "data-at" attribute on the web page.
This is the URL of the webpage I am scraping:
https://express.liatoyotaofcolonie.com/inventory?f=dealer.name%3ALia%20Toyota%20of%20Colonie&f=submodel%3ACamry&f=trim%3ALE&f=year%3A2020
The method containing the web-scraping code:
#Override
protected String doInBackground(Void... params) {
String title = "";
Document doc;
Log.d(TAG, queryString.toString());
try {
doc = Jsoup.connect(queryString.toString()).get();
Elements content = doc.select("[data-at]");
for (Element e: content) {
Log.d(TAG, e.text());
}
} catch (IOException e) {
Log.e(TAG, e.toString());
}
return title;
}
The results in Logcat
The element I want to retrieve
One of the elements that is actually being retrieved

This is because some of the content - including the one you are looking for - is created asyncronously and is not present in initial DOM (Javascript ;))
When you view the source of the page you will notice that there is only 17 data-at occurences, while running document.querySelector("[data-at]") 29 nodes are returned.
What you are able to get in the JSoup is static content of the page (initial DOM). You wont be able to fetch dynamically created content as you do not run required JS scripts.
In order to overcome this, you will have to either fetch and parse required resources manually (eg trace what AJAX calls are made by the browser) or use headless browser setup. Selenium + headless Chrome should be enough.
Letter option will allow you to scrape ANY posible web application, including SPA apps, which is not possible using plaing Jsoup.

I don't quite know what to do about this, but I'm going to try one more time... The "Problematic Lines" in your code are these:
doc = Jsoup.connect(queryString.toString()).get();
Elements content = doc.select("[data-at]");
It is the queryString that you have requested - the URL points to a page that contains quite a bit of script code. When you load up a browser and click the button (or menu-option) that reads: "View Source", the HTML you see is not the same exact HTML that is broadcast to and received by JSoup.
If the HTML that is broadcast contains any <SCRIPT TYPE="text/javascript"> ... </SCRIPT> in it (and the named URL in your question does), AND those <SCRIPT> tags are involved in the initial loading of the page, then JSoup will not know anything about it... It only parses what it receives, it cannot process any dynamic content.
There are four ways that I know of to get the "Post Script Loaded" version of the HTML from a dynamic web-page, and I will type them here, now. The first is likely the most popular method (in Java) that I have heard about on Stack Overflow:
Selenium This Answer will show how the tool can run Java-Script. These are some Selenium Docs. And then there is this page right here has a great "first class" for using the tool to retrieve post-script processed HTML. Again, there is no way JSoup can retrieve HTML that is sent to the browser by script (JS/AJAX/Angular/React) since it just a parser.
Puppeteer This requires running a language called Node.js Perhaps calling a simple Node.js program from Java could work, but it would be a "Two Language" solution. I've never used it. Here is an answer that shows getting, sort of, what you are trying to get... The HTML after the script.
WebView Android Java Programmers have a popular class called "WebView" (documented here), that I have recently been told about (yesterday ... but it has been out for years) that will execute script in a browser, and return the HTML. Here is an answer that shows "JavaScript Injection" to retrieve DOM Tree elements from a "WebView" instance (which is how I was told it was done)
Splash My favorite tool, which I don't think anyone has heard of, but has been the simplest for me... So there is an A.P.I. called the "Splash API". Here is their explanation for a "Java-Script Rendering Service." Since this one I have been using... I'll post a code snippet that shows how "Splash Tool" can retrieve post-script processed HTML below.
To run the Splash API (only if you have access to the docker loading program) ... You start a Splash Server as below. These two lines are typed into a GCP (Google Cloud Platform) Shell instance, and the server starts right up without any configurations:
Pull the image:
$ sudo docker pull scrapinghub/splash
Start the container:
$ sudo docker run -it -p 8050:8050 --rm scrapinghub/splash
In your code, just prepend the String to your URL's:
"http://localhost:8050/render.html?url="
So in your code, you would use the following command (instead), and the script would (more likely) load all the HTML Elements that you are not finding:
String SPLASH_URL = "http://localhost:8050/render.html?url=";
doc = Jsoup.connect(SPLASH_URL + queryString.toString()).get();

Java (Netbeans) : Connecting Google Search To JTextField

Is there any way to connect google search to jtextfield, i want to create a software that can search directly through a jtextfield and the results would show up on a different JPanel. i found a code that will make it search on google directly but you need an existing browser to do it, i just want the results to appear on a different panel, for example when you search something on the textfield the results will popup on another panel.
this is the code that i found but it needs an existing browser i want the software to be standalone thats all thank you:
try {
String search = "#q="+jTextField.getText().toString().trim();
search = search.replaceAll(" ","+");
String url = "http:////www.google.com//"+search;
java.awt.Desktop.getDesktop().browse(java.net.URI.create(url));
}
catch (java.io.IOException e) {
System.out.println(e.getMessage());
}

If you're not interested in writing your own browser you could use this.

How can I detect a close in the browser tab?

Hello I am looking for information on the close tab (not browser) event if there is one in java for a applet. I am wondering if there is an event for that or a way to check a way to check for that. I would like to just capture the event and make a little popup box , stating Your session will expire or something along those lines. Is that possible at all or to a point with java or Javascript?
UPDATE: okay with the information you guys pointed me into i was able to get information on a simple enough javascript. Now it is working fine in IE , Chrome and Firefox but for some reason Safari 5.1.7 isn't liking the code. Not sure why. Here is the code if it helps.
jQuery(function() {
var hiddenBtn = document.getElementById("javaform:browserCloseSubmit");
try{
opera.setOverrideHistoryNavigationMode('compatible');
history.navigationMode = 'compatible';
}catch(e){}
//Sends the information to the javaBean.java file.
function ReturnMessage()
{
return hiddenBtn.click();
}
//UnBind Function
function UnBindWindow()
{
jQuery(window).unbind('beforeunload', ReturnMessage);
}
//Bind Exit Message Dialogue
jQuery(window).bind('beforeunload', ReturnMessage);
});

You have the onBeforeUnload event you can catch in JavaScript. See here.

Use window.onbeforeunload
window.onbeforeunload = function () {
return "Are you sure you want to exit?";
};
Note that it will also end in Are you sure you want to leave this page (or are you sure you want to reload this page if you are reloading)

Open and maximise a browser window in OATS Java (not javascript)

I am a tester and just installed oracle application test suite to use testing eBus apps
Anyway the only language it supports for coding test scripts (I don't want to use the recorder for a number of reasons). The problem I am having is that everything I search or google is javascript not java (even googling with -script I still ended up looking at javascript. This just gets rejected by the oats editor
The only other examples I have seen, appear to be defining a variable then setting the value of that variable as the window they want to maximize. Aside from the fact that my java skills are not up to doing that - I do not need to do this for a newly opened browser window do I? (The assumption is that this will be the only browser window open (ie test is executed with browser closed)
Is there any easy way to do this?
Below is the very simple initiate of the browser which is generated from a recording plus part of the first step which loads the url the test starts at: (I realize the first step is not complete below -I didn't paste it all, just enough to hopefully allow someone to show me what I need to edit to force the browser to load maximized, or maximize it immediately after loading?
public void initialize() throws Exception {
browser.launch();
}
/**
* Add code to be executed each iteration for this virtual user.
*/
public void run() throws Exception {
beginStep("[1] Login (/RF.jsp)", 0);
{
web
.window(2,
"/web:window[#index='0' or #title='about:blank']")
.navigate(
"http://somepageiwantolaunch");
web.window(4, "/web:window[#index='0' or #title='Login']")
.waitForPage(null);

I am not sure whether you already got the answer for this.. if not this code should help you
browser.launch();
DOMBrowser currentExecutionBrowser = web.window("/web:window[#index='0' or #index='1']");
currentExecutionBrowser.maximize();
Let me know if this helps!

There is a function in the Oracle Functional Tester API Reference which has a build in function called object.WindowState It says you can get or set using this function and it has values
0 - Normal, 1- minimized and 2-maximised.
Only issue is that these examples look more like VB than Javascript but presumably there is a similar function built into to the Oracle libraries for Java.
I did a quick search for Oracle Openscript API and came up with this link which asks for the same thing. They suggest using Help->Search from within the openscript application and then searching for "openscript API" which should provide a list of the functions available.
Hope that helps.

To Maximize browser in OATS, follow the below code
Open script ha in built methods which helps coding easy
browser.launch();
web.window(12, "/web:window[#index='0' or #title='about:blank']").navigate("http://www.google.com/");
web.window(12, "/web:window[#index='0' or #title='about:blank']").maximize();
for more OATS Tips/Tricks follow here
http://www.testinghive.com/category/oracle-application-testing-suite-tips

If it is the only browser window open, you can use the below code. It must be used with caution since the code maximizes any window that is open above the browser window.
try {
Robot a = new Robot();
a.keyPress(KeyEvent.VK_ALT);
a.keyPress(KeyEvent.VK_SPACE);
a.keyRelease(KeyEvent.VK_SPACE);
a.keyRelease(KeyEvent.VK_ALT);
a.keyPress(KeyEvent.VK_X);
a.keyRelease(KeyEvent.VK_X);
} catch (AWTException e) {
}

How to deal with file uploading in test automation using selenium or webdriver

I think that everybody who uses Webdriver for test automation must be aware of its great advantages for web development.
But there is a huge issue if file uploading is part of your web flow. It stops being test automation. The security restriction of browsers (invoking file selection) practically makes it impossible to automate tests.
Afaik the only option is to have Webdriver click the file upload button, sleep the thread, have developer/tester manually select the file, and then do the rest of the web flow.
How to deal with this, is there a workaround for it? Because it really can't be done like this. It wouldn't make sense.
This is the only case I know of when browser security restrictions do not apply:
<script language=javascript>
function window.onload(){
document.all.attachment.focus();
var WshShell=new ActiveXObject("WScript.Shell")
WshShell.sendKeys("D:\MyFile.doc")
}
</script>

Webdriver can handle this quite easily in IE and Firefox. Its a simple case of finding the element and typing into it.
driver = webdriver.Firefox()
element = driver.find_element_by_id("fileUpload")
element.send_keys("myfile.txt")
The above example is in Python but you get the idea

Using AWT Robots is one option, if you're using Java, which you are. But it's not a good option, it is not very dependable, and not clean at all. Look here
I use HttpClient and run a few tests outside of Selenium. That's more dependable and cleaner.
See the code below. You'll need more exception handling and conditionals to get it to suit your job.
HttpClient c = new HttpClient();
String url = "http://" + cargoHost + ":" + cargoPort + contextPath + "/j_security_check";
PostMethod post = new PostMethod(url);
post.setParameter("j_username", username);
post.setParameter("j_password", password);
c.executeMethod(post);
url = "http://" + cargoHost + ":" + cargoPort + contextPath + "/myurl.html";
MultipartPostMethod mPost = new MultipartPostMethod(url);
String fileNameWithPath = this.getClass().getClassLoader().getResource(filename).getPath();
File f1 = new File(fileNameWithPath);
mPost.addParameter(elementName, f1);
mPost.addParameter("action", "upload");
mPost.addParameter("ajax", "true");
c.executeMethod(mPost);
mPost.getResponseBodyAsString();

The suggestion of typing into the text box works only if the textbox is enabled.
Quite a few applications force you to go through the file system file browser for obvious reasons.
What do you do then?
I don't think the WebDriver mavens thought of just presenting keys into the KeyBoard buffer (this used to be a "no brainer" in earlier automation days)
===
After several days of little sleep, head banging and hair pulling I was able to get some of the Robot-based solution suggested here (and elsewhere).
The problem i encountered was that the dialog text box that was populated with the correct file path and name could not respond to the KeyPress/Release Events of terminating the file name with VK_ENTER as in:
private final static int Enter = KeyEvent.VK_ENTER;
keyboard.keyPress(Enter);
keyboard.keyRelease(Enter);
What happens is that the file path and file name are typed in correctly but the dialog remains opened - against my constant hoping and praying that the key emulation will terminate it and get processed by the app under testing.
Does anyone know how to get this robot to behave a bit better?

Just thought I'd provide an FYI to author's original post of using ActiveX. Another workaround would be to integrate with desktop GUI automation tools to do the job. For example, google "Selenium AutoIt". For a more cross-platform solution, consider tools like Sikuli over AutoIt.
This of course, is not considering WebDriver's support for uploads on IE & Firefox via SendKeys, or considering for other browsers where that method doesn't work.

After banging my head on this problem for far too many hours, I wanted to share with the community that Firefox 7.0.1 seems to have an issue with the FirefoxDriver sendKeys() implementation noted above (at least I couldn't get it to work on my Windows 7 x64 box), I haven't found a workaround, but updating to Firefox 8.0.1 seems to have fixed the problem. For those of you wondering, it's also possible to use Selenium RC to solve this problem (though you need to account for all of your target operating systems and the native key presses required to interact with their file selection dialogs). Hopefully the issues I had to work around save other people some time, in summary:
https://gist.github.com/1511360

If you have your are using a grid, you could make the folder of the testfiles open for sharing.
This way you could select the upload input field and set its value to \\pc-name\myTestFiles
If you're not, you should go with local files on each system.