I'm practicing and trying to parse behance.net to retrieve .jpg files.
First, I tried with JSOUP, but I only receives and JS code without any useful code. Then i tried with selenium:
System.setProperty("webdriver.chrome.driver", "S:\\behance-id\\src\\main\\resources\\chromedriver.exe");
WebDriver driver = new ChromeDriver();
driver.get("https://www.behance.net/gallery/148589707/Hercules-and-Randy");
String str = driver.getPageSource();
And I got same result. Through Google Chrome inspect option I found what I need:
But I cannot acces to this source page via Selenium and JSOUP and other instruments.
I only receive this with <script> tags:
Is it possible?
That page is loading its resources dynamically, after the original HTML, so you should use waits in Selenium. This is the Java example for waiting an element to be loaded in page, from the documentation:
WebDriver driver = new ChromeDriver();
driver.get("https://google.com/ncr");
driver.findElement(By.name("q")).sendKeys("cheese" + Keys.ENTER);
// Initialize and wait till element(link) became clickable - timeout in 10 seconds
WebElement firstResult = new WebDriverWait(driver, Duration.ofSeconds(10))
.until(ExpectedConditions.elementToBeClickable(By.xpath("//a/h3")));
// Print the first result
System.out.println(firstResult.getText());
The documentation can be found at https://www.selenium.dev/documentation/webdriver/waits/
Related
I would like to automate the authentication process in https://appleid.apple.com/ using java webdriver selenium, but html elements of the form doesn't loaded in the DOM
to my knowledge, Selenium webdriver interpret the markup as a browser does.
And It apply the CSS styles, run the JavaScript and the dynamically
rendered content be added to the DOM
Why HTML elements are not loaded in the DOM ?
How can I proceed, to fix it and load all elements in the DOM, exactly like a browser ?
N.B : https://appleid.apple.com/ website use Mustache.JS (logic-less template)
public static void main(String[] args) {
System.setProperty("webdriver.chrome.driver", "chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addArguments("--headless", "--disable-gpu", "--window-size=1920,1200", "--ignore-certificate-errors");
WebDriver driver = new ChromeDriver(options);
driver.get("https://appleid.apple.com/");
waitForPageLoadComplete(driver, 30);
//can't found input name element
WebElement inputName = driver.findElement(By.id("account_name_text_field"));
System.out.println(driver.getPageSource());
}
The element you are trying to find is inside an iFrame. You will need to switch to this iFrame first and then proceed with finding the element as you already have.
driver.switchTo().frame("aid-auth-widget-iFrame");
WebElement inputName = driver.findElement(By.id("account_name_text_field"));
You can find some additional information about switching to iFrames here: https://www.guru99.com/handling-iframes-selenium.html
Is there a way in Selenium (java) to get the "page source" as shown on the elements page (F12) in chrome.
I have to test an application the page is heavily modified by javascript. I already tried:
String html = (String)((JavascriptExecutor)driver).executeScript("return document.getElementsByTagName('html')[0].innerHTML");
and
String html = driver.getPageSource();
but both show the "effective" html.
Try to wait until the expected elements are loaded - see https://www.seleniumhq.org/docs/04_webdriver_advanced.jsp
E.g.
driver.get("http://somedomain/url_that_delays_loading");
WebElement myDynamicElement = (new WebDriverWait(driver, 10)).
.until(ExpectedConditions.presenceOfElementLocated(By.id("myDynamicElement")));
I have successfully saved the source code of a webpage but some webpages dynamically serve the info. That makes it so I cannot get the info that’s on the page.. If I right click and "inspect element" it’s there.. Is there any way to copy what is loaded from that webpage and save it to a text file?
if you're looking for the web page source use this
Create a driver instance
WebDriver driver=new WebDriver();
call the below function when you think the dynamically served content is appeared on the web page
driver.getPageSource();
this should work.
use selenium webdriver to get page source and even you can get the values of a particular element say inputbox, checkbox,dropdown etc from the webpage. Also you can enter the values and submit the form
reate a new instance of the html unit driver
// Notice that the remainder of the code relies on the interface,
// not the implementation.
WebDriver driver = new HtmlUnitDriver();
// And now use this to visit Google
driver.get("http://www.google.com");
System.out.println( driver.getPageSource());
driver.quit();
}
}
refer getPageSource() in Selenium WebDriver(a.k.a Selenium2) using Java
https://code.google.com/p/selenium/wiki/GettingStarted
I have tried, quite a lot and I am not even getting any sort of errors but it is not printing anything (I want it to print the title of the page)
WebDriver driver = new HtmlUnitDriver();
WebElement element = driver.findElement(By.cssSelector("a[href*='Alerts.htm']"));
element.click();
System.out.println(driver.getTitle());
Here is the HTML code (the part I wish to click), there is a title for both , the page I want to click and the current page.
<li title="Alerts"><span>Alerts</span></li>
I am not any errors but it should print the title, which it is not doing.
I have followed many sorts of instructions found here and on the web.
Things I have tried so far:
By locator = By.xpath("//li[#title='Alerts']/a");
WebElement element = driver.findElement(locator);
element.click();
WebElement element = driver.findElement(By.partialLinkText("Alert"));
element.click();
Where am I going wrong?
The title of an HTML document is defined within a <title> tag, typically in the <head> section.
This is the title that the getTitle method returns.
See http://www.w3schools.com/tags/tag_title.asp.
I am not sure of this. But I think HTMLUnitDriver is a headless browser instance. Kindly try in another browser, firefox perhaps.
WebDriver driver = new FirefoxDriver();
You first need to open a page in a browser!
WebDriver driver = new ...
driver.get(url) // THIS LAUNCHES AN ACTUAL BROWSER
// now you can actually do things
driver.findElement ...
driver.getTitle()
driver.quit() // TO DISMISS THE BROWSER
hi i am very new to selenium and i want to collect all those elements that have a particular span id.
Problem: When selenium opens the webpage, it shows(by default) only the first three divs containing approx 50 lines of data per div. I want to fetch information(text) from spans contained in all divs. Is there a way to fetch information from those unloaded divs? If not how can i load those divs by controlling the scroll-bar?
You can use below code for scrolling your page down but fetching information from div without loading will not be possible.
WebDriver driver = = new FirefoxDriver();
JavascriptExecutor jse = (JavascriptExecutor)driver;
jse.executeScript("window.scrollBy(0,300)", "");
OR
JavascriptExecutor js = (JavascriptExecutor)driver;
js.executeScript("window.scrollTo(0,Math.max(document.documentElement.scrollHeight," +
"document.body.scrollHeight,document.documentElement.clientHeight));");