I have a page crawler developed in Java using Selenium libraries. The crawler goes through a website that launches through Javascript 3 applications which are displayed as HTML in popup windows.
The crawler has no issues when launching 2 of the applications, but on the 3rd one the crawler freezes forever.
The code I'm using is similar to
public void applicationSelect() {
...
//obtain url by parsing tag href attributed
...
this.driver = new HtmlUnitDriver(BrowserVersion.INTERNET_EXPLORER_8);
this.driver.seJavascriptEnabled(true);
this.driver.get(url); //the code does not execute after this point for the 3rd app
...
}
I have also tried clicking on the web element through the following code
public void applicationSelect() {
...
WebElement element = this.driver.findElementByLinkText("linkText");
element.click(); //the code does not execute after this point for the 3rd app
...
}
Clicking on it produces exactly the same result. For the above code, I've made sure I am getting the right element.
Can anyone tell me what could be the problem I'm having?
On the application side, I cannot disclose any information about the html code. I know this makes things harder for trying to solve the problem and for that I apologize in advance.
=== Update 2013-04-10 ===
So, I added the sources to my crawlers and saw where in this.driver.get(url) it was getting stuck on.
Basically, the driver gets lost in an infinite refresh loop. Within a WebClient object instantiated by HtmlUnitDriver, an HtmlPage is loaded which continually refreshes seemingly without end.
Here is the code from WaitingRefreshHandler, which is contained in com.gargoylesoftware.htmlunit:
public void handleRefresh(final Page page, final URL url, final int requestedWait) throws IOException {
int seconds = requestedWait;
if (seconds > maxwait_ && maxwait_ > 0) {
seconds = maxwait_;
}
try {
Thread.sleep(seconds * 1000);
}
catch (final InterruptedException e) {
/* This can happen when the refresh is happening from a navigation that started
* from a setTimeout or setInterval. The navigation will cause all threads to get
* interrupted, including the current thread in this case. It should be safe to
* ignore it since this is the thread now doing the navigation. Eventually we should
* refactor to force all navigation to happen back on the main thread.
*/
if (LOG.isDebugEnabled()) {
LOG.debug("Waiting thread was interrupted. Ignoring interruption to continue navigation.");
}
}
final WebWindow window = page.getEnclosingWindow();
if (window == null) {
return;
}
final WebClient client = window.getWebClient();
client.getPage(window, new WebRequest(url));
}
The instruction "client.getPage(window, new WebRequest(url))" calls WebClient once again to reload the page, only to once more call this very same refresh method. This seems to go on indefinetly, not filling up the memory quickly only because of the "Thread.sleep(seconds * 1000)", which forces a 3m wait before trying again.
Does anyone have any suggestion on how I can work around this issue? I got a suggestion to create 2 new HtmlUnitDriver and WebClient classes which extend the original ones. Then override the relevant methods in order to avoid this problem.
Thanks again.
I solved my eternal refresh problem by creating a do nothing RefreshHandler class:
public class RefreshHandler implements com.gargoylesoftware.htmlunit.RefreshHandler {
public RefreshHandler() { }
public void handleRefresh(final Page page, final URL url, final int secods) { }
}
In addition, I extended the HtmlUnitDriver class and by overriding the method modifyWebClient, I set the new RefreshHandler:
public class HtmlUnitDriverExt extends HtmlUnitDriver {
public HtmlUnitDriverExt(BrowserVersion version) {
super(version);
}
#Override
protected WebClient modifyWebClient(WebClient client) {
client.setRefreshHandler(new RefreshHandler());
return client;
}
}
The method modifyWebClient is a do nothing method created in HtmlUnitDriver exactly for this purpose.
Cheers.
Related
I am trying to build a bot with selenium. the problem is that from time to time the website logging me out without any notice. I know how to detect it, and I know the way to handle it. the problem is that it is not reasonable to check before every line if the server logged me out. this is what I can do:
ChromeDriver driver = new ChromeDriver();
driver.get(url);
Connect(driver, loginData);
if(isConnected(driver) == false)
reconnect(driver, loginData);
driver.findElement(By.id("element-id")).click();
if(isConnected(driver) == false)
reconnect(driver, loginData);
...
But checking if I need to reconnect every line is not a good solution.
I thought about making a thread that checks all the time if I disconnected but I don't know how to pause the main thread until I reconnect to the server when I find out I disconnected
You can have a solution similar to your own, but instead of creating your own wrapper to ChromeDriver, you can use EventFiringWebDriver for that.
you can run the check inside a while loop in a side thread, and if the server disconnects you, stop the main thread. after you reconnect, restart it again.
The website logging you out seems to be time out. As I guess, you are testing in a non-production environment, please look for how to set the time_out = 0 (no-expire).
Thread solutions would be too clumsy and a single non synchronous miss can produce undetectable erroneous results.
I solved it by wrapping ChromeDriver with a class of my own that checks before every operation if a disconnect has detected. if anyone has a better solution I would like to hear. This is my solution:
public java.util.List<WebElement> findElements(By by) throws disconnectException {
if(keepLogin) {
if(checkConnectionOver()) {
throw new disconnectException();
}
}
return driver.findElements(by);
}
public WebElement findElement(By by) throws disconnectException {
if(keepLogin) {
if(checkConnectionOver()) {
throw new disconnectException();
}
}
return driver.findElement(by);
}
public Object executeScript(String script, Object... args) throws disconnectException {
if(keepLogin) {
if(checkConnectionOver()) {
throw new disconnectException();
}
}
return driver.executeScript(script, args);
}
public void get(String url) {
driver.get(url);
}
public String getCurrentUrl() {
return driver.getCurrentUrl();
}
I have created WebDriver tests which run for the same duration.
Some of the tests lest say ‘PopupX’ will appear at 30 seconds.
On some of the test ‘PopupX’ will appear at 60 Seconds
On some of the test ‘PopupX’ will appear at 35 Seconds
You understand the id.
The popup always has a unique Id being the ‘X’ to close the application and even different popups have the same ‘X’ close option.
Does anyone know of a way to constantly search and close ‘A form of continuous polling’ which will close the popup if it was to appear at anytime in any of the test cases?
I know the following method listed below works and can successfully close the popup:
public void closeGiveawayPopup() throws InterruptedException {
WebDriverWait wait = new WebDriverWait(driver, 10);
try {
List<WebElement> elements = wait
.until(ExpectedConditions.presenceOfAllElementsLocatedBy(By.cssSelector(".close")));
for (WebElement element : elements) {
if (element.isDisplayed()) {
element.click();
Thread.sleep(1000);
wait.until(ExpectedConditions.invisibilityOfAllElements(elements));
Thread.sleep(2000);
}
}
} catch (Exception e) {
throw (e);
}
I have tried to add the following method below in the TestNG 'BeforeMethod' annotation but the url dosnt even load:
public void closeGiveawayPopup() throws Exception {
try {
List<WebElement> elements = getDriver().findElements(By.cssSelector(".close"));
for (WebElement element : elements) {
if (element.isDisplayed()) {
element.click();
}
}
} catch (Exception e) {
throw (e);
}
}
Details of the popup have been listed below:
I don't think you are doing anything wrong Phil. #BeforeMethod annotation is the way to go. The only thing that needs improving is to include (if you don't have it already) a #BeforeClass annotation for your browser startup/shutdown respectively. Perhaps, this is why 'the browser does not even open' as you mention in your description above. So something like this:
#BeforeClass
public void initialSetup(String browser){
Webdriver driver = new browser ();
driver.get("https://www.buyagift.co.uk/");
}
#BeforeMethod
public void closeGiveAwayPopup(){
//same as your code
}
#Test(priority=1)
public void makeSureIarrivedOnTheHomePage(){
titleGrabbed = driver.getTitle().toString();
titleExpected = "Experience Days and Gifts from Buyagift";
Assert.assertEquals(titleGrabbed ,titleExpected )
}
#Test(priority=2)
public void clickLoginButton(){
//more actions
}
#AfterClass
public void shutDown(){
driver.quit();
}
The only way I can think of in order to have this 'continuous polling' effect that you are after is to make your #Test really really small (1 step at a time). This way hopefully you can achieve the desired effect. So don't bundle a lot of WebDriver actions under 1 #Test as you are increasing the risk of a pop-up appearing in the meantime.
The only thing that puzzles me is your comment 'the url does not even load'. Are you sure you are grabbing the actual pop-up and not closing the whole window?
If that is the case, or if the above does not work for you, try using Xpath instead of CSS. From your screenshot you need to expand both div[#id='competition_inner_ and outer details'] and find the close button somewhere (probably you are looking for an < input >, < i > or < button > tag, then I will be able to help you more and give you the full Xpath so you can try for yourself.
So something like this:
driver.findElement(By.xpath("//div[#id='competiton_giveaway_popup']//button[#id='close']"));
PS. Another thing that came to mind, double check that you are using actual Testng annotations instead of Junit as it is easy to get confused and misclick sometimes when importing from IDE (I'm looking at you Eclipse!). So you can delete the #BeforeMethod annotation and write it again making sure it points to Testng.
PS2. On your step#9 you have missed the
WebDriverWait wait = new WebDriverWait(driver, 10);
line. I hope this is not the one giving you trouble, please double check and let me know.
Best of luck!
I'm currently working on a JavaFX project. On GUI initialization I want to read some infos out of a HTML document using Selenium and FirefoxDriver. Normally I would use a crawler to get the infos but this document is full of JavaScript so I was only able to get to the infos using Selenium (I know, it's really bad).
Now I've got the problem that this process takes up to 15 seconds and I want to show the progress of Selenium on a JavaFX progress bar. So I've set up a Thread doing all the work and trying to update the GUI but the Thread freezes until Selenium is finished.
This is my attempt:
public class SeleniumThread extends Thread
{
private MainViewController main;
#Override
public void run()
{
try
{
WebDriver driver = new FirefoxDriver();
driver.get("http://---.jsp");
main.getMain().getPrimaryStage().toFront();
main.getPbStart().setProgress(0.1);
WebElement query = driver.findElement(By.id("user"));
query.sendKeys(new String[] {"Username"});
query = driver.findElement(By.id("passwd"));
query.sendKeys(new String[] {"Password"});
query.submit();
driver.get("http://---.jsp");
main.getPbStart().setProgress(0.2);
sleep(1000);
main.getPbStart().setProgress(0.25);
driver.get("http://---.jsp");
main.getPbStart().setProgress(0.4);
sleep(1000);
main.getPbStart().setProgress(0.45);
driver.get("---.jsp");
main.getPbStart().setProgress(0.6);
sleep(1000);
main.getPbStart().setProgress(0.65);
query = driver.findElement(By.cssSelector("button.xyz"));
query.click();
sleep(1000);
main.getPbStart().setProgress(0.85);
System.out.println(driver.getPageSource());
driver.quit();
}
catch(InterruptedException e)
{
// Exception ...
}
}
public MainViewController getMain()
{
return main;
}
public void setMain(MainViewController main)
{
this.main = main;
}
}
MainViewController
public void startup()
{
if(main.getCc().getMjUsername() != null &&
main.getCc().getMjPassword() != null &&
main.getCc().getMjUsername().length() != 0 &&
main.getCc().getMjPassword().length() != 0)
{
SeleniumThread st = new SeleniumThread();
st.setMain(this);
st.setDaemon(true);
st.run();
}
}
I've read that I should use a Worker like Task for it, but I have no clue how to implement it. And I need to pass a parameter to this Task, because I need to set my primaryStage to the front and update the progress bar.
I hope you can understand my problem. I'd be grateful for every help.
You are trying to update the UI from a different thread. The UI can only be updated from the UI thread. To achieve this, wrap the calls to update the progress:
Platform.runLater(() -> {main.getPbStart().setProgress(0.65);});
This will push the update of the UI into the UI thread.
You look to be trying to make JavaFX calls directly from within a background thread, and while I know little about JavaFX, I do know that this is not allowed, that JavaFX calls must be made on the JavaFX Application thread. See Concurrency in JavaFX.
You're not even creating a background thread. You call st.run(); which runs st on the calling thread -- not what you want. You should be calling st.start()!
As a side note, you seem to be extending Thread where you really want to be implementing Runnable. Thus you really should be calling new Thread(myRunnable).start();
Currently learning play! 2.0, I wanted to add an "opening" to my homepage : the first time a user connects to the homepage, an opening would display for a few seconds and then he would be automatically redirected to the real home page. This opening would not be shown again for the rest of the session.
For now that piece of code looks like :
public class Application extends Controller {
private static ScheduledExecutorService executor = executors.newSingleThreadScheduledExecutor();
public static Result home() {
boolean introSeen = Cache.get("introSeen") == null ? false : true;
if(introSeen) {
System.out.println("Cache good, sending you home !");
return ok(home.render(Configs.HOME));
//return redirect("/");
}
else {
Cache.set("introSeen", true, 2*60*60);
Runnable task = new Runnable(){
#Override
public void run() {
home();
executor.shutdown();
}
};
executor.schedule(task, 7, TimeUnit.SECONDS);
return ok(intro.render(Configs.HOME));
}
}
I've added the println to make sure the cache variable registration and the delayed task were working, and they do. But for an unknown reason, the code
return ok(home.render(Configs.HOME));
is not working and the intro view (intro) stays on screen.
I've tried to user redirect instead of "ok" too, but nothing so far.
Thanks for your help :)
Why don't you just use Meta refresh or some JavaScript technique to do that ?
You can use session cookie for checking if next time user should see the opening or target page.
wow I couldn't think of a decent title so I went for the acronym approach :-)
basically I'm working in GWT and I want to notify the user of a panel changing it's text.
I've done this by using a Timer() and CSS
public void flashObject() {
final Timer flashing = new Timer()
{
public void run()
{
flashNewException();
}
};
flashing.scheduleRepeating(rate);
new Timer()
{
public void run()
{
if(stay){
panel.addClass(CSS_HIGHLIGHT);
} else {
panel.removeClass(CSS_HIGHLIGHT);
}
flashing.cancel();
}
}.schedule(length);
}
private void flashNewException() {
if(on){
// GWT.log("flashin");
panel.addClass(CSS_HIGHLIGHT);
on = false;
} else {
// GWT.log("stop flashin");
panel.removeClass(CSS_HIGHLIGHT);
on = true;
}
}
So this basically take's a panel add's and removes the CSS class allowing the panel to 'flash'.
The trouble is if I run this in FF alongside the rest of my code FF will sometimes crash (I have another two timer's running elsewhere). I'm also running GWT-EXT.
I appreciate this may not be the crux of my problem but I'd like to ask, do you think this is the correct way to flash a panel in GWT / GWT-Ext? How optimised is GWT to convert Timer's into javascript and how capable is FireFox at dealing with multiple Timers?
As an extra point, if I kill 'plugin-container.exe' from my task list FireFox will recover...
I've took this as a solid bit of coding and I believe my GWT error's where elsewhere