We Keep Coding

Downloading Files with ChromeDriver

I have a project where I need to download an audio file in ChromeDriver. The behavior here is different from in regular Chrome, where if I visit the URL, it'll automatically start downloading a file. If I do the same thing manually in ChromeDriver, it will not download the file.
I've tried different configurations of the chrome options/preferences. I've also found options that worked with old versions of chrome, that no longer work anymore.
Here is one of the better resources I found, but it still didn't work, even with their updated blog post
https://dkage.wordpress.com/2012/03/10/mid-air-trick-make-selenium-download-files/
When I attempt to use his solution, my chromedriver abruptly crashes itself in a non chrome-esque way. It just disappears. Not "Something went wrong" page like you'd normally expect. I end up with Java not being able to find my Session, cause it stopped existing.
Has anyone been successful at downloading files through Selenium webdriver in Chrome? If I need to use another browser, I can.
I'm currently using Chrome Canary.

I have the same problem. One solution that might work is to use another library, that is able to operate outside of the browser. I found these stackoverflow post discussiong this issue:
https://sqa.stackexchange.com/questions/2197/how-to-download-a-file-using-seleniums-webdriver
it contains this blogpost wich gives you some sugestions.
https://blog.codecentric.de/en/2010/07/file-downloads-with-selenium-mission-impossible/
Window automation
The first approach smells like “brute force”: when searching the net for a solution to the problem, you easily end up with suggestions, to control the native window with some window automation software like AutoIt. Means you have to prepare AutoIt such, that it waits for any browser download dialog, the point at which Selenium is giving up, takes control of the window, saves the file, and closes the window. After that Selenium can continue as usual.
This might eventually work, but I found it to be techical overkill. And as it turned out, there was a much simpler solution to the problem.
Change the browsers default behaviour
The second possibility is to change the default behaviour of the browser. When clicking on a PDF for example, the browser should not open a dialog and ask the user what to do with the file, but rather save it without comments and questions in a predefined directory. To accomplish that, a file download has to be initiated manually, saved to disk and marked as the default behaviour for these file types from now on.
Well, that could work. You “only” have to assure that all developers, hudson instances, etc. share the same browser profile. And depending on the amount of different file types, that could be some manual work.
Direct download
Taking a step back, why do we want to download the file with Selenium in the first place? Wouldn’t it be much cooler, to download the file without Selenium, but rather with wget? You would have solved the second problem as you go. Seems a good idea, since wget is not only available for Linux but also for Windows.
Problem solved? Not quite: what about files, that are not freely accessible? What, when I first need to create some state with Selenium in order to access a generated file? The solution seems ok for public files, but is not applicable for all situations.

Jsoup: Getting a link that doesn't show in the HTML

I am working on a little app for myself. I am trying to get a list of links from a site. The site is for example: http://kinox.to/Stream/Prison_Break.html
If you hover over the big window in the middle that says kinox.to best online, it show the link that I want in the bottom left. The problem is if I look at the html file I can't find the link anywhere. I guess it has to do something with the site using JavaScript or Ajax.
Is it possible to somehow get the link using JSoup or are there any other Java libraries that could help me?

I did not look closely into the page you try to load, but here is what I think the problem may be: The link is loaded/generated dynamically via JavaScript. Jsoup does not run JavaScript, so therefore you can't find the link in the html.
Two possible solutions:
1) Use something like selenium webdriver to access the content. The Java bindings allow to remote control a real browser which should have no problems loading the page and running all scripts within. Solution 1 is simple to program, but runs slowly. It may depend on an extern browser program which must be installed on the machine. An alternative to webdriver is the JavaFx webkit engine in case you are on java 8.
2) Analyse the traffic and the JavaScript on the page and find out where the link comes from. This may take a bit of time to find out, but when you succeed you can use Jsoup to get all the data you need. This solution should run much faster than solution 1.

One solution and probably the easiest would be to use Selenium:
WebDriver driver = new FirefoxDriver();
driver.get("http://kinox.to/Stream/Prison_Break.html");
String mylink = driver.findElement(By.cssSelector("#AjaxStream > a")).getText();

How to code an automated bot that can browse and do operations on a webpage. JAVA

need to code a bot that needs to do the following:
Go to a jsp page and search for something by:
writing something on a search box
clicking the search button(submit button)
clicking one of the the resulting buttons/links(same jsp page
with different output)
get the entire html of the new page(same jsp page with different
output)
The 4th one can be done with screen scraping and I do not think I need help with it. But I need some guidance to do the options from 1 to 3. Any links or just some keyword that will help me Google to learn about it will be appreciated. I plan to do this with java.

My suggestion is to use Selenium (http://docs.seleniumhq.org/download/).
Install Selenium IDE in your firefox, and it can record what you do on a website, store it into a script and reply it.
This video (http://www.youtube.com/watch?v=gsHyDIyA3dg) is gonna be helpful if you are a beginner.
And if you want to do it in Java, its easy, just export the scripts in Selenium IDE to JUnit Webdriver code.
Of course you can use Selenium Java webdriver in Java to write your program to operate on website directly.

Selenium automates browsers. That's it. What you do with that power is entirely up to you.
The above steps can be done by using selenium(which is a testing tool in java)

Even points 1 to 3 are screenscraping - you're figuring out (using either manual or automated means) what's there in the page and performing actions on them. You could try exploring the Apache HTTP Client for an easy way to run HTTP commands and get responses.
I hope you're doing this for legitimate means - screenscraping is almost always frowned upon if done without permission.

Java(or any lang) library for offline rendering of web pages?

I am developing a java application. I have scenario to take screen shot of the URL that comes in to the server.
Is there any java(or any lang) browser library to load webpages and get some screenshots of the loaded page. It would be nice if the lib allows DOM traversal.
Update:
java(or any lang): Any other language is not a problem but the library should co-operate with java.
I have tried to setup Qt Jambi and spent a lot of time on this but the result is nothing.
If you provide any concrete material to setup Jambi, it would be appreciative.
I also gave a try to spynner.py. My native language is Java and i thought i could use spynner.py with Jython. But, PyQt cannot be used with Jython. So, i am not expecting any answers related to Python.
Basically, I need a library to do:
Take Screen shot.
Some DOM traversing.
Some Javascript Execution.
and to get the result of the Executed JS code.
Thanks.
I appreciate all the responses. I ended up with phantomjs. It fits well for my needs. Its a command line tool.

Selenium/Webdriver provides all this functionality.
Webdriver provides a simple api allowing you to "drive" a browser instance. Many browsers are supported.
See here for a simple example:
http://seleniumhq.org/docs/03_webdriver.html#getting-started-with-selenium-webdriver
Traversal of the dom using the "By" locators:
Good examples here: http://www.qaautomation.net/?p=388
driver.findElement(By.name("q"));
Execution of Javascript:
http://code.google.com/p/selenium/wiki/FrequentlyAskedQuestions#Q:_How_do_I_execute_Javascript_directly?
WebDriver driver; // Assigned elsewhere
JavascriptExecutor js = (JavascriptExecutor) driver;
js.executeScript("return document.title");
Screenshot capture:
http://seleniumhq.org/docs/04_webdriver_advanced.html#taking-a-screenshot
File scrFile = ((TakesScreenshot)driver).getScreenshotAs(OutputType.FILE);

In java, you should read the following stackoverflow posts :
Programmatic web browser Java library
Take a screenshot of a webpage with JavaScript?
Embed a web browser within a java application
Because you say "or any lang" :
In Python, you have Spynner :
Spynner is a stateful programmatic web browser module for Python with Javascript/AJAX support based upon the QtWebKit framework.
According to the documentation, here's a small snippet :
import spynner
browser = spynner.Browser()
browser.load("http://www.wordreference.com")
browser.runjs("console.log('I can run Javascript!')")
browser.runjs("_jQuery('div').css('border', 'solid red')") # and jQuery!
browser.select("#esen")
browser.fill("input[name=enit]", "hola")
browser.click("input[name=b]")
browser.wait_page_load()
print browser.url, len(browser.html)
browser.close()

This site does the screenshot job:
Tutorial:
http://www.paulhammond.org/webkit2png/
The program:
http://www.paulhammond.org/2009/03/webkit2png-0.5/webkit2png-0.5.txt
Could it be any easier ? :)
There are some other tools mentioned at that page:
"
If you use a mac, but don't like the command line then you may want to try Paparazzi or Little Snapper.
If you use linux you may be more interested in khtml2png, Matt Biddulph's Mozilla screenshot script or Roland Tapken's QT Webkit script.
"
You could use Rhino, Gecko for the javascript execution.
For dom traversal there are many options, but if you are using Rhino you could use jQuery to make it even easier!
Hope that works out for you!

If you need a screenshot, I guess the quality of rendering is important for you.
We had a similar scenario. What we ended up doing is to run firefox on headless mode, actually browse the webpage and get a screen shot in memory. It is not trivial, but I can give you more details if you wanted to go for it.

Selecting folders, not files, in Google Chrome

I'm creating an extension for Google Chrome, so any code has to be compatible with Chrome and Chrome only. In this extension, I need the user to select a folder from his local machine. This simple task is becoming quite a problem. The chrome extensions options page will not run applets, so I couldn't really do Java. It's Google Chrome only so an ActiveX object is out of the question as well. I just need a simple way of selecting a folder(not a file) and passing its path to Javascript. Might this be possible in Flash Actionscript? It seems FileReference and FileReferenceList classes in AS only allow you to choose a file, and not a folder. Is there another possibility besides Flash? All the options page files DO rest on the local users machine, so it's not server side.
Thank you for your time.

You can use the webkit-directory attribute on your element to select directories and get the same sort of result as from the "multiple" attribute.
A demo of this: http://www.thecssninja.com/demo/webkitdirectory/
The chromium bug: http://crbug.com/58977