I want test if a web site URL will redirect to secured site or not. For example, if I type example.com in the address bar, it should redirect to https://example.com.
From Selenium, I tried using both get("") and navigate("") with no luck. It shows an exception as wrong URL. How can I test this or proceed another way?
Even Javascript will not work.
It's very easy to achieve this using get() & getCurrentUrl(). You should type the actual URL, like www.example.com instead of just using example.com. Even tough you type the URL without the www, the browser makes that change automatically but not Selenium, hence it throws an exception. Try something like this:
driver.get("www.example.com");
//add wait for page to load completely
if(driver.getCurrentUrl().startsWith("https"))
System.out.println("Success");
else
System.out.println("Failure");
Related
I need to download files using headless web browser in Java. I checked HtmlUnit where I was able to download file in some simple cases but I was not able to download when Ajax initialized downloading (actually it is more complicated as there are two requests, the first one download the URL where the second request actually download file from the given URL). I have replaced HtmlUnit with Selenium. I already checked two WebDrivers, HtmlUnitDriver and ChromeDriver.
HtmlUnitDriver - similar behaviour like HtmlUnit
ChromeDriver - I am able to download files in visible mode but when I turned on headless mode, files are no longer downloaded
ChromeOptions lChromeOptions = new ChromeOptions();
HashMap<String, Object> lChromePrefs = new HashMap<String, Object>();
lChromePrefs.put("profile.default_content_settings.popups", 0);
lChromePrefs.put("download.default_directory", _PATH_TO_DOWNLOAD_DIR);
lChromeOptions.setExperimentalOption("prefs", lChromePrefs);
lChromeOptions.addArguments("--headless");
return new ChromeDriver(lChromeOptions);
I know that downloading files in headless mode is turned off because of security reasons but there must be some workaround
I used 2.28 httpunit before, few minutes ago I started to work with 2.29 but still it seems that Ajax function stops somewhere. This is the way I retrieve data after click and expect a file data: _link.click().getWebResponse().getContentAsStream()
Does WebConnectionWrapper shows all the requests/responses that are made on the website? Do You know how can I debug this to have better insight? I see that the first part of the Ajax function after link is clicked is being properly called (there are 2 http requests in this function). I even tried to create my custom http request to retrive data/file after first response is fetched inside WebConnectionWrapper -> getResponse but it returns 404 error which indicates that this second request had been somehow done but I dont see any log/debug information neither in _link.click().getWebResponse().getContentAsStream() nor WebConnectionWrapper -> getResponse()
Regarding HtmlUnit you can try this:
Calling click() on a dom element is a sync call. This means, this returns after the response of this call is retrieved and processed. Usually all the JS libs out there doing some async magic (like starting some processing with setTimeout(,10) for various (good) reasons. Your code will be aware of this.
A better approach is to do something like this
Page page = _link.click();
webClient.waitForBackgroundJavaScript(1000);
Sometimes the Ajax requests are doing an redirect to the new content. We have to address this new stuff by checking the current window content
page = page.getEnclosingWindow().getEnclosedPage();
Or maybe better
In case of downloads the (binary) response might be opened in a new window
WebWindow tmpWebWindow = webClient.getCurrentWindow();
tmpWebWindow = tmpWebWindow.getTopWindow();
page = tmpWebWindow.getEnclosedPage();
This might be the response you are looking for.
page.getWebResponse().getContentAsStream();
Its a bit tricky to guess what is going on with your web application. If you like you can reach me via private mail or discuss this in the HtmlUnit user mailing list.
I want to log in to a https website using Jsoup and make subsequent calls 3-4 services to check whether a job is done or not.
public class JSOUPTester {
public static void main(String[] args){
System.out.println("Inside the JSOUP testing method");
String url = "https://someloginpage.com";
try{
Document doc = Jsoup.connect(url).get();
String S = doc.getElementById("username").text();// LINE 1
String S1 = doc.getElementById("password").text();// LINE 2
}catch(Exception e){
e.printStackTrace();
}
}
}
Exception:
java.lang.NullPointerException
JSOUPTester.main(JSOUPTester.java:7)
I have checked in the chrome that these pages contain elements with id "username" and "password".
The lines above are throwing NullPointerException. What I am doing wrong here?
A Number of things can be the cause of this. Without the URL I can't be certain, but here are some clues:
Some pages load their content via AJAX. Jsoup can#t deal with this, since it does not interpret any JavaScript. You can check for this by downloading the page with curl, or in a browser while turnig off JavaScript. To deal with pages that use JavaScript to render themselves, you can use tools like Selenium webdriver or HTMLUnit.
The webserver of the page that you try to load might require a cookie to be present. You need to look at the network traffic that happens surfing loading of that page. In chrome or firefox you can see this in the developer tools in the network tab.
The webserver might respond differently for different clients. That is why you may have to set the UserAgent string to a known Browser in your JSoup http request.
Jsoup.connect("url").userAgent("Mozilla/5.0")
JSoup has a size limitation of 1MB for the downloaded html source. You can turn this off or set it to a larger value if needed.
Jsoup.connect("url").maxBodySize(0)
Jsoup might timeout on the request. To change timeout behavior use
Jsoup.connect("url").timeout(milliseconds)
There might be other reasons I did not think of now.
I have an idea to make something pretty sweet but I'm not sure if it's possible. Here is an example of a very basic ajax function that I might use to establish a connection a server...
function getFakePage(userId)
{
var ajaxObject, path, params;
ajaxObject = getAjaxObject();
params = "?userId=" + userId
path = getInternalPath() + "someServlet" + params;
ajaxObject.open("GET", path, true);
ajaxObject.send();
// On ready state change stuff here
}
So let's say I have a URL like this...
https://localhost:8443/Instride/user/1/admin
And I wanted to use javascript to redirect the user to this this URL. Normally I would just do this...
window.location = "https://localhost:8443/Instride/user/1/admin";
But my idea is to create a javascript (no js frameworks please) function that could combine the ajax code with the window.location code. Basically what I would like to accomplish is to create a connection with the server via ajax, send a servlet on that server the url I would like the user to be redirected to, and then redirect the user to that URL. So that for however long it takes the user to connect to my server from wherever they are in the world they see a loading icon instead of a blank white page.
So to clarify exactly what I am trying to accomplish; I do not want to put window.location within the success of my ajax function (because that would be encompass two round trips), and I do not want to return a huge chunk of HTML for the requested resource and add it to the page. I want to establish a connection to the server with ajax, send a servlet the URL the user wants to go to, and then somehow override the ajax function to redirect that user. Is this possible?
And I know some of you might think this is stupid but it's not when you're talking about overseas users with slow dial up connections staring at white pages. If it's possible, I'd love to hear some insight. Thank you very much!
First, let me say that the best solution is finding what is causing the slowness and fixing it.
Now as to your question, yes you could do it. You could even shoehorn it onto an existing application. But it wouldn't be pretty. And it comes with it's own set of problems. But here are the steps:
Browser calls ajax cache service requesting "somepage.html"
Browser loads loading icon
Server creates somepage.html and caches it in a temporary cache, (ehcache or other library would be good, probably with file backing for the cache depending on size)
Server responds to ajax request with ID for cached page
Browser now redirects to "somepage.html?cacheId={cacheId}" where the id is from the ajax call.
Server uses a filter to see if any cache can be served up for the page instead of the actual page, thus speeding up the request.
Having said that, it would be better to just have the new page load quickly with a loading icon while it did any of the heavy lifting through ajax.
You can't do an AJAX request and a location change in one. If you want to do only one request you have to choose one of those methods. ie. return some data and replace content on your current page, or load a completely new page.
It doesn't make any sense to want to want to do both. What you could want is stateful URLs; where your URL matches the content displayed, even if that content comes from an AJAX request. In that case an easy solution is the use the # part of the URL which you can change freely (window.location.hash). Some modern browsers support changing the whole URL without causing the page to reload. I've used # with great success myself.
I Am serving an authenticated image using django. The image is behind a view which require login, and in the end I have to check more things than just the authentication.
Because of a reason to complicated to explain here, I cannot use the real url to the image, but I Am serving it with a custom url leading to the authenticated view.
From java the image must be reachable, to save or display. For this part I use Apache httpclient.
In Apacahe I tried a lot of things (every example and combination of examples...) but can't seem to get it working.
For other parts of the webapp I use django-rest-framwork, which I succesfully connected to from java (and c and curl).
I use the login_reuired decorator in django, which makes the attempt to get to the url redirect to a login page first.
Trying the link and the login in a webviewer, I see the 200 code (OK) in the server console.
Trying the link with the httpclient, I get a 302 Found in the console.... (looking up 302, it means a redirect..)
this is what I do in django:
in urls.py:
url(r'^photolink/(?P<filename>.*)$', 'myapp.views.photolink',name='photolink'),
in views.py:
import mimetypes
import os
#login_required
def photolink(request, filename):
# from the filename I get the image object, for this question not interesting
# there is a good reason for this complicated way to reach a photo, but not the point here
filename_photo = some_image_object.url
base_filename=os.path.basename(filename_photo)
# than this is the real path and filename to the photo:
path_filename=os.path.join(settings.MEDIA_ROOT,'photos',mac,base_filename)
mime = mimetypes.guess_type(filename_photot)[0]
logger.debug("mimetype response = %s" % mime)
image_data = open(path_filename, 'rb').read()
return HttpResponse(image_data, mimetype=mime)
by the way, if i get this working i need another decorator to pass some other tests....
but i first need to get this thing working....
for now it's not a secured url.... plain http.
in java i tried a lot of things... using apache's httpclient 4.2.1
proxy, cookies, authentication negociation, with follow redirects... and so on...
Am I overlooking some basic thing here?...
it seems the login of the website client is not suitable for automated login...
so the problem can be in my code in django....or in the java code....
In the end the problem was, using HTTP authorization.
Which is not by default used in the login_required decorator.
adding a custom decorator that checks for HTTP authorization did the trick:
see this example: http://djangosnippets.org/snippets/243/
I'm using the jQuery Address library to re-write my URL depending on what the user is doing in the page. The intention is to allow users to bookmark pages and come back later. The page never refreshes as all server interaction is done via ajax.
jQuery Address is writing URLs like this:
http://localhost:9000/#/u/scott_tiger
I need a to set up a route in Play to be able to route that request through to the appropriate controller. So I set this up:
GET /#/u/{username} Controller.showUser
This doesn't work though, the route definition gets ignored. I've tried loads of things such as trying to escape the "#" and replacing it with a variable that I've populated with Character.toString(35). None of this works.
Does anyone know how I can either define the route properly or get jQuery Address not to write the "#".
EDIT: The "#" doesn't get sent to the server does it. Doh! OK, question is revised.
No. The # and the part of the URL after that is not sent to the server. So your play app on the server will never see such URLs.
HTML5 solution
You need to handle these URLs on the client side using JavaScript. In modern browsers with good HTML5 support, you can modify the address without reloading the page. See Manipulating the browser history on how to do it for these browsers. And see When can I use... for browser support.
#-URLs
On Internet Explorer and older versions of other browsers you need to use # URLs and use JavaScript to load the state (e.g. get the user page /u/scott_tiger in your example). See How to run a JavaScript function when the user is visiting an hash link (#something) using JQuery? for how to do this in JavaScript. Also if a user bookmarks a page with a #-URL you need to reload the state.
See also: What's the shebang/hashbang (#!) in Facebook and new Twitter URLs for?
JavaScript libraries
You may use JavaScript libraries to handle this for you history.js is an example. Also bigger frameworks like Backbone.js handle this.
Does anyone know how I can get jQuery Address not to write the "#".
If you don't write the #-part of the URL, the state can not be linked. So you can not get back to e.g. Scott Tigers profile page if you bookmark the page, because the URL is only http://localhost:9000/ and you will arrive on the front page, while the user though he would arrive on the profile page.
Armed with my new understanding of URLs (thanks #Jonas) I realised that I'd missed half of the story.
I'm using JQuery Address to change the URL depending on what you click in the application. This works great and on lots of browsers. What I was missing was using JQuery Address to watch for external address changes (bookmarks, history, back/forward) and respond accordingly. i.e. set the page up correctly by firing the appropriate Ajax calls and rendering that data appropriately.
Changing the address
$.address.title("new title describing application state");
$.address.parameter("q", "val1");
$.address.parameter("g", "val2");
$.address.update();
Restoring the state
$.address.externalChange(function(event) {
var val1 = event.parameters["q"];
var val2 = event.parameters["g"];
// do something with those values
});