Using java, how do I get the content from a webpage where cookies and javascript are required to load the page?
Without code or more information it's hard to help.
You'll want a URLConnection with a CookieHandler, you can find some example code here:
https://blogs.oracle.com/CoreJavaTechTips/entry/cookie_handling_in_java_se
You should try learning about HttpURLConnection and how to send information like cookies to websites via Http headers
Related
I am designing a third party application that requires a POST request to be sent to a php file on a website and hopefully I should get a response. The site requires me to be logged in in order to make this request normally through the site by pressing a button on it. If I do
Url obj = new URL("http://www.dota2lounge.com/ajax/bumpTrade.php";
HttpUrlConnection con = (HttpUrlConnection) obj.openConnection();
con.setRequestProperty("User-Agent", "Chrome/36.0.1916.144");
And then continue to carry out the POST request, will the site recognize that I am sending this from my Chrome browser in which I am already logged in? Thanks
will the site recognize that I am sending this from my Chrome browser in which I am already logged in?
No, it will not. Imagine how easy it would be to spoof the authentication system of a web application if it worked that way.
Logins typically work by sending Cookies or other headers. You need to send those to authenticate your request. For this to work as if you were logged in with your Chrome application, you'll need to find the corresponding cookies that Chrome stored and send those.
You can find from the link i shared how you can make the authentication.
https://stackoverflow.com/a/3283496/1257445
After you have made an authentication you can make a post request using the session
So I've figured out how to access a non-secure webpage in java. However, I do not know how to access a webpage that fist requires you log in on the login page to view. Could anyone help me with this?
Your question seems similar to this one which points out that not only will you need to send along credentials in your HTTP request (as post or get parameters) but also you may need to consider handling cookies.
How do I login and download a file from a https web page from Java?
I need to validate the PDF report. I need to get the report embeded in HTML. If I read that URL using:
File file = new File("url");
or
HttpWebConnection.getResponse();
it requests the URL in separate session, hence it cannot get the file.
Does ieDriver have something like HtmlUnit?
HttpWebConnection.getResponse()
or somebody can suggest alternative.
Unfortunately it does not.
If you want to get the response code you will need a proxy. If you are using Java then Browser Mob is what you need. You may also try making XmlHttpRequest from javascript and get the status code in that way.
You could also stick to the method you are using right now (separate request from Java) but pass the cookie with session (you can obtain the cookie from WebDriver)
I currently have to post HTML form to a java server like ( www.example.com/file.do ).Java server accept only HTML request. So whenever i post request with html file its work fine. But whenever i post it from php its not working ( same html file just change the extension to php). I have also try with mod-rewrite by changing php to html with .htaccess but nothing happen still not works. So i am thinking is there any difference in html form post and php form post? Or is there any other solution so i can post html form through php to java?
Thanks to all.
I think you are posting from localhost. Try to post from a live server. Hopefully this will solve your problem.
technically there is no difference.
Check whether there are some special http headers set, which are not accepted by your server, i.e.
X-Powered-By etc.
To create a POST request from a PHP script, you need to use curl; but to submit a form from the browser, it doesn't matter what the extension of the file that is rendered is.
I am working on small Java project to programmatically connect to a website with username/password, after login, browse to different links on the site to download some data. First, I need to connect to the website with username/password,
second, while I keep the session open, go to other links to download data.
How do I do this in Java?
Any help will be highly appreciated!
Check out the Apache HTTPClient, it can do all this for you.
Edit: Apache HTTPClient has authentication and cookie handling features included, which will save you a lot of work doing this yourself.
If you want to extract some data HtmlUnit can help you a lot it can manage the authentication and also help you with data extraction.
Investigate with your browser how the web page submits the username/pass data? HTTP Form POST, Ajax, etc..? Use a plugin like Firebug to see network traffic.
You can use URLConnection to create HTTP requests. You will neet to simulate a username/pass login and remember the cookie for use in consequent HTTP requests to simulate a session. Here are some examples: send HTTP POST request, get a cookie, send a cookie.