We Keep Coding

How to automate Gmail Login to use HttpsURLConnection and Jsoup in Java 8

I am looking into how to automate the login to an https site that has a form that I need to invoke later on. I came across the example below which explains how to login to gmail; However, the issue that I an having is regardless of the credential used, I always get 200Ok. can someone help troubleshooting this issue for me?
public class HttpUrlConnectionExample {
private List<String> cookies;
private HttpsURLConnection conn;
private final String USER_AGENT = "Mozilla/5.0";
public static void main(String[] args) throws Exception {
String url = "https://accounts.google.com/ServiceLoginAuth";
String gmail = "https://mail.google.com/mail/#inbox";
HttpUrlConnectionExample http = new HttpUrlConnectionExample();
// make sure cookies is turn on
CookieHandler.setDefault(new CookieManager());
// 1. Send a "GET" request, so that you can extract the form's data.
String page = http.GetPageContent(url);
String postParams = http.getFormParams(page, "username#gmail.com", "password");
// 2. Construct above post's content and then send a POST request for
// authentication
http.sendPost(url, postParams);
// 2. Construct above post's content and then send a POST request for
// 3. success then go to gmail.
String result = http.GetPageContent(gmail);
System.out.println(result);
}
private void sendPost(String url, String postParams) throws Exception {
URL obj = new URL(url);
conn = (HttpsURLConnection) obj.openConnection();
// Acts like a browser
conn.setUseCaches(false);
conn.setRequestMethod("POST");
conn.setRequestProperty("Host", "accounts.google.com");
conn.setRequestProperty("User-Agent", USER_AGENT);
conn.setRequestProperty("Accept",
"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
for (String cookie : this.cookies) {
conn.addRequestProperty("Cookie", cookie.split(";", 1)[0]);
}
conn.setRequestProperty("Connection", "keep-alive");
conn.setRequestProperty("Referer", "https://accounts.google.com/ServiceLogin");
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
conn.setRequestProperty("Content-Length", Integer.toString(postParams.length()));
conn.setDoOutput(true);
conn.setDoInput(true);
// Send post request
DataOutputStream wr = new DataOutputStream(conn.getOutputStream());
wr.writeBytes(postParams);
wr.flush();
wr.close();
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'POST' request to URL : " + url);
System.out.println("Post parameters : " + postParams);
System.out.println("Response Code : " + responseCode);
BufferedReader in =
new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// System.out.println(response.toString());
}
private String GetPageContent(String url) throws Exception {
URL obj = new URL(url);
conn = (HttpsURLConnection) obj.openConnection();
// default is GET
conn.setRequestMethod("GET");
conn.setUseCaches(false);
// act like a browser
conn.setRequestProperty("User-Agent", USER_AGENT);
conn.setRequestProperty("Accept",
"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
if (cookies != null) {
for (String cookie : this.cookies) {
conn.addRequestProperty("Cookie", cookie.split(";", 1)[0]);
}
}
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader in =
new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// Get the response cookies
setCookies(conn.getHeaderFields().get("Set-Cookie"));
return response.toString();
}
public String getFormParams(String html, String username, String password)
throws UnsupportedEncodingException {
System.out.println("Extracting form's data...");
Document doc = Jsoup.parse(html);
// Google form id
Element loginform = doc.getElementById("gaia_loginform");
Elements inputElements = loginform.getElementsByTag("input");
List<String> paramList = new ArrayList<String>();
for (Element inputElement : inputElements) {
String key = inputElement.attr("name");
String value = inputElement.attr("value");
if (key.equals("Email"))
value = username;
else if (key.equals("Passwd"))
value = password;
paramList.add(key + "=" + URLEncoder.encode(value, "UTF-8"));
System.out.println(paramList.toString());
}
// build parameters list
StringBuilder result = new StringBuilder();
for (String param : paramList) {
if (result.length() == 0) {
result.append(param);
} else {
result.append("&" + param);
}
}
return result.toString();
}
public List<String> getCookies() {
return cookies;
}
public void setCookies(List<String> cookies) {
this.cookies = cookies;
}
}

Without saying it's impossible, you'll have a LOT of hard and tedious work to succeed on this path. The Gmail login is a model of complication. It is not a mere form parameters sending...
Hopefully, there is a really highspeed path: Gmail API.
You'll find the handy Java quickstart in order to automate the Gmail login.
Here is a summary:
Prerequesites
Java 1.7 or greater.
Gradle 2.3 or greater.
Access to the internet and a web browser.
A Google account with Gmail enabled.
Step 1: Turn on the Gmail API
Simple step by step instructions to prepare and setup your Gmail API use.
Step 2: Prepare the project
Prepare the project dependencies with Gradle. You may setup the dependencies either manually or with another builder (Maven etc).
Step 3: Set up the sample
You can copy or download a full working sample code to login into Gmail.
See also:
In this example, we will log into the GitHub website by using the FormElement class.
// # Constants used in this example
final String USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36";
final String LOGIN_FORM_URL = "https://github.com/login";
final String USERNAME = "yourUsername";
final String PASSWORD = "yourPassword";
// # Go to login page
Connection.Response loginFormResponse = Jsoup.connect(LOGIN_FORM_URL)
.method(Connection.Method.GET)
.userAgent(USER_AGENT)
.execute();
// # Fill the login form
// ## Find the form first...
FormElement loginForm = (FormElement)loginFormResponse.parse()
.select("div#login > form").first();
checkElement("Login Form", loginForm);
// ## ... then "type" the username ...
Element loginField = loginForm.select("#login_field").first();
checkElement("Login Field", loginField);
loginField.val(USERNAME);
// ## ... and "type" the password
Element passwordField = loginForm.select("#password").first();
checkElement("Password Field", passwordField);
passwordField.val(PASSWORD);
// # Now send the form for login
Connection.Response loginActionResponse = loginForm.submit()
.cookies(loginFormResponse.cookies())
.userAgent(USER_AGENT)
.execute();
System.out.println(loginActionResponse.parse().html());
public static void checkElement(String name, Element elem) {
if (elem == null) {
throw new RuntimeException("Unable to find " + name);
}
}
All the form data is handled by the FormElement class for us (even the form method detection). A ready made Connection is built when invoking the FormElement#submit method. All we have to do is to complete this connection with addional headers (cookies, user-agent etc) and execute it.

After login using java code to website server sends response code 500

I want to make java program to download a file from website, but i'm stuck in following situation. When I run the following code logging to page gets done successfully, but after it is done, when I send request for the next page, it shows response code 500.
public class Ss2mydb
{
private List<String> cookies;
private HttpURLConnection conn;
private final String USER_AGENT = "Mozilla/5.0";
public static void main(String[] args) throws Exception
{
String indexPage = "http://10.100.100.142/index.asp";
String validatePage = "http://10.100.100.142/validate.asp";
String ccmenuPage = "http://10.100.100.142/callcentre/ccmenu.asp";
String reportPage = "http://10.100.100.142/topmgmt/reports/PROJECTVIJAY/get_download_cafs.asp";
String reportPageDownload = "http://10.100.100.142/topmgmt/reports/PROJECTVIJAY/get_download_cafs.asp?view=N";
Ss2mydb http = new Ss2mydb();
// make sure cookies is turn on
CookieHandler.setDefault(new CookieManager());
// 1. Send a "GET" request, so that you can extract the form's data.
String page = http.GetPageContent(indexPage);
String postParams = http.getFormParams(page, "username", "password");
// 2. Construct above post's content and then send a POST request for
// authentication
http.sendPost(validatePage, postParams);
// System.exit(0);
// 3. success then go to gmail.
http.GetPageContent2(ccmenuPage);
String result = http.GetPageContent2(reportPage);
System.out.println(result);
}
private void sendPost(String url, String postParams) throws Exception {
URL obj = new URL(url);
conn = (HttpURLConnection) obj.openConnection();
// Acts like a browser
conn.setUseCaches(false);
conn.setRequestMethod("POST");
conn.setRequestProperty("Host", "10.100.100.142");
conn.setRequestProperty("User-Agent", USER_AGENT);
conn.setRequestProperty("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
for (String cookie : this.cookies) {
System.out.println(" sendPost : "+cookie.split(";", 1)[0]);
conn.addRequestProperty("Cookie",cookie.split(";", 1)[0]);
}
conn.setRequestProperty("Connection", "keep-alive");
conn.setRequestProperty("Referer", "http://10.100.100.142/index.asp");
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
conn.setRequestProperty("Content-Length", Integer.toString(postParams.length()));
conn.setDoOutput(true);
conn.setDoInput(true);
// Send post request
DataOutputStream wr = new DataOutputStream(conn.getOutputStream());
wr.writeBytes(postParams);
wr.flush();
wr.close();
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'POST' request to URL : " + url);
System.out.println("Post parameters : " + postParams);
System.out.println("Response Code : " + responseCode);
// setCookies(conn.getHeaderFields().get("Set-Cookie"));
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// System.out.println(response.toString());
}
private String GetPageContent2(String url) throws Exception {
URL obj = new URL(url);
conn = (HttpURLConnection) obj.openConnection();
// default is GET
conn.setRequestMethod("GET");
conn.setUseCaches(false);
// act like a browser
conn.setRequestProperty("User-Agent", USER_AGENT);
conn.setRequestProperty("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
System.out.println(" L : "+cookies);
if (cookies != null) {
for (String cookie : this.cookies) {
System.out.println(" GetPageContent : "+cookie.split(";", 1)[0]);
conn.addRequestProperty("Cookie", cookie.split(";", 1)[0]);
}
}
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// Get the response cookies
// System.out.println(" Get : "+conn.getHeaderFields().get("Set-Cookie"));
// setCookies(conn.getHeaderFields().get("Set-Cookie"));
return response.toString();
}
private String GetPageContent(String url) throws Exception {
URL obj = new URL(url);
conn = (HttpURLConnection) obj.openConnection();
// default is GET
conn.setRequestMethod("GET");
conn.setUseCaches(false);
// act like a browser
conn.setRequestProperty("User-Agent", USER_AGENT);
conn.setRequestProperty("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
System.out.println(" L : "+cookies);
if (cookies != null) {
for (String cookie : this.cookies) {
System.out.println(" GetPageContent : "+cookie.split(";", 1)[0]);
conn.addRequestProperty("Cookie", cookie.split(";", 1)[0]);
}
}
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// Get the response cookies
System.out.println(" Get : "+conn.getHeaderFields().get("Set-Cookie"));
setCookies(conn.getHeaderFields().get("Set-Cookie"));
return response.toString();
}
public String getFormParams(String html, String username, String password)
throws UnsupportedEncodingException {
System.out.println("Extracting form's data...");
Document doc = Jsoup.parse(html);
// Google form id
Element loginform = doc.getElementById("right");
Elements inputElements = loginform.getElementsByTag("input");
List<String> paramList = new ArrayList<String>();
for (Element inputElement : inputElements) {
String key = inputElement.attr("name");
String value = inputElement.attr("value");
if (key.equals("USERname"))
value = username;
else if (key.equals("password"))
value = password;
if(!key.equals(""))
paramList.add(key + "=" + URLEncoder.encode(value, "UTF-8"));
}
// build parameters list
StringBuilder result = new StringBuilder();
for (String param : paramList) {
if (result.length() == 0) {
result.append(param);
} else {
result.append("&" + param);
}
}
return result.toString();
}
public List<String> getCookies() {
return cookies;
}
public void setCookies(List<String> cookies) {
this.cookies = cookies;
}
}

500 means "Internal server error" (see HTTP codes and their meaning).
It does mean that the request you sent was understood and was valid HTTP, but either there is an internal, unrelated issue, or the request you send is missing some parameter or is sending some incorrect value, and that omission/mistake causes the server logic to crash (either because it fails a validation of the request content or in a deeper, uncontrolled way).
The only way to get more specific info would be seeing the server logs (and that if the developer was careful checking parameters and logging info).
Your best option is to analize the traffic between the web browser and the server using some tool like wireshark, and then analize the traffic between your client and the server and try to spot the differences.

Send request to Google Elevation API in Java

I'm try to send Post request to Google elevation API and expecting response
private final String ELEVATION_API_URL = "https://maps.googleapis.com/maps/api/elevation/json";
private final String USER_AGENT = "Mozilla/5.0";
String urlParameters = "locations=6.9366681,79.9393521&sensor=true&key=<API KEY>";
URL obj = new URL(ELEVATION_API_URL);
java.net.HttpURLConnection con = (java.net.HttpURLConnection)obj.openConnection();
//add reuqest header
con.setRequestMethod("POST");
con.setRequestProperty("User-Agent", USER_AGENT);
con.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
con.setRequestProperty("Content-Language", "en-US");
String urlParameters = request;
// Send post request
con.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
wr.writeBytes(urlParameters);
wr.flush();
wr.close();
int responseCode = con.getResponseCode();
I'm sending request in this manner but I'm getting response code as 400.This is working when request sent from browser. What is wrong with this code.

To Allow me to get XML back I made the following changes to your project
StringBuilder response = new StringBuilder(); // placed on the top of your Class
**wr.writeBytes(urlParameters.toString());** // as you have it in your code
System.out.println("ResponseMessage : " + connection.getResponseMessage());
System.out.println("RequestMethod : " + connection.getRequestMethod());
in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
wr.flush();
wr.close();
// I changed the URL to :
private final String ELEVATION_API_URL = "https://maps.googleapis.com/maps/api/elevation/xml";
//**I get XML in the response**
return response.toString();

I think there is a problem with the url parameters.
Firstly because sending an empty elevation api request does return a code 400 (Invalid request. Missing the 'path' or 'locations' parameter.).
Secondly because this works (returning 200) :
public void test() throws Exception {
String ELEVATION_API_URL = "https://maps.googleapis.com/maps/api/elevation/json";
String USER_AGENT = "Mozilla/5.0";
String urlParameters = "locations=6.9366681,79.9393521&sensor=true";
URL obj = new URL(ELEVATION_API_URL + "?" + urlParameters);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
//add reuqest header
con.setRequestMethod("POST");
con.setRequestProperty("User-Agent", USER_AGENT);
con.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
con.setRequestProperty("Content-Language", "en-US");
//String urlParameters = request;
// Send post request
con.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
wr.writeBytes(urlParameters);
wr.flush();
wr.close();
int responseCode = con.getResponseCode();
}

Java HttpURLConnection post method Passing Parameters to cgi

I am a newbie in Java web programming. I am using HttpURLConnection to send data to a server via POST. I set the headers then get the output stream and write some bytes then close the output stream. As far as I know it is the correct way but the server send me an unknown exception.
Could you give me some hint why this exception happens
public static void main(String[] args) throws Exception {
String url = "http://www.nlm.nih.gov/cgi/mesh/2014/MB_cgi";
String cookie = retrieveCookies(url);
String urlParameters = "?term=Cancer&exact=Find Exact Term&field=all";
String page = postHttpPage(url , urlParameters, cookie);
System.out.println(page);
System.out.println();
}
public static String postHttpPage(String url, String urlParameters, String cookie) throws Exception {
System.out.println("\nSending 'POST' request to URL : " + url);
URL obj = new URL(url);
HttpURLConnection conn = (HttpURLConnection) obj.openConnection();
return postPage(conn, urlParameters, cookie);
}
private static String postPage(HttpURLConnection conn, String urlParameters, String cookie) throws Exception {
conn.setRequestMethod("POST");
conn.setRequestProperty("User-Agent", "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.57 Safari/537.36");
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
conn.setRequestProperty("Content-Length", String.valueOf(urlParameters.getBytes().length));
conn.setRequestProperty("Content-Language", "en-US");
conn.setRequestProperty("Cookie", cookie);
//conn.setInstanceFollowRedirects(true);
// Send post request
//conn.setDoInput(true);
conn.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(conn.getOutputStream());
wr.writeBytes(urlParameters);
//System.out.println("wr : " + wr.size());
wr.flush();
wr.close();
StringBuilder response = new StringBuilder();
int responseCode = conn.getResponseCode();
System.out.println("Post parameters : " + urlParameters);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
return response.toString();
}
public static String retrieveCookies(String url) throws IOException{
URL obj = new URL(url);
HttpURLConnection conn = (HttpURLConnection) obj.openConnection();
String cookies=conn.getHeaderField("Set-Cookie");
System.out.print("cookies: " + cookies);
conn.disconnect();
return cookies;
}

The problem is about the server. The server did not respond you if you send many requests and also sometimes it does not work might be due to the high amount of requests.

Java HttpURLConnection post method not working

I am using HttpURLConnection to send data to a server via POST. I set the headers then get the output stream and write some bytes then close the output stream.
I am trying to get the next page for schedule details from given url. But some how i am not getting the result. Please help anybody if you know any issue in this code.
I am getting first page with error instead of second page.
"The station combination you have chosen is invalid. Please call the LIRR Travel Information Center at (718) 217-5477 and ask for a representative for more information."
public static void main(String[] args) throws Exception {
String url = "http://lirr42.mta.info";
String cookie = retrieveCookies(url);
String urlParameters = "FromStation=56&ToStation=8&RequestDate=08/24/2013&RequestTime=01:00&RequestAMPM=PM&sortBy=1&schedules=schedules";
String page = postHttpPage(url + "/index.php", urlParameters, cookie);
System.out.println(page);
System.out.println();
}
public static String postHttpPage(String url, String urlParameters, String cookie) throws Exception {
System.out.println("\nSending 'POST' request to URL : " + url);
URL obj = new URL(url);
HttpURLConnection conn = (HttpURLConnection) obj.openConnection();
return postPage(conn, urlParameters, cookie);
}
private static String postPage(HttpURLConnection conn, String urlParameters, String cookie) throws Exception {
conn.setRequestMethod("POST");
conn.setRequestProperty("User-Agent", "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.57 Safari/537.36");
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
conn.setRequestProperty("Content-Length", String.valueOf(urlParameters.getBytes().length));
conn.setRequestProperty("Content-Language", "en-US");
conn.setRequestProperty("Cookie", cookie);
//conn.setInstanceFollowRedirects(true);
// Send post request
//conn.setDoInput(true);
conn.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(conn.getOutputStream());
wr.writeBytes(urlParameters);
//System.out.println("wr : " + wr.size());
wr.flush();
wr.close();
StringBuffer response = new StringBuffer();
int responseCode = conn.getResponseCode();
System.out.println("Post parameters : " + urlParameters);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
return response.toString();
}
public static String retrieveCookies(String url) throws IOException{
URL obj = new URL(url);
HttpURLConnection conn = (HttpURLConnection) obj.openConnection();
String cookies=conn.getHeaderField("Set-Cookie");
System.out.print("cookies: " + cookies);
conn.disconnect();
return cookies;
}

You are missing the "?" Between index.php and the parameters.

You should make
String urlParameters = "FromStation=56&ToStation=8&RequestDate=08/24/2013&RequestTime=01:00&RequestAMPM=PM&sortBy=1&schedules=schedules";
into
String urlParameters = "?FromStation=56&ToStation=8&RequestDate=08/24/2013&RequestTime=01:00&RequestAMPM=PM&sortBy=1&schedules=schedules";

HttpURLConnection.getResponseCode();
this method may help you.
It forces the connection really establish the connection.
may be you want to use Apache HttpClient to make your life easier...