I am a newbie in Java web programming. I am using HttpURLConnection to send data to a server via POST. I set the headers then get the output stream and write some bytes then close the output stream. As far as I know it is the correct way but the server send me an unknown exception.
Could you give me some hint why this exception happens
public static void main(String[] args) throws Exception {
String url = "http://www.nlm.nih.gov/cgi/mesh/2014/MB_cgi";
String cookie = retrieveCookies(url);
String urlParameters = "?term=Cancer&exact=Find Exact Term&field=all";
String page = postHttpPage(url , urlParameters, cookie);
System.out.println(page);
System.out.println();
}
public static String postHttpPage(String url, String urlParameters, String cookie) throws Exception {
System.out.println("\nSending 'POST' request to URL : " + url);
URL obj = new URL(url);
HttpURLConnection conn = (HttpURLConnection) obj.openConnection();
return postPage(conn, urlParameters, cookie);
}
private static String postPage(HttpURLConnection conn, String urlParameters, String cookie) throws Exception {
conn.setRequestMethod("POST");
conn.setRequestProperty("User-Agent", "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.57 Safari/537.36");
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
conn.setRequestProperty("Content-Length", String.valueOf(urlParameters.getBytes().length));
conn.setRequestProperty("Content-Language", "en-US");
conn.setRequestProperty("Cookie", cookie);
//conn.setInstanceFollowRedirects(true);
// Send post request
//conn.setDoInput(true);
conn.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(conn.getOutputStream());
wr.writeBytes(urlParameters);
//System.out.println("wr : " + wr.size());
wr.flush();
wr.close();
StringBuilder response = new StringBuilder();
int responseCode = conn.getResponseCode();
System.out.println("Post parameters : " + urlParameters);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
return response.toString();
}
public static String retrieveCookies(String url) throws IOException{
URL obj = new URL(url);
HttpURLConnection conn = (HttpURLConnection) obj.openConnection();
String cookies=conn.getHeaderField("Set-Cookie");
System.out.print("cookies: " + cookies);
conn.disconnect();
return cookies;
}
The problem is about the server. The server did not respond you if you send many requests and also sometimes it does not work might be due to the high amount of requests.
Related
I try to send a post request using a java program, I tested the post request URL in postman software, its working fine and post-operation are successful. But when I tried to replicate the same using java program with Http Url connection it pops out the 403 status as Forbidden.
Java code
public class Alexacreate {
public static void main(String Arg[]) throws MalformedURLException, IOException, JSONException {
JSONObject productjson = new JSONObject();
productjson.put("InternalID", "P987240");
String input = productjson.toString();
URL urlForUPdate = new URL("https://my348141.sapbydesign.com/sap/byd/odata/cust/v1/alexatest/MaterialCollection");
HttpURLConnection conn = (HttpURLConnection) urlForUPdate.openConnection();
conn.setRequestProperty("Authorization", "Basic RGV2VXNlcjAxOldlbGNvbWUwMQ==");
conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2");
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Type", "application/json");
conn.setRequestProperty("Accept", "application/json");
conn.setRequestProperty("x-csrf-token", "mQG3DNW_MMwaoIyvaqgepg==");
conn.setDoOutput(true);
conn.setDoInput(true);
System.out.println(input);
OutputStream os = conn.getOutputStream();
os.write(input.getBytes());
os.flush();
conn.connect();
System.out.println(conn.getResponseMessage());
if (conn.getResponseCode() == 201) {
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(conn.getInputStream(), "UTF-8"));
String outPut;
while ((outPut = bufferedReader.readLine()) != null) {
}
System.out.println("Created");
} else {
System.out.println("Not created");
}
}
}
I want to make java program to download a file from website, but i'm stuck in following situation. When I run the following code logging to page gets done successfully, but after it is done, when I send request for the next page, it shows response code 500.
public class Ss2mydb
{
private List<String> cookies;
private HttpURLConnection conn;
private final String USER_AGENT = "Mozilla/5.0";
public static void main(String[] args) throws Exception
{
String indexPage = "http://10.100.100.142/index.asp";
String validatePage = "http://10.100.100.142/validate.asp";
String ccmenuPage = "http://10.100.100.142/callcentre/ccmenu.asp";
String reportPage = "http://10.100.100.142/topmgmt/reports/PROJECTVIJAY/get_download_cafs.asp";
String reportPageDownload = "http://10.100.100.142/topmgmt/reports/PROJECTVIJAY/get_download_cafs.asp?view=N";
Ss2mydb http = new Ss2mydb();
// make sure cookies is turn on
CookieHandler.setDefault(new CookieManager());
// 1. Send a "GET" request, so that you can extract the form's data.
String page = http.GetPageContent(indexPage);
String postParams = http.getFormParams(page, "username", "password");
// 2. Construct above post's content and then send a POST request for
// authentication
http.sendPost(validatePage, postParams);
// System.exit(0);
// 3. success then go to gmail.
http.GetPageContent2(ccmenuPage);
String result = http.GetPageContent2(reportPage);
System.out.println(result);
}
private void sendPost(String url, String postParams) throws Exception {
URL obj = new URL(url);
conn = (HttpURLConnection) obj.openConnection();
// Acts like a browser
conn.setUseCaches(false);
conn.setRequestMethod("POST");
conn.setRequestProperty("Host", "10.100.100.142");
conn.setRequestProperty("User-Agent", USER_AGENT);
conn.setRequestProperty("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
for (String cookie : this.cookies) {
System.out.println(" sendPost : "+cookie.split(";", 1)[0]);
conn.addRequestProperty("Cookie",cookie.split(";", 1)[0]);
}
conn.setRequestProperty("Connection", "keep-alive");
conn.setRequestProperty("Referer", "http://10.100.100.142/index.asp");
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
conn.setRequestProperty("Content-Length", Integer.toString(postParams.length()));
conn.setDoOutput(true);
conn.setDoInput(true);
// Send post request
DataOutputStream wr = new DataOutputStream(conn.getOutputStream());
wr.writeBytes(postParams);
wr.flush();
wr.close();
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'POST' request to URL : " + url);
System.out.println("Post parameters : " + postParams);
System.out.println("Response Code : " + responseCode);
// setCookies(conn.getHeaderFields().get("Set-Cookie"));
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// System.out.println(response.toString());
}
private String GetPageContent2(String url) throws Exception {
URL obj = new URL(url);
conn = (HttpURLConnection) obj.openConnection();
// default is GET
conn.setRequestMethod("GET");
conn.setUseCaches(false);
// act like a browser
conn.setRequestProperty("User-Agent", USER_AGENT);
conn.setRequestProperty("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
System.out.println(" L : "+cookies);
if (cookies != null) {
for (String cookie : this.cookies) {
System.out.println(" GetPageContent : "+cookie.split(";", 1)[0]);
conn.addRequestProperty("Cookie", cookie.split(";", 1)[0]);
}
}
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// Get the response cookies
// System.out.println(" Get : "+conn.getHeaderFields().get("Set-Cookie"));
// setCookies(conn.getHeaderFields().get("Set-Cookie"));
return response.toString();
}
private String GetPageContent(String url) throws Exception {
URL obj = new URL(url);
conn = (HttpURLConnection) obj.openConnection();
// default is GET
conn.setRequestMethod("GET");
conn.setUseCaches(false);
// act like a browser
conn.setRequestProperty("User-Agent", USER_AGENT);
conn.setRequestProperty("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
System.out.println(" L : "+cookies);
if (cookies != null) {
for (String cookie : this.cookies) {
System.out.println(" GetPageContent : "+cookie.split(";", 1)[0]);
conn.addRequestProperty("Cookie", cookie.split(";", 1)[0]);
}
}
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// Get the response cookies
System.out.println(" Get : "+conn.getHeaderFields().get("Set-Cookie"));
setCookies(conn.getHeaderFields().get("Set-Cookie"));
return response.toString();
}
public String getFormParams(String html, String username, String password)
throws UnsupportedEncodingException {
System.out.println("Extracting form's data...");
Document doc = Jsoup.parse(html);
// Google form id
Element loginform = doc.getElementById("right");
Elements inputElements = loginform.getElementsByTag("input");
List<String> paramList = new ArrayList<String>();
for (Element inputElement : inputElements) {
String key = inputElement.attr("name");
String value = inputElement.attr("value");
if (key.equals("USERname"))
value = username;
else if (key.equals("password"))
value = password;
if(!key.equals(""))
paramList.add(key + "=" + URLEncoder.encode(value, "UTF-8"));
}
// build parameters list
StringBuilder result = new StringBuilder();
for (String param : paramList) {
if (result.length() == 0) {
result.append(param);
} else {
result.append("&" + param);
}
}
return result.toString();
}
public List<String> getCookies() {
return cookies;
}
public void setCookies(List<String> cookies) {
this.cookies = cookies;
}
}
500 means "Internal server error" (see HTTP codes and their meaning).
It does mean that the request you sent was understood and was valid HTTP, but either there is an internal, unrelated issue, or the request you send is missing some parameter or is sending some incorrect value, and that omission/mistake causes the server logic to crash (either because it fails a validation of the request content or in a deeper, uncontrolled way).
The only way to get more specific info would be seeing the server logs (and that if the developer was careful checking parameters and logging info).
Your best option is to analize the traffic between the web browser and the server using some tool like wireshark, and then analize the traffic between your client and the server and try to spot the differences.
I am trying to login to twitter using this code below:
//private final String USER_AGENT = "Mozilla/5.0";
private final String USER_AGENT = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36";
public static void main(String[] args) throws Exception {
runThisClass("This is a samplest tweet234!");
}
public static void runThisClass(String tweet)throws Exception{
String url = "https://mobile.twitter.com/session/new";
String url2 = "https://mobile.twitter.com/session";
String url3 = "https://mobile.twitter.com";
String gmail = "https://mobile.twitter.com/compose/tweet";
HttpUrlConnectionExample http = new HttpUrlConnectionExample();
// make sure cookies is turn on
CookieHandler.setDefault(new CookieManager());
// 1. Send a "GET" request, so that you can extract the form's data.
String page = http.GetPageContent(url);
String postParams = http.getFormParams(page, "username", "password");
// 2. Construct above post's content and then send a POST request for
// authentication
http.sendPost(url2, postParams);
// 3. success then go to compose tweet page.
String result = http.GetPageContent(gmail);
//4.Construct another HTTP Post to tweet:
postParams=http.getTweetFormParams(result,tweet);
http.sendPostTweet(url3, postParams);
}
private void sendPost(String url, String postParams) throws Exception {
URL obj = new URL(url);
conn = (HttpsURLConnection) obj.openConnection();
// Acts like a browser
conn.setUseCaches(false);
conn.setRequestMethod("POST");
conn.setRequestProperty("Host", "twitter.com");
conn.setRequestProperty("User-Agent", USER_AGENT);
conn.setRequestProperty("Accept",
"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
for (String cookie : this.cookies) {
conn.addRequestProperty("Cookie", cookie.split(";", 1)[0]);
}
conn.setRequestProperty("Connection", "keep-alive");
conn.setRequestProperty("referer", "https://mobile.twitter.com/session");
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
conn.setRequestProperty("Content-Length", Integer.toString(postParams.length()));
conn.setDoOutput(true);
conn.setDoInput(true);
System.out.println("SendPostPost Params: "+postParams);
// Send post request
DataOutputStream wr = new DataOutputStream(conn.getOutputStream());
wr.writeBytes(postParams);
wr.flush();
wr.close();
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'POST' request to URL : " + url);
System.out.println("Post parameters : " + postParams);
System.out.println("Response Code : " + responseCode);
BufferedReader in =
new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
System.out.println(response.toString());
}
Now the problem I face is here(in method sendPost()):
int responseCode = conn.getResponseCode();
On a general PC, this works fine and I get response code as 200. But when run on android, this gives me 400. I wonder whats wrong here. Any help is very much appreciated and required.
I'm try to send Post request to Google elevation API and expecting response
private final String ELEVATION_API_URL = "https://maps.googleapis.com/maps/api/elevation/json";
private final String USER_AGENT = "Mozilla/5.0";
String urlParameters = "locations=6.9366681,79.9393521&sensor=true&key=<API KEY>";
URL obj = new URL(ELEVATION_API_URL);
java.net.HttpURLConnection con = (java.net.HttpURLConnection)obj.openConnection();
//add reuqest header
con.setRequestMethod("POST");
con.setRequestProperty("User-Agent", USER_AGENT);
con.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
con.setRequestProperty("Content-Language", "en-US");
String urlParameters = request;
// Send post request
con.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
wr.writeBytes(urlParameters);
wr.flush();
wr.close();
int responseCode = con.getResponseCode();
I'm sending request in this manner but I'm getting response code as 400.This is working when request sent from browser. What is wrong with this code.
To Allow me to get XML back I made the following changes to your project
StringBuilder response = new StringBuilder(); // placed on the top of your Class
**wr.writeBytes(urlParameters.toString());** // as you have it in your code
System.out.println("ResponseMessage : " + connection.getResponseMessage());
System.out.println("RequestMethod : " + connection.getRequestMethod());
in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
wr.flush();
wr.close();
// I changed the URL to :
private final String ELEVATION_API_URL = "https://maps.googleapis.com/maps/api/elevation/xml";
//**I get XML in the response**
return response.toString();
I think there is a problem with the url parameters.
Firstly because sending an empty elevation api request does return a code 400 (Invalid request. Missing the 'path' or 'locations' parameter.).
Secondly because this works (returning 200) :
public void test() throws Exception {
String ELEVATION_API_URL = "https://maps.googleapis.com/maps/api/elevation/json";
String USER_AGENT = "Mozilla/5.0";
String urlParameters = "locations=6.9366681,79.9393521&sensor=true";
URL obj = new URL(ELEVATION_API_URL + "?" + urlParameters);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
//add reuqest header
con.setRequestMethod("POST");
con.setRequestProperty("User-Agent", USER_AGENT);
con.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
con.setRequestProperty("Content-Language", "en-US");
//String urlParameters = request;
// Send post request
con.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
wr.writeBytes(urlParameters);
wr.flush();
wr.close();
int responseCode = con.getResponseCode();
}
I am using HttpURLConnection to send data to a server via POST. I set the headers then get the output stream and write some bytes then close the output stream.
I am trying to get the next page for schedule details from given url. But some how i am not getting the result. Please help anybody if you know any issue in this code.
I am getting first page with error instead of second page.
"The station combination you have chosen is invalid. Please call the LIRR Travel Information Center at (718) 217-5477 and ask for a representative for more information."
public static void main(String[] args) throws Exception {
String url = "http://lirr42.mta.info";
String cookie = retrieveCookies(url);
String urlParameters = "FromStation=56&ToStation=8&RequestDate=08/24/2013&RequestTime=01:00&RequestAMPM=PM&sortBy=1&schedules=schedules";
String page = postHttpPage(url + "/index.php", urlParameters, cookie);
System.out.println(page);
System.out.println();
}
public static String postHttpPage(String url, String urlParameters, String cookie) throws Exception {
System.out.println("\nSending 'POST' request to URL : " + url);
URL obj = new URL(url);
HttpURLConnection conn = (HttpURLConnection) obj.openConnection();
return postPage(conn, urlParameters, cookie);
}
private static String postPage(HttpURLConnection conn, String urlParameters, String cookie) throws Exception {
conn.setRequestMethod("POST");
conn.setRequestProperty("User-Agent", "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.57 Safari/537.36");
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
conn.setRequestProperty("Content-Length", String.valueOf(urlParameters.getBytes().length));
conn.setRequestProperty("Content-Language", "en-US");
conn.setRequestProperty("Cookie", cookie);
//conn.setInstanceFollowRedirects(true);
// Send post request
//conn.setDoInput(true);
conn.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(conn.getOutputStream());
wr.writeBytes(urlParameters);
//System.out.println("wr : " + wr.size());
wr.flush();
wr.close();
StringBuffer response = new StringBuffer();
int responseCode = conn.getResponseCode();
System.out.println("Post parameters : " + urlParameters);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
return response.toString();
}
public static String retrieveCookies(String url) throws IOException{
URL obj = new URL(url);
HttpURLConnection conn = (HttpURLConnection) obj.openConnection();
String cookies=conn.getHeaderField("Set-Cookie");
System.out.print("cookies: " + cookies);
conn.disconnect();
return cookies;
}
You are missing the "?" Between index.php and the parameters.
You should make
String urlParameters = "FromStation=56&ToStation=8&RequestDate=08/24/2013&RequestTime=01:00&RequestAMPM=PM&sortBy=1&schedules=schedules";
into
String urlParameters = "?FromStation=56&ToStation=8&RequestDate=08/24/2013&RequestTime=01:00&RequestAMPM=PM&sortBy=1&schedules=schedules";
HttpURLConnection.getResponseCode();
this method may help you.
It forces the connection really establish the connection.
may be you want to use Apache HttpClient to make your life easier...