I'm trying to GET a url using HTTPUrlConnection, however I'm always getting a 500 code, but when I try to access that same url from the browser or using curl, it works fine!
This is the code
try{
URL url = new URL("theurl");
HttpURLConnection httpcon = (HttpURLConnection) url.openConnection();
httpcon.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
httpcon.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:14.0) Gecko/20100101 Firefox/14.0.1");
System.out.println(httpcon.getHeaderFields());
}catch (Exception e) {
System.out.println("exception "+e);
}
When I print the headerfields, it shows the 500 code.. when I change the URL to something else like google.com , it works fine. But I don't understand why it doesn't work here but it works fine on the browser and with curl.
Any help would be highly appreciated..
Thank you,
This is mostly happening because of encoding.
If you are using browser OK, but getting 500 ( internal server error ) in your program,it is because the browsers have a highly sophisticated code regarding charsets and content-types.
Here is my code and it works in the case of ISO8859_1 as charset and english language.
public void sendPost(String Url, String params) throws Exception {
String url=Url;
URL obj = new URL(url);
HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();
con.setRequestProperty("Acceptcharset", "en-us");
con.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
con.setRequestProperty("charset", "EN-US");
con.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
String urlParameters=params;
// Send post request
con.setDoOutput(true);
con.setDoInput(true);
con.connect();
//con.
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
wr.writeBytes(urlParameters);
wr.flush();
wr.close();
int responseCode = con.getResponseCode();
System.out.println("\nSending 'POST' request to URL : " + url);
System.out.println("Post parameters : " + urlParameters);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
//print result
System.out.println(response.toString());
this.response=response.toString();
con.disconnect();
}
and in the main program , call it like this:
myclassname.sendPost("https://change.this2webaddress.desphilboy.com/websitealias/orwebpath/someaction","paramname="+URLEncoder.encode(urlparam,"ISO8859_1"))
The status code 500 suggests that the code at web server have been crashed .Use HttpURLConnection#getErrorStream() to get more idea of the error. Refer Http Status Code 500
I ran into the problem of "URL works in browser, but when I do http-get in java I get a 500 Error".
In my case the problem was that the regular http-get ended up in an infinite redirect loop between /default.aspx and /login.aspx
URL oUrl = new URL(url);
HttpURLConnection con = (HttpURLConnection) oUrl.openConnection();
con.setRequestMethod("GET");
...
int responseCode = con.getResponseCode();
What was happening was: The server serves up a three-part cookie and con.getResponseCode() only used one of the parts. The cookie data in the header looked like this:
header.key = null
value = HTTP/1.1 302 Found
...
header.key = Location
value = /default.aspx
header.key = Set-Cookie
value = WebCom-lbal=qxmgueUmKZvx8zjxPftC/bHT/g/rUrJXyOoX3YKnYJxEHwILnR13ojZmkkocFI7ZzU0aX9pVtJ93yNg=; path=/
value = USE_RESPONSIVE_GUI=1; expires=Wed, 17-Apr-2115 18:22:11 GMT; path=/
value = ASP.NET_SessionId=bf0bxkfawdwfr10ipmvviq3d; path=/; HttpOnly
...
So the server when receiving only a third of the needed data got confused: You're logged in! No wait, you have to login. No, you're logged in, ...
To work around the infinite redirect-loop I had to manually look for re-directs and manually parse through the header for "Set-cookie" entries.
con = (HttpURLConnection) oUrl.openConnection();
con.setRequestMethod("GET");
...
log.debug("Disable auto-redirect. We have to look at each redirect manually");
con.setInstanceFollowRedirects(false);
....
int responseCode = con.getResponseCode();
With this code the parsing of the cookie, if we get a redirect in the responseCode:
private String getNewCookiesIfAny(String origCookies, HttpURLConnection con) {
String result = null;
String key;
Set<Map.Entry<String, List<String>>> allHeaders = con.getHeaderFields().entrySet();
for (Map.Entry<String, List<String>> header : allHeaders) {
key = header.getKey();
if (key != null && key.equalsIgnoreCase(HttpHeaders.SET_COOKIE)) {
// get the cookie if need, for login
List<String> values = header.getValue();
for (String value : values) {
if (result == null || result.isEmpty()) {
result = value;
} else {
result = result + "; " + value;
}
}
}
}
if (result == null) {
log.debug("Reuse the original cookie");
result = origCookies;
}
return result;
}
Make sure that your connection allows following redirects - this is one of the possible reasons for difference in behaviour between your connection and the browser (allows redirect by default).
It should be returning code 3xx, but there maybe something else somewhere that changes it to 500 for your connection.
I faced the same issue, and our issue was there was a special symbol in one of the parameter values. We fixed it by using URLEncoder.encode(String, String)
In my case it turned out that the server always returns HTTP/1.1 500 (in Browser as in Java) for the page I wanted to access, but successfully delivers the webpage content nonetheless.
A human accessing the specific page via Browser just doesn't notice, since he will see the page and no error message, in Java I had to read the error stream instead of the input stream (thanks #Muse).
I have no idea why, though. Might be some obscure way to keep Crawlers out.
This is an old question, but I have had same issue and solved it this way.
This might help other is same situation.
In my case I was developing system on local environment, and every thing worked fine when I checked my Rest Api from browser but I got all the time thrown HTTP error 500 in my Android system.
The problem is when you work on Android, it works on VM (Virtual Machine), that said it means your local computer firewall might preventing your Virtual Machine accessing the local URL (IP) address.
You need just to allow that in your computer firewall. The same thing apply if you trying to access system from out side your network.
Check the parameter
httpURLConnection.setDoOutput(false);
Only for GET Method and set to true on POST, this save me lot of time!!!
Related
I am trying to make an API call to get data from Navitia.
The URL looks like this :
https://{myToken}#api.navitia.io/v1/journeys?from={...}
When I copy paste the complete URL in a private tab browser it works fine, but when I call the API from my code I get a 401 error.
This is the code to call the API :
String sURL = "https://{myToken}#api.navitia.io/v1/journeys?from=" +longDeparture +";" + latDeparture + "&to="+ longArrival+ ";"+latArrival +"&";
URL url = new URL(sURL);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.connect();
int responseCode = conn.getResponseCode();
if (responseCode != 200) {
throw new RuntimeException("HttpResponseCode: " + responseCode);
} else {...}
But it works here when copy pasting :
enter image description here
Finally, I stopped calling the API from the Java code.
What I do know is I create a process to curl the api from the terminal and I just fetch the output. I guess it's not the best solution but it works fine now.
I'm creating an application that needs to get an answer from a webservice that response as json object. I've already test URL on browser and it works very well but using Java(J2SE) it doesn't works and throws an exception 451 from server.
I don't know why it works on browser and doesn't works on j2se and how to fix it.
How could I fix this ?
The URL: https://economia.awesomeapi.com.br/all/USD-BRL
Exception
Server returned HTTP response code: 451 for URL: https://economia.awesomeapi.com.br/all/USD-BRL
Method
public void getJsonFromURL(){
try{
URL u = new URL("https://economia.awesomeapi.com.br/all/USD-BRL");
HttpURLConnection c = (HttpURLConnection) u.openConnection();
c.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
c.setRequestProperty("Accept", "application/json");
c.setRequestProperty("access-control-allow-origin", "*");
c.setRequestProperty("server", "keycdn-engine");
c.setRequestMethod("GET");
c.setDoOutput(true);
c.setDoInput(true);
c.connect();
int status = c.getResponseCode();
System.out.println("Code Response: " + status);
//Exception
Server returned HTTP response code: 451 for URL: https://economia.awesomeapi.com.br/all/USD-BRL
}catch(Exception e){
System.out.println("Erro: " + e.getLocalizedMessage());
}
}
Seems I found it, the server doesn't like if the user-agent header isn't there. The request below gave me a result. Even if the parameter has no value, its accepted.
I couldn't be some certificate/ssl/tls issue since the connection was established and the server actually replied, it just didn't like the request very much. This line did it:
con.setRequestProperty("User-Agent", "");
I want to get the Content Length of this file by java:
https://www.subf2m.co/subtitles/farsi_persian-text/SImp4fRrRnBK6j-u2RiPdXSsHSuGVCDLz4XZQLh05FnYmw92n7DZP6KqbHhwp6gfvrxazMManmskHql6va6XEfasUDxGevFRmkWJLjCzsCK50w1lwNajPoMGPTy9ebCC0&name=Q2FwdGFpbiBNYXJ2ZWwgRmFyc2lQZXJzaWFuIGhlYXJpbmcgaW1wYWlyZWQgc3VidGl0bGUgLSBTdWJmMm0gW3N1YmYybS5jb10uemlw
When I insert this url in Firefox or Google Chrome, it downloads a file. but when i want to see that file's size by Java HttpsURlConnection, server returns me Response Code 403 and Content Length -1. why this happens? Thanks
try {
System.out.println("program started -----------------------------------------");
String str_url = "https://www.subf2m.co/subtitles/farsi_persian-text/SImp4fRrRnBK6j-u2RiPdXSsHSuGVCDLz4XZQLh05FnYmw92n7DZP6KqbHhwp6gfvrxazMManmskHql6va6XEfasUDxGevFRmkWJLjCzsCK50w1lwNajPoMGPTy9ebCC0&name=Q2FwdGFpbiBNYXJ2ZWwgRmFyc2lQZXJzaWFuIGhlYXJpbmcgaW1wYWlyZWQgc3VidGl0bGUgLSBTdWJmMm0gW3N1YmYybS5jb10uemlw";
URL url = new URL(str_url);
HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
con.setConnectTimeout(150000);
con.setReadTimeout(150000);
con.setRequestMethod("HEAD");
con.setInstanceFollowRedirects(false);
con.setRequestProperty("Accept-Encoding", "identity");
con.setRequestProperty("connection", "close");
con.connect();
System.out.println("responseCode: " + con.getResponseCode());
System.out.println("contentLength: " + con.getContentLength());
} catch (IOException e) {
System.out.println("error | " + e.toString());
e.printStackTrace();
}
output:
program started -----------------------------------------
responseCode: 403
contentLength: -1
The default Java user-agent is blocked by some online services (most notably, Cloudflare). You need to set the User-Agent header to something else.
con.setRequestProperty("User-Agent", "My-User-Agent");
In my experience, it doesn't matter what you set it to, as long as it's not the default one:
con.setRequestProperty("User-Agent", "aaa"); // works perfectly fine
EDIT: looks like this site uses Cloudflare with DDoS protection active - your code won't run the JavaScript challenge needed to actually get the content of the file.
I've got a simple Tomcat-based Java app that functions as a sort of firewall - I take requests from the "outside", reroute them to resources on the "inside", and return the result to the "outside."
This works fine for GETs, but I'm trying to add a POST function for a different request and I cannot get it working. The "inside" remote server is password protected and I cannot get the remote server to accept the authentication credentials (they work for the GET so the credentials are fine.) Instead, the Tomcat server calls the Authenticator over and over, and finally fails. Here's the error I'm getting:
java.net.ProtocolException: Server redirected too many times (20)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1848)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
at com.mystuff.house.server.MyServlet.doPost(MyServlet.java:191)
I'm sure I'm doing something stupid, but I can't see where it is. Here's the guts of the servlet doPost() routine:
URL url = new URL("HTTP", "10.10.1.101", -1, "/myresource");
URLConnection con = url.openConnection();
HttpURLConnection http = (HttpURLConnection) con;
http.setRequestMethod("POST");
http.setDoOutput(true);
String encoded = String.valueOf(Base64.getEncoder().encode((a.getUsername().concat(":").concat(a.getPassword())).getBytes()));
http.setRequestProperty("Authorization", "Basic "+encoded);
http.setRequestProperty("Content-Type", "application/x-www-form-urlencoded; charset=UTF-8");
// Read the POST payload from the front end post, write to back end post
InputStream r = request.getInputStream();
OutputStream os = http.getOutputStream();
int j = 0;
while ((j = r.read()) != -1) {
os.write((byte) j);
}
http.connect();
// Try reading the result from the back end, push it back to the front end
try {
InputStream i = http.getInputStream();
OutputStream o = response.getOutputStream();
// read/write bytes until EOF
j = 0;
while ((j = i.read()) != -1) {
o.write((byte) j);
}
} catch (Exception ex) {
System.out.println("AIEEEE! Error receiving page from HTTP call");
ex.printStackTrace();
}
The problem with this, after some investigation, turned out to be that the authentication was not valid for the specific URL that I was trying to hit on the remote server.
I had expected to get a 403, 401 or 407 back from the remote server but that never happened, instead this "redirect" happened. So that's something to be aware of if you are trying to hit password-protected URLs from Java code.
I am facing an Issue while connecting to an HTTP request for a particular url in my java Code. For eg: http://www.linkedin.com . This throws Server Not available Error. (TimeOutException) .
But, I want my connection to redirect http request to Location header Value if responseCode is 301 or 302.
Code Snippet:
HttpURLConnection urlConn =(HttpURLConnection) new URL(url).openConnection(); //For eg : http://www.linkedin.com
urlConn.setConnectTimeout(10*1000);
urlConn.setReadTimeout(10*1000);
boolean redirect = false;
int status = urlConn.getResponseCode(); //No response. Throws exception
if (status == HttpURLConnection.HTTP_MOVED_TEMP || status == HttpURLConnection.HTTP_MOVED_PERM)
{
redirect = true;
}
String contenttype = urlConn.getContentType();
if(redirect)
{
String newUrl = urlConn.getHeaderField("Location");//No I18N
urlConn = (HttpURLConnection) new URL(newUrl).openConnection();
urlConn.connect();
contenttype = urlConn.getContentType();
}
Your code works for me. Obviously, you are facing some network issues.
Is http://www.linkedin.com working in your browser?
If yes , then you can check whether it's using some proxy.