I am working on fetching a JSON on an externally hosted server. I have written 2 separate JUnit tests to test the network request with 1 in the Android environment and one just running the standard PC JUnit (not run on android).
When I use the non-Android based JUnit (the simple java program) test, the URLConnection works fine and I receive a response code of 200 from the urlConnection and the JSON. However, when I run the same static function on the Android Device, I receive a response code of 404 (File or location not found). The url itself is encoded and does not contain any non-ASCII characters.
For the purpose of not spamming the host server, I have replaced the url with http://example.com/JSONLink
Things I have tried:
1. The original implementation:
URL url = new URL("http://example.com/JSONLink");
System.out.println(url);
urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setRequestMethod("GET");
int problem = urlConnection.getResponseCode();
urlConnection.connect();
System.out.println("The error code is " + problem);
InputStream inputStream = urlConnection.getInputStream();
At this point, urlConnection throws a File Not Found Error. I have also tried to switch
urlConnection.getInputStream();
to
urlConnection.getErrorStream();
The Error stream gives me a HTML file which states that the server was unable to locate the file.
2. Ensured that the Android Manifest included the User-permission for Internet
<uses-permission android:name="android.permission.INTERNET" />
Attempted the deprecated apache HTTPDefaultClient Approach as such:
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpGet httpGet = new HttpGet("http://example.com/JSONLink");
HttpResponse httpResponse = httpClient.execute(httpGet);
HttpEntity httpEntity = httpResponse.getEntity();
InputStream inputStream = httpEntity.getContent();
Use OkiHTTP
Each one of these attempts have all resulted in the 404 error. While I believe I understand what the 404 error means, I don't understand why I am getting it when I am giving it a valid url which can be accessed by any browser (including using chrome on the phone/emulator).
If this problem is server side, is there a way to imitate a browser just to fetch the json?
Thanks in advance
The first step that is always recommended in such case is to cross check the server. And this answer's your question
If this problem is server side, is there a way to imitate a browser just to fetch the json?
Yes. You can try any one of these chrome plugins to test your server is giving expected response from your browser.
1) Advanced REST client
https://chrome.google.com/webstore/detail/advanced-rest-client/hgmloofddffdnphfgcellkdfbfbjeloo
2) Postman
https://chrome.google.com/webstore/detail/postman/fhbjgbiflinjbdggehcddcbncdddomop
Related
I am sending commands to a server using http, and I currently need to parse a response that the server sends back (I am sending the command via the command line, and the servers response appears in my browser).
There are a lot of resources such as this: Saving a web page to a file in Java, that clearly illustrate how to scrape a page such as cnn.com. However, since this is a response page that is only generated when the camera receives a specific command, my attempts to use the method described by Mike Deck (in the link above) have met with failure. (Specifically, when my program requests the page again the server returns a 401 error.)
The response from the server opens a new tab in my browser. Essentially, I need to know how to save the current web page using java, since reading in a file is probably the most simple way to approach this. Do any of you know how to do this?
TL;DR How do you save the current webpage to a webpage.html or webpage.txt file using java?
EDIT: I used Base64 from the Apache commons codec, which solved my 401 authentication issue. However, I am still getting a 400 error when I attempt to connect my InputStream (see below). Does this mean a connection isn't being established in the first place?
URL url = new URL ("http://"+ipAddress+"/axis-cgi/record/record.cgi?diskid=SD_DISK");
byte[] encodedBytes = Base64.encodeBase64("root:pass".getBytes());
String encoding = new String (encodedBytes);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("POST");
connection.setDoInput (true);
connection.setRequestProperty ("Authorization", "Basic " + encoding);
connection.connect();
InputStream content = (InputStream)connection.getInputStream();
BufferedReader in = new BufferedReader (new InputStreamReader (content));
String line;
while ((line = in.readLine()) != null) {
System.out.println(line);
}
EDIT 2: Changing the request to a GET resolved the issue.
So while scrutinizing my code above, I decided to change
connection.setRequestMethod("POST");
to
connection.setRequestMethod("GET");
This solved my problem. In hindsight, I think the server was not recognizing the HTTP because it is not set up to handle the various trappings that come along with post.
I have the following code, it works totally fine on my local development server, but when I uploaded to the deployment server, I always hit file not found Exception
String urlStr = "http://" + getContext().getRequest().getServerName() +
getContext().getServletContext().getContextPath() + "test.action";
URL url = new URL(urlStr);
InputStream input = url.openStream(); //Error always occurs here, it gives me the correct URL but it says file not found.
Can anyone help me with this?
Because its a HTTP URL the correct way would be as follows.
String urlStr = "http://" + getContext().getRequest().getServerName() +
getContext().getServletContext().getContextPath() + "test.action";
URL url = new URL(urlStr);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
if (conn.getResponseCode() == HttpURLConnection.HTTP_ACCEPTED) {
InputStream input = conn.getInputStream();
}
I think that #deadlock's comments is probably the key to solving this.
You are getting a FileNotFoundException because the remote server is sending a 404 Not Found response. The most likely explanation is that you are attempting to connect using the wrong URL. Print out the URL string before you try to connect.
All the evidence is pointing to the fact that the server is sending "404 Not Found" responses ... for both versions of the code. This normally means that your URL is wrong. But it is also possible for it to be other things:
You may be using different proxies in the Java and browser cases, resulting in the Java case reaching some server that doesn't understand the URL.
It is conceivable that the server is implementing some anti web scraping mechanism, and sending you 404 responses `cos this thinks (rightly) that your requests aren't coming from a web browser,
I'm trying to connect and post to a simple java webservice, running the post's URL from chrome succeeded, but android code skip the following lines (without throwing errors), but the webservice doesn't accept the post
HttpPost post = new HttpPost(setFacebookEventsAddress+userId+"/"+accesstoken);
post.setHeader("Accept", "application/json");
post.setHeader("Content-type", "application/json");
HttpResponse response = client.execute(post);
the webservice method signature handling the above request:
#GET
#Path("setData/{user_id}/{accessToken}")
#Produces(MediaType.APPLICATION_JSON+ ";charset=utf-8")
public String setData(#PathParam("user_id") String user_id,
#PathParam("accessToken") String accessToken) {
since I manage to post throw my browser, anyone can help with what's wrong with my android code?
URL url = new URL(setFacebookEventsAddress+userId+"/"+accesstoken);
HttpURLConnection con = (HttpURLConnection) url
.openConnection();
ja = readStream(con.getInputStream());
Using HttpURLConnection instead of HttpPost did the trick for me, thanks for all the helpers!
It is not possible to say with any certainty (given the evidence), but my guess would be that the expression
setFacebookEventsAddress + userId + "/" + accesstoken
is evaluating to a different URL to the one you are using from the web browser.
I suggest that you try the following:
Turn on request logging on your server, and compare the URLs in the requests being sent.
Modify your client to print out the response status code and the response body. The latter is likely to be an error page that will give you more clues.
Another possible problem is that your code doesn't appear to be sending any body with the POST request.
On revisiting this, the problem was that you were using / trying to do a POST to a web service that you had configured to support GET only. I expect that if you had looked at the status code you would have found that the response code was "Method not supported".
String url = "http://maps.googleapis.com/maps/api/directions/xml?origin=Chicago,IL&destination=Los+Angeles,CA&waypoints=Joplin,MO|Oklahoma+City,OK&sensor=false";
URL google = new URL(url);
HttpURLConnection con = (HttpURLConnection) google.openConnection();
and I use BufferedReader to print the content I get 403 error
The same URL works fine in the browser. Could any one suggest.
The reason it works in a browser but not in java code is that the browser adds some HTTP headers which you lack in your Java code, and the server requires those headers. I've been in the same situation - and the URL worked both in Chrome and the Chrome plugin "Simple REST Client", yet didn't work in Java. Adding this line before the getInputStream() solved the problem:
connection.addRequestProperty("User-Agent", "Mozilla/4.0");
..even though I have never used Mozilla. Your situation might require a different header. It might be related to cookies ... I was getting text in the error stream advising me to enable cookies.
Note that you might get more information by looking at the error text. Here's my code:
try {
HttpURLConnection connection = ((HttpURLConnection)url.openConnection());
connection.addRequestProperty("User-Agent", "Mozilla/4.0");
InputStream input;
if (connection.getResponseCode() == 200) // this must be called before 'getErrorStream()' works
input = connection.getInputStream();
else input = connection.getErrorStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(input));
String msg;
while ((msg =reader.readLine()) != null)
System.out.println(msg);
} catch (IOException e) {
System.err.println(e);
}
HTTP 403 is a Forbidden status code. You would have to read the HttpURLConnection.getErrorStream() to see the response from the server (which can tell you why you have been given a HTTP 403), if any.
This code should work fine. If you have been making a number of requests, it is possible that Google is just throttling you. I have seen Google do this before. You can try using a proxy to verify.
Most browsers automatically encode URLs when you enter them, but the Java URL function doesn't.
You should Encode the URL with URLEncoder URL Encoder
I know this is a bit late, but the easiest way to get the contents of a URL is to use the Apache HttpComponents HttpClient project: http://hc.apache.org/httpcomponents-client-ga/index.html
you original page (with link) and the targeted linked page are not the same domain.
original-domain and target-domain.
I found the difference is in request header:
with 403 forbidden error,
request header have one line:
Referer: http://original-domain/json2tree/ipfs/ipfsList.html
when I enter url, no 403 forbidden,
the request header does NOT have above line referer: original-domain
I finally figure out how to fix this error!!!
on your original-domain web page, you have to add
<meta name="referrer" content="no-referrer" />
it will remove or prevent sending the Referer in header, works both for links and for Ajax requests made
I'm facing this problem with Java. I want to get some HTML informations from a URL. This code was working for so long, but suddenly, it stopped working.
When I access this URL using the browser, it opens with no problem.
The code:
URL site = new URL(this.url);
java.net.URLConnection yc = site.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream()));
String inputLine;
String objetivo = "<td height=\"28\" colspan=\"2\"";
while ((inputLine = in.readLine()) != null && !inputLine.contains(objetivo)) {
}
inputLine = in.readLine();
The Exception:
java.io.IOException: Server returned HTTP response code: 500 for URL: http://www.myurl.com
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at Sites.websites.Site1.getData(Site1.java:53)
at util.Util.lerArquivo(Util.java:278)
at util.Util.main(Util.java:983)
What's wrong? Did the host block me?
HTTP status code 500 usually means that the webserver code has crashed. You need to determine the status code beforehand using HttpURLConnection#getResponseCode() and in case of errors, read the HttpURLConnection#getErrorStream() instead. It may namely contain information about the problem.
If the host has blocked you, you would rather have gotten a 4nn status code like 401 or 403.
See also:
How to use URLConnection to fire and handle HTTP requests?
This Status Code 500 is an Internal Server Error. This code indicates that a part of the server (for example, a CGI program) has crashed or encountered a configuration error.
i think the problem does'nt lie on your side, but rather on the side of the Http server.
the resources you used to access may have been moved or get corrupted, or its configuration just may have altered or spoiled
I had this problem i.e. works fine when pasted into browser but 505s when done through java. It was simply the spaces that needed to be escaped/encoded.
The problem must be with the parameters you are passing(You must be passing blank parameters). For example : http://www.myurl.com?id=5&name=
Check if you are handling this at the server you are calling.
Change the content-type to "application/x-www-form-urlencoded", i solved the problem.
You may look within the first server response and see if the server sent you a cookie.
To check if the server sent you a cookie, you can use HttpURLConnection#getHeaderFields() and look for headers named "Set-Cookie".
If existing, here's the solution for your problem. 100% Working for this case!
In my case, I had changed the Content-Type to Accept and it resolved the issue.
URL url = new URL(GET_URL);
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestMethod("GET");
con.setRequestProperty("Accept", "application/json; charset=utf-8");