We Keep Coding

Android HttpURLConnection: HTTP response always 403

public static class GetTitleTask extends AsyncTask<String, Void, String> {
protected String doInBackground(String... urls) {
try {
String youtubeUrl = urls[0];
if (youtubeUrl != null) {
URL url = new URL("http://www.youtube.com/oembed?url=" + youtubeUrl + "&format=json");
HttpURLConnection connection = null;
BufferedReader reader = null;
try {
connection = (HttpURLConnection) url.openConnection();
connection.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:221.0) Gecko/20100101 Firefox/31.0");
connection.connect();
if (connection.getResponseCode() != HttpURLConnection.HTTP_OK) {
Log.d(TAG, "HTTP response code is " + connection.getResponseCode());
return null;
}
InputStream stream = connection.getInputStream();
reader = new BufferedReader(new InputStreamReader(stream));
StringBuffer buffer = new StringBuffer();
String line = "";
while ((line = reader.readLine()) != null) {
buffer.append(line+"\n");
Log.d("Response: ", "> " + line);
}
return buffer.toString();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
if (connection != null) {
connection.disconnect();
}
try {
if (reader != null) {
reader.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
return null;
}
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
protected void onPostExecute(String title) {
Log.v(TAG, "onPostExecute: " + title);
super.onPostExecute(title);
// ...
}
}
Input for example: https://www.youtube.com/watch?v=hT_nvWreIhg
This always returns null. The HTTP response code is 403. That is why I tried adding a user agent, but it didn't help.
Is there any way to fix this? I would like to get the video title without using the YouTube API.

I tried fetching it with curl. Your URL uses http instead of https://..., change it to https://www.youtube.com/oembed?url=.
Trying to fetch the URL when it starts with just http resulted in a 403 status for me. But https was successful.
URL url = new URL("http://www.youtube.com/oembed?url=" + youtubeUrl + "&format=json");
Should be changed to:
URL url = new URL("https://www.youtube.com/oembed?url=" + youtubeUrl + "&format=json");

The server does not agree to fulfil the request (error code 403). This is probably due to the protocol. Try using HttpsURLConnection instead of HttpURLConnection.

Android cUrl api

i am new to programming in android, i wanted to get json data from the website https://api.clashofclans.com/v1/clans in my android app.
How can i request the data in json format
curl -X GET --header 'Accept: application/json' --header "authorization: Bearer " 'https://api.clashofclans.com/v1/clans' .
i dont know how to add api key in the combination and open network connection. thanks in advance.
urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setReadTimeout(10000 /* milliseconds */);
urlConnection.setConnectTimeout(15000 /* milliseconds */);
urlConnection.setRequestMethod("GET");
urlConnection.addRequestProperty("Accept","application/json");
urlConnection.addRequestProperty("authorization","my api-key here");
urlConnection.connect();
// If the request was successful (response code 200),
// then read the input stream and parse the response.
if (urlConnection.getResponseCode() == 200) {
inputStream = urlConnection.getInputStream();
jsonResponse = readFromStream(inputStream);
} else {
Log.e(LOG_TAG, "Error response code: " +
urlConnection.getResponseCode());
}
i am getting "Error response code: 403", can u tell me where my code is wrong
the api key is based on ip address.

Im sending it to my method this is where i return it as String:
public void getJson() {
String json = getJSONFromURLConnection("https://api.clashofclans.com/v1/clans?name=illuminati");
if (json != null) {
Log.i("JSON", json.toString());
}
}
Here getting all info:
private String getJSONFromURLConnection(String urlString) {
BufferedReader reader = null;
try {
URL url = new URL(urlString);
urlConnection = (HttpURLConnection) url.openConnection();
String apikey = YOUR_KEY;
urlConnection.setRequestMethod("GET");
urlConnection.addRequestProperty("Authorization", "Bearer " + apikey);
urlConnection.setRequestProperty("Accept", "application/json");
if (urlConnection.getResponseCode() == 200) {
InputStream stream = urlConnection.getInputStream();
reader = new BufferedReader(new InputStreamReader(stream));
StringBuffer buffer = new StringBuffer();
String line;
while ((line = reader.readLine()) != null) {
buffer.append(line);
}
return buffer.toString();
} else {
Log.e("Error", "Error response code: " +
urlConnection.getResponseCode());
}
} catch (IOException e) {
e.printStackTrace();
} finally {
if (urlConnection != null)
urlConnection.disconnect();
}
return null;
}
Created and tested, works for me hope you get the result you need.

Not able to get Jsonstring from this link

String fileURL="https://tools.keycdn.com/geo.json?host=192.168.6.9";
URL url = new URL(fileURL);
HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();
int responseCode = httpConn.getResponseCode();
System.out.println(responseCode);
Full code:
try {
String url = "http://tools.keycdn.com/geo.json?host=192.168.6.9";
URL obj = new URL(url);
HttpURLConnection conn = (HttpURLConnection) obj.openConnection();
conn.setInstanceFollowRedirects(true);
HttpURLConnection.setFollowRedirects(true);
conn.setReadTimeout(5000);
conn.addRequestProperty("Accept-Language", "en-US,en;q=0.8");
conn.addRequestProperty("User-Agent", "Mozilla");
conn.addRequestProperty("Referer", "google.com");
System.out.println("Request URL ... " + url);
boolean redirect = false;
// normally, 3xx is redirect
int status = conn.getResponseCode();
if (status != HttpURLConnection.HTTP_OK) {
if (status == HttpURLConnection.HTTP_MOVED_TEMP
|| status == HttpURLConnection.HTTP_MOVED_PERM
|| status == HttpURLConnection.HTTP_SEE_OTHER)
redirect = true;
}
System.out.println("Response Code ... " + status);
if (redirect) {
// get redirect url from "location" header field
String newUrl =conn.getHeaderField("Location");
newUrl=newUrl.replace("https://", "http://");
// get the cookie if need, for login
String cookies = conn.getHeaderField("Set-Cookie");
// open the new connnection again
conn = (HttpURLConnection) new URL(newUrl).openConnection();
conn.setRequestProperty("Cookie", cookies);
conn.addRequestProperty("Accept-Language", "en-US,en;q=0.8");
conn.addRequestProperty("User-Agent", "Mozilla");
conn.addRequestProperty("Referer", "google.com");
System.out.println("Redirect to URL : " + newUrl);
}
BufferedReader in = new BufferedReader(
new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer html = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
html.append(inputLine);
}
in.close();
System.out.println("URL Content... \n" + html.toString());
System.out.println("Done");
} catch (Exception e) {
e.printStackTrace();
}
}
OUTPUT:
Request URL ... http://tools.keycdn.com/geo.json?host=192.168.6.9
Response Code ... 301
Redirect to URL : http://tools.keycdn.com/geo.json?host=192.168.6.9
URL Content...
301 Moved Permanently301 Moved Permanentlynginx
Done
Iam not able to get json from this link, It gives 301 response code, I tried to get the redirected URL from HTTP header part, eventhough, It returns same URL and same 301 response code, please give java code solution to get JSON string from this URL.

Try this :
String fileURL="https://tools.keycdn.com/geo.json?host=192.168.6.9";
URL url;
try
{
final String USER_AGENT = "Mozilla/5.0";
url = new URL(fileURL);
HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();
// optional default is GET
httpConn.setRequestMethod("GET");
//add request header
httpConn.setRequestProperty("User-Agent", USER_AGENT);
int responseCode = httpConn.getResponseCode();
System.out.println(responseCode);
BufferedReader in = new BufferedReader(
new InputStreamReader(httpConn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
//print result
System.out.println(response.toString());
}
catch (Exception e)
{
e.printStackTrace();
}

HttpUrlConnection BadRequest - Statuscode 400

I have implemented a class using HttpUrlConnection to get some data from the google geocoding api. When I'm using this code on android, it works properly. But as soon as I am using this code in another "normal" java program, I am getting the status-code 400 (BadRequest) sometimes. Here is my code:
HttpURLConnection c = null;
StringBuilder sb = new StringBuilder();
try {
URL u = new URL(url);
c = (HttpURLConnection) u.openConnection();
c.setRequestMethod("GET");
c.setRequestProperty("Content-length", "0");
c.setUseCaches(false);
c.setAllowUserInteraction(false);
c.setConnectTimeout(timeout);
c.setReadTimeout(timeout);
c.connect();
int status = c.getResponseCode();
switch (status) {
case HttpURLConnection.HTTP_OK:
case HttpURLConnection.HTTP_CREATED:
BufferedReader br = new BufferedReader(new InputStreamReader(c.getInputStream()));
String line;
while ((line = br.readLine()) != null) {
sb.append(line + "\n");
}
br.close();
}
} catch (SocketTimeoutException ex){
// Handle ...
} catch (MalformedURLException ex) {
// Handle ...
} catch (IOException ex) {
// Handle ...
} finally {
if (c != null) {
try {
c.disconnect();
} catch (Exception ex) {
}
}
}
I have a reliable internet connection and also the URL I am using to receive the data works, whenever I try it with my web browser.
Thanks in advance!

Bad Request is often caused by inadequat URLs. As you mentioned not every URL gives this error, only a view of them. So it has to be something to do with that. Try the following code to ensure the correct encoding of the URL you are using:
String url = ...; // your url
url = URLEncoder.encode(url,"UTF-8");
// Use 'url' ...

URLConnection is not allowing me to access data on Http errors (404,500,etc)

I am making a crawler, and need to get the data from the stream regardless if it is a 200 or not. CURL is doing it, as well as any standard browser.
The following will not actually get the content of the request, even though there is some, an exception is thrown with the http error status code. I want the output regardless, is there a way? I prefer to use this library as it will actually do persistent connections, which is perfect for the type of crawling I am doing.
package test;
import java.net.*;
import java.io.*;
public class Test {
public static void main(String[] args) {
try {
URL url = new URL("http://github.com/XXXXXXXXXXXXXX");
URLConnection connection = url.openConnection();
DataInputStream inStream = new DataInputStream(connection.getInputStream());
String inputLine;
while ((inputLine = inStream.readLine()) != null) {
System.out.println(inputLine);
}
inStream.close();
} catch (MalformedURLException me) {
System.err.println("MalformedURLException: " + me);
} catch (IOException ioe) {
System.err.println("IOException: " + ioe);
}
}
}
Worked, thanks: Here is what I came up with - just as a rough proof of concept:
import java.net.*;
import java.io.*;
public class Test {
public static void main(String[] args) {
//InputStream error = ((HttpURLConnection) connection).getErrorStream();
URL url = null;
URLConnection connection = null;
String inputLine = "";
try {
url = new URL("http://verelo.com/asdfrwdfgdg");
connection = url.openConnection();
DataInputStream inStream = new DataInputStream(connection.getInputStream());
while ((inputLine = inStream.readLine()) != null) {
System.out.println(inputLine);
}
inStream.close();
} catch (MalformedURLException me) {
System.err.println("MalformedURLException: " + me);
} catch (IOException ioe) {
System.err.println("IOException: " + ioe);
InputStream error = ((HttpURLConnection) connection).getErrorStream();
try {
int data = error.read();
while (data != -1) {
//do something with data...
//System.out.println(data);
inputLine = inputLine + (char)data;
data = error.read();
//inputLine = inputLine + (char)data;
}
error.close();
} catch (Exception ex) {
try {
if (error != null) {
error.close();
}
} catch (Exception e) {
}
}
}
System.out.println(inputLine);
}
}

Simple:
URLConnection connection = url.openConnection();
InputStream is = connection.getInputStream();
if (connection instanceof HttpURLConnection) {
HttpURLConnection httpConn = (HttpURLConnection) connection;
int statusCode = httpConn.getResponseCode();
if (statusCode != 200 /* or statusCode >= 200 && statusCode < 300 */) {
is = httpConn.getErrorStream();
}
}
You can refer to Javadoc for explanation. The best way I would handle this is as follows:
URLConnection connection = url.openConnection();
InputStream is = null;
try {
is = connection.getInputStream();
} catch (IOException ioe) {
if (connection instanceof HttpURLConnection) {
HttpURLConnection httpConn = (HttpURLConnection) connection;
int statusCode = httpConn.getResponseCode();
if (statusCode != 200) {
is = httpConn.getErrorStream();
}
}
}

You need to do the following after calling openConnection.
Cast the URLConnection to HttpURLConnection
Call getResponseCode
If the response is a success, use getInputStream, otherwise use getErrorStream
(The test for success should be 200 <= code < 300 because there are valid HTTP success codes apart from than 200.)
I am making a crawler, and need to get the data from the stream regardless if it is a 200 or not.
Just be aware that it if the code is a 4xx or 5xx, then the "data" is likely to be an error page of some kind.
The final point that should be made is that you should always respect the "robots.txt" file ... and read the Terms of Service before crawling / scraping the content of a site whose owners might care. Simply blatting off GET requests is likely to annoy site owners ... unless you've already come to some sort of "arrangement" with them.