Scrape posts on Imgur using HttpURLConnection - java

I want to scrape posts from Imgur.
Let's say we have this link(permalink): https://imgur.com/gallery/ZXjNfqu/comment/1717354251
I want to load this url via HttpURLConnection and read some data from content(HTML).
I have this code:
try{
//create connection
HttpURLConnection connection = (HttpURLConnection)(new URL(url)).openConnection();
connection.setRequestMethod("GET");
//get response
int responseCode = connection.getResponseCode();
if(responseCode == HttpURLConnection.HTTP_OK){
//read html
StringBuilder body = new StringBuilder();
try(var reader = new BufferedReader(new InputStreamReader(connection.getInputStream()))){
String line;
while((line = reader.readLine()) != null){
body.append(line);
}
}
//print html
System.out.println(body.toString());
}else{
//read error content
StringBuilder body = new StringBuilder();
try(var reader = new BufferedReader(new InputStreamReader(connection.getErrorStream()))){
String line;
while((line = reader.readLine()) != null){
body.append(line);
}
}
String bodyStr = body.toString().trim();
throw new Exception("Bad HTTP Request: " + responseCode +
" - " + connection.getResponseMessage() +
(bodyStr.isEmpty() ? "" : " - " + bodyStr));
}
}catch(Exception e){ e.printStackTrace(); }
Via HttpURLConnection I get only a part of the real page(what I see in browser). From HTML is missing at least image url and comments.
How I can get all content via HttpURLConnection?
P.S.: Is not necessary to use HttpURLConnection
Edit 1: I think it's because of javascript. But I'm not sure.

Related

java how to receive web service response in JSON

i am new to web services i am calling a web service that should returns JSON with the folliwng code - the problem is i am getting the response in xml format
when i am trying the same parameters using google rest api - the response is in jSON
any ideas what i am doing wrong ?
public static String getSFData(String urlSuffix) throws MalformedURLException, ProtocolException , IOException
{
String header = "Basic XXXXX";
URL url = new URL("https://api2.successfactors.eu/odata/v2/"+urlSuffix);
HttpsURLConnection connection = (HttpsURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("authorization",header);
connection.setRequestProperty("Content-Type", "application/json");
BufferedReader bf = new BufferedReader(new InputStreamReader(connection.getInputStream()));
StringBuffer stringBuffer = new StringBuffer();
String line;
while ((line = bf.readLine()) != null )
{
stringBuffer.append(line);
}
String response = stringBuffer.toString();
System.out.println("response"+response);
return response;
}
UPDATE
You could try the API URL like http://api2.successfactors.eu/odata/v2/User?
$format=json to get data in JSON.
Use StringBuilder instead of StringBuffer.
Try the following after set content type.
connection.connect();
int status = connection.getResponseCode();
switch (status) {
case 200:
BufferedReader bf = new BufferedReader(new InputStreamReader(connection.getInputStream()));
StringBuilder stringBuilder = new StringBuilder();
String line;
while ((line = bf.readLine()) != null) {
stringBuilder.append(line);
}
String response = stringBuilder.toString();
System.out.println("response : " + response);
}

Can Facebook return a https status code different than 200 when requesting user info is successful?

I have created a code that checks if user is logged into Facebook. This is how it looks like:
URL url = new URL("https://graph.facebook.com/me?access_token=" + this.fb_token);
System.out.println("Attempting to open connection");
HttpsURLConnection conn = (HttpsURLConnection)url.openConnection();
int responseCode = conn.getResponseCode();
System.out.println(responseCode);
BufferedReader reader;
if(responseCode != 200) {
reader = new BufferedReader(new InputStreamReader(conn.getErrorStream(), "UTF-8"));
}
else{
reader = new BufferedReader(new InputStreamReader(conn.getInputStream(), "UTF-8"));
}
String json = reader.readLine();
reader.close();
conn.disconnect();
return json;
Next, I parse the Json, check if it's en error or data code and log the user.
My question is: Is if(responseCode != 200) check enough? Can Facebook return a different status code in case of successful authentication?

HttpURLConnection update from Http Client

Hello I was wondering if somebody could help me with the following, I have a database that is currently populated. I used to call it using the http client and it worked fine but now I'm trying to update the code since its been deprecated to use the httpurlconnection but i have no success. I ve looked up some tutorials and tried a few thing but it doesn't seem to be working. the database is called through a php file and returns it in a json format.If i were to call the php file from my browser the response is the following: [{"id":"15","logo":"logo url","title":"title"}]
The error that I get on the console is the following:java.lang.NullPointerException: Attempt to invoke virtual method 'void java.io.InputStream.close()' on a null object reference
Which its not making much sense to me since the script pulls information
I have the following code, i left the commented section just in case i need any of it, It also includes the old way i used to call the DB Thank you!:
public void loadNews(){
InputStream is = null;
String result = "";
ArrayList<NameValuePair>();
try {
URL url = new URL("http://databasecall.php");
//HttpClient httpclient = new DefaultHttpClient();
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
//urlConnection.setRequestMethod("GET");
//urlConnection.setRequestProperty("Content-length", "0");
//urlConnection.setUseCaches(false);
//urlConnection.setAllowUserInteraction(false);
//urlConnection.setConnectTimeout(15000);
//urlConnection.setReadTimeout(15000);
//urlConnection.connect();
int responseCode = urlConnection.getResponseCode();
Log.i("Tag:", Integer.toString(responseCode)); //tag 200
//HttpPost httppost = new HttpPost("http://databasecall.php");
//httppost.setEntity(new UrlEncodedFormEntity(nameValuePairs));
//HttpResponse response = httpclient.execute(httppost);
//HttpEntity entity = response.getEntity();
//is = entity.getContent();
/*}catch(Exception e){
Log.e("log_tag", "Error in http connection "+e.toString());
}*/
//convert response to string
//try{
//BufferedReader reader = new BufferedReader(new InputStreamReader(is,"iso-8859-1"),8);
if (responseCode == HttpURLConnection.HTTP_OK) {
InputStream in = new BufferedInputStream(urlConnection.getInputStream());
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
//BufferedReader reader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream(), "iso-8859-1"), 8);
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
is.close();
result = sb.toString();
Log.i("Tag:", result);
}
}catch(Exception e){
Log.e("log_tag", "Error converting result " + e.toString());
}
Updated API
try {
String urlParameters = "name=toni&class=one&param3=ok";
byte[] postData = urlParameters.getBytes(Charset.forName("UTF-8"));
int postDataLength = postData.length;
String request = "http://rocks.php";
URL url = new URL(request);
HttpURLConnection cox = (HttpURLConnection) url.openConnection();
cox.setDoOutput(true);
cox.setDoInput(true);
cox.setInstanceFollowRedirects(false);
cox.setRequestMethod("POST");
cox.setRequestProperty("Content-Type",
"application/x-www-form-urlencoded");
cox.setRequestProperty("charset", "utf-8");
cox.setRequestProperty("Content-Length",
Integer.toString(postDataLength));
cox.setUseCaches(false);
OutputStreamWriter writer = new OutputStreamWriter(
cox.getOutputStream());
writer.write(urlParameters);
writer.flush();
String line;
BufferedReader reader = new BufferedReader(new InputStreamReader(
cox.getInputStream()));
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
writer.close();
reader.close();
} catch (Exception e) {
result = e.toString();
Sucess = false;
e.printStackTrace();
}

Using HTTP Get request to overcome the Geocoder limits in Android

Is it possible to use HTTP API and perform HTTP Get request for Google maps in order to overcome the limits of using Geocoder API when requesting latitude and longitude of places?
Something like-
URL url = new URL("https://www.google.com/maps/place/Paris/");
connection.setDoOutput(true);
connection.setDoInput(true);
connection.setRequestMethod("GET");
System.out.println("Value" + connection.getResponseCode());
System.out.println(connection.getResponseMessage());
System.out.println("content"+connection.getContent());
or
URL url = new URL("https://www.google.com/maps/place/Paris/");
BufferedReader br = new BufferedReader(new InputStreamReader(url.openStream()));
String strTemp = "";
while (null != (strTemp = br.readLine())) {
System.out.println(strTemp);
}
Expecting the response to contain the lat and long of the place as in Google maps site, that way my client appears as a regular web client of google maps.
The Places API request has quota limit too, you can see the detail in this page: https://developers.google.com/places/webservice/usage
Also, you need an API key to do your Places API request, a sample way to do a Places API URL request in Android should be like this:
URL placeUrl = new URL("https://maps.googleapis.com/maps/api/place/textsearch/json?query=restaurants+in+Sydney&key=AddYourOwnKeyHere");
HttpURLConnection connection = (HttpURLConnection)placeUrl.openConnection();
connection.setRequestMethod("GET");
connection.connect();
responseCode = connection.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
BufferedReader reader = null;
InputStream inputStream = connection.getInputStream();
StringBuffer buffer = new StringBuffer();
if (inputStream == null) {
// Nothing to do.
return null;
}
reader = new BufferedReader(new InputStreamReader(inputStream));
String line;
while ((line = reader.readLine()) != null) {
buffer.append(line + "\n");
}
if (buffer.length() == 0) {
return null;
}
Log.d(TAG, buffer.toString());
}
else {
Log.i(TAG, "Unsuccessful HTTP Response Code: " + responseCode);
}
You should do this URL request in background thread, for example, in the doInBackground() method of AsyncTask.
You can also visit this tutorial for more detail about how to use Places API in Android.

How to set parameters in a GET request in Java

So I want to send a GET request with parameters. But it only seems to have conventions for the url you send the request to. Unlike the POST request, I see no way to pass parameters in it.
How I send the GET request now, without parameters (might be wrong):
String url = "http://api.netatmo.net/api/getuser";
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
// optional default is GET
con.setRequestMethod("GET");
//add request header
con.setRequestProperty("User-Agent", USER_AGENT);
int responseCode = con.getResponseCode();
Log.v(TAG, ("\nSending 'GET' request to URL : " + url));
Log.v(TAG, ("Response Code : " + responseCode));
BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
//print result
Log.v(TAG, (response.toString()));
How I send the POST request with parameters:
String url = "https://api.netatmo.net/oauth2/token";
URL obj = new URL(url);
HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();
//add request header
con.setRequestMethod("POST");
con.setRequestProperty("User-Agent", USER_AGENT);
con.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
String urlParameters = "grant_type=password&client_id=myid&client_secret=mysecret&username=myusername&password=mypass";
// Send post request
con.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
wr.writeBytes(urlParameters);
wr.flush();
wr.close();
int responseCode = con.getResponseCode();
Log.v(TAG, "\nSending 'POST' request to URL : " + url);
Log.v(TAG, "Post parameters : " + urlParameters);
Log.v(TAG, "Response Code : " + responseCode);
BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
//print result
Log.v(TAG, response.toString());
access_token = response.substring(17, 74);
refresh_token = response.substring(93,150);
getRequest = "/api/getuser?access_token=" + access_token + " HTTP/1.1";
Log.v(TAG, access_token);
Log.v(TAG, refresh_token);
Log.v(TAG, getRequest);
As per the HTTP specification GET supports only path params or url params and hence you cannot put the params in HTTP request body as you do in POST request.
As Sotirios mentioned in the comments, technically you can still push params in the GET body, but if the APIs are respecting the specs, they will not provide you a way to do it.
Have you tried to add the query params to the request java.net.URL?
String url = "http://api.netatmo.net/api/getuser?access_token=" + access_token;
URL obj = new URL(url);
I was encountering the same problem, trying this:
String bla = "http://api.netatmo.net/api/devicelist?access_token=" + AUTH_TOKEN;
URL url = new URL(bla);
BufferedReader reader = new BufferedReader(new InputStreamReader(url.openStream()));
String line = "";
String message = "";
while ((line = reader.readLine()) != null)
{
message += line;
}
I got an exception that the syntax was not correct. When I changed the syntax (by for example encoding with UTF 8) the API would just return errors (like 404 not found...).
I finally got it working using this:
try
{
System.out.println("Access Token: " + AUTH_TOKEN);
String url = "http://api.netatmo.net/api/devicelist";
String query = "access_token=" + URLEncoder.encode(AUTH_TOKEN, CHARSET);
URLConnection connection = new URL(url + "?" + query).openConnection();
connection.setRequestProperty("Accept-Charset", CHARSET);
InputStream response = connection.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(response));
String line = "";
String message = "";
while ((line = reader.readLine()) != null)
{
message += line;
}
return message;
} catch (MalformedURLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Note: CHARSET = "UTF-8"
Turns out the url the API provided confused me greatly. I fixed the url and it works now.

Categories