We Keep Coding

Scrape posts on Imgur using HttpURLConnection

I want to scrape posts from Imgur.
Let's say we have this link(permalink): https://imgur.com/gallery/ZXjNfqu/comment/1717354251
I want to load this url via HttpURLConnection and read some data from content(HTML).
I have this code:
try{
//create connection
HttpURLConnection connection = (HttpURLConnection)(new URL(url)).openConnection();
connection.setRequestMethod("GET");
//get response
int responseCode = connection.getResponseCode();
if(responseCode == HttpURLConnection.HTTP_OK){
//read html
StringBuilder body = new StringBuilder();
try(var reader = new BufferedReader(new InputStreamReader(connection.getInputStream()))){
String line;
while((line = reader.readLine()) != null){
body.append(line);
}
}
//print html
System.out.println(body.toString());
}else{
//read error content
StringBuilder body = new StringBuilder();
try(var reader = new BufferedReader(new InputStreamReader(connection.getErrorStream()))){
String line;
while((line = reader.readLine()) != null){
body.append(line);
}
}
String bodyStr = body.toString().trim();
throw new Exception("Bad HTTP Request: " + responseCode +
" - " + connection.getResponseMessage() +
(bodyStr.isEmpty() ? "" : " - " + bodyStr));
}
}catch(Exception e){ e.printStackTrace(); }
Via HttpURLConnection I get only a part of the real page(what I see in browser). From HTML is missing at least image url and comments.
How I can get all content via HttpURLConnection?
P.S.: Is not necessary to use HttpURLConnection
Edit 1: I think it's because of javascript. But I'm not sure.

how to detect encoding when i'm using bufferedReader

i know this question was asked many times however i'm stuck with this problem and nothing i've read helped me.
i have this code:
BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line;
while((line = reader.readLine()) != null)content += line+"\r\n";
reader.close();
i'm trying to get content of this webpage http://www.garazh.com.ua/tires/catalog/Marangoni/E-COMM/description/ and all nonlatin symbols have been displayed wrong.
i tried set encoding like:
BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream(), "WINDOWS-1251"));
and at this point everething was well! but i cant change encoding for each website i try to parse and i need some solution.
so guys, i know that there is not that easy to detect encoding as it seems but i'm realy need it. if someone had such problem please explain me how you have solved it!
any help appriciated!
this is entire code of the function i'm using to get content:
protected Map<String, String> getFromUrl(String url){
Map<String, String> mp = new HashMap<String, String>();
String newCookie = "", redirect = null;
try{
String host = this.getHostName(url), content = "", header = "", UA = this.getUA(), cookie = this.getCookie(host, UA), referer = "http://"+host+"/";
URL U = new URL(url);
URLConnection conn = U.openConnection();
conn.setRequestProperty("Host", host);
conn.setRequestProperty("User-Agent", UA);
conn.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "ru-ru,ru;q=0.8,en-us;q=0.5,en;q=0.3");
conn.setRequestProperty("Accept-Encoding", "gzip,deflate");
conn.setRequestProperty("Accept-Charset", "utf-8;q=0.7,*;q=0.7");
conn.setRequestProperty("Keep-Alive", "115");
conn.setRequestProperty("Connection", "keep-alive");
conn.setRequestProperty("Connection", "keep-alive");
if(referer != null)conn.setRequestProperty("Referer", referer);
if(cookie != null && !cookie.contentEquals(""))conn.setRequestProperty("Cookie", cookie);
for(int i=0; ; i++){
String name = conn.getHeaderFieldKey(i);
String value = conn.getHeaderField(i);
if(name == null && value == null)break;
else if(name != null)if(name.contentEquals("Set-Cookie"))newCookie += value + " ";
else if(name.toLowerCase().trim().contentEquals("location"))redirect = value;
header += name + ": " + value + "\r\n";
}
if(!newCookie.contentEquals("") && !newCookie.contentEquals(cookie))this.setCookie(host, UA, newCookie.trim());
try{
BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line;
while((line = reader.readLine()) != null)content += line+"\r\n";
reader.close();
}
catch(Exception e){/*System.out.println(url+"\r\n"+e);*/}
mp.put("url", url);
mp.put("header", header);
mp.put("content", content);
}
catch(Exception e){
mp.put("url", "");
mp.put("header", "");
mp.put("content", "");
}
if(redirect != null && this.redirectCount < 3){
mp = getFromUrl(redirect);
this.redirectCount++;
}
return mp;
}

Use jsoup for example. Detecting character encoding of a random website is complex issue because of lying/non-existent headers and 2 different meta tags. For example, the page you linked doesn't send the charset in Content-Type header.
And you're going to need a HTML parser anyway, you didn't think of going with a regex, did you?
Here's example usage:
Connection connection = Jsoup.connect("http://www.garazh.com.ua/tires/catalog/Marangoni/E-COMM/description/");
connection
.header("Host", host)
.header("User-Agent", UA)
.header("Accept", "text/html,application/xhtml+xml,application/xmlq=0.9,*/*q=0.8")
.header("Accept-Language", "ru-ru,ruq=0.8,en-usq=0.5,enq=0.3")
.header("Accept-Encoding", "gzip,deflate")
.header("Accept-Charset", "utf-8q=0.7,*q=0.7")
.header("Keep-Alive", "115")
.header("Connection", "keep-alive");
connection.followRedirects(true);
Document doc = connection.get();
Map<String, String> cookies = connection.response().cookies();
Elements titles = doc.select(".title");
for( Element title : titles ) {
System.out.println(title.ownText());
}
Output:
Шины Marangoni E-COMM
Описание шины Marangoni E-COMM

You want to look for the 'Content-Type' header:
Content-Type: text/html; charset=utf-8
The "charset" part there is what you're looking for.

how to make http get request in Android

I am new to android.So i can any one sho me how to make a http get request such as
GET /photos?size=original&file=vacation.jpg HTTP/1.1
Host: photos.example.net:80
Authorization: OAuth realm="http://photos.example.net/photos",
oauth_consumer_key="dpf43f3p2l4k3l03",
oauth_token="nnch734d00sl2jdk",
oauth_nonce="kllo9940pd9333jh",
oauth_timestamp="1191242096",
oauth_signature_method="HMAC-SHA1",
oauth_version="1.0",
oauth_signature="tR3%2BTy81lMeYAr%2FFid0kMTYa%2FWM%3D"
in android(java)?

You're gonna want to get familiar with InputStreams and OutputStreams in Android, if you've done this in regular java before then its essentially the same thing. You need to open a connection with the request property as "GET", you then write your parameters to the output stream and read the response through an input stream. You can see this in my code below:
try {
URL url = null;
String response = null;
String parameters = "param1=value1&param2=value2";
url = new URL("http://www.somedomain.com/sendGetData.php");
//create the connection
connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setRequestProperty("Content-Type",
"application/x-www-form-urlencoded");
//set the request method to GET
connection.setRequestMethod("GET");
//get the output stream from the connection you created
request = new OutputStreamWriter(connection.getOutputStream());
//write your data to the ouputstream
request.write(parameters);
request.flush();
request.close();
String line = "";
//create your inputsream
InputStreamReader isr = new InputStreamReader(
connection.getInputStream());
//read in the data from input stream, this can be done a variety of ways
BufferedReader reader = new BufferedReader(isr);
StringBuilder sb = new StringBuilder();
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
//get the string version of the response data
response = sb.toString();
//do what you want with the data now
//always remember to close your input and output streams
isr.close();
reader.close();
} catch (IOException e) {
Log.e("HTTP GET:", e.toString());
}

Foursquare Java API v2 - Checkin Example

I'm trying to create a Foursquare application using its Java API v2 but I couldn't find any sample source code for the checkin process. I don't need to full source code (authentication, venue search, etc), I just need to checkin part.
Can somebody help me?

import java.net.*;
import java.io.*;
class HelloCheckin {
public static void main(String[] args) {
try {
// Construct data
String data = URLEncoder.encode("ll", "UTF-8") + "=" + URLEncoder.encode("53.576317,0.113386", "UTF-8");
data += "&" + URLEncoder.encode("venueId", "UTF-8") + "=" + URLEncoder.encode("4e144a2cc65bedaeefbb824a", "UTF-8");
data += "&" + URLEncoder.encode("oauth_token", "UTF-8") + "=" + URLEncoder.encode("YOUR_OAUTHTOKEN", "UTF-8");
// Send data
URL url = new URL("https://api.foursquare.com/v2/checkins/add");
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(data);
wr.flush();
// Get the response
BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line;
while ((line = rd.readLine()) != null) {
// Process line...
}
wr.close();
rd.close();
} catch (Exception e) {
}
} }
The body of this code came from Simple Java Post

Using Java to send data to a form on a website hosted locally

I have a program in Java where I retrieve contents from a database.
Now I have a form in the program, and what I want to do is, on the press of a button, some string (text) content retrieved from the database, should be sent over to a website that I'm hosting locally. The content so sent, should be displayed on the website when refreshed.
Can someone guide me as to how I can achieve this (the sending of data to be displayed over the website)?
Will appreciate a lot, if you could kindly show some sample snippets or give me a reference to some tutorial that can help.
---- Okay so i found a link to a snippet that's supposed to do this, but im unable to understand at this stage as to how exactly this snippet works...can someone please guide me into knowing this better ?
here's the code
try {
// Construct data
String data = URLEncoder.encode("key1", "UTF-8") + "=" + URLEncoder.encode("value1", "UTF-8");
data += "&" + URLEncoder.encode("key2", "UTF-8") + "=" + URLEncoder.encode("value2", "UTF-8");
// Send data
URL url = new URL("http://hostname:80/cgi");
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(data);
wr.flush();
// Get the response
BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line;
while ((line = rd.readLine()) != null) {
// Process line...
}
wr.close();
rd.close();
} catch (Exception e) {
}

I'm not sure on how you store and manage any of the records but from Java you can send a HTTP Post to the Url (In your case http://localhost/, probably).
Have a look at http://www.exampledepot.com/egs/java.net/post.html for a snippet on how to do this.
Your Website could then store the received information in a database and display it when you refresh.
Update heres the function
Just a side not this is by no means the best way to do this and I have no idea on how this scales but for simple solutions this has worked for me in the past.
/**
* Posts a Set of forms variables to the Remote HTTP Host
* #param url The URL to post to and read
* #param params The Parameters to post to the remote host
* #return The Content of the remote page and return null if no data was returned
*/
public String post(String url, Map<String, String> params) {
//Check if Valid URL
if(!url.toLowerCase().contains("http://")) return null;
StringBuilder bldr = new StringBuilder();
try {
//Build the post data
StringBuilder post_data = new StringBuilder();
//Build the posting variables from the map given
for (Iterator iter = params.entrySet().iterator(); iter.hasNext();) {
Map.Entry entry = (Map.Entry) iter.next();
String key = (String) entry.getKey();
String value = (String)entry.getValue();
if(key.length() > 0 && value.length() > 0) {
if(post_data.length() > 0) post_data.append("&");
post_data.append(URLEncoder.encode(key, "UTF-8"));
post_data.append("=");
post_data.append(URLEncoder.encode(value, "UTF-8"));
}
}
// Send data
URL remote_url = new URL(url);
URLConnection conn = remote_url.openConnection();
conn.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(post_data.toString());
wr.flush();
// Get the response
BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
while ((inputLine = rd.readLine()) != null) {
bldr.append(inputLine);
}
wr.close();
rd.close();
} catch (Exception e) {
//Handle Error
}
return bldr.length() > 0 ? bldr.toString() : null;
}
You would then use the function as follows:
Map<String, String> params = new HashMap<String, String>();
params.put("var_a", "test");
params.put("var_b", "test");
params.put("var_c", "test");
String reponse = post("http://localhost/", params);
if(reponse == null) { /* error */ }
else {
System.out.println(reponse);
}

The big question is how will you authenticate the "update" from your Java program to your website?
You could easily write a handler on your website, say "/update" which saves the POST body (or value of a request parameter) to a file or other persistent store but how will you be sure that only you can set that value, instead of anybody who discovers it?