i'm trying to create stand-alone webserver to programmatically search torrent file (ex. from torrentz.eu) and put on download.
I'm totally getting mad by download single torrent file, it seems server response is different using browser or java.
this is the script:
connection = (HttpURLConnection)url.openConnection();
connection.setRequestProperty("Cookie", cookies);
System.setProperty("http.agent", "");
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
connection.setRequestMethod("GET");
connection.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
connection.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
connection.setRequestProperty("Connection", "keep-alive");
connection.setRequestProperty("Content-Language", "en-US");
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
connection.setRequestProperty("Connection", "Keep-Alive");
connection.setConnectTimeout(22000);
connection.setReadTimeout(12000);
connection.setUseCaches(false);
connection.setDoInput(true);
connection.setDoOutput(true);
connection.connect();
respCode = connection.getResponseCode();
if(respCode != 200){
// do something..
return false;
}
ByteArrayOutputStream list = new ByteArrayOutputStream();
stream = connection.getInputStream();
byte[] buffer = new byte[512];
int c;
while ((c = stream.read(buffer)) != -1) {
if(c > 0){
list.write(buffer, 0, c);
}
}
list.flush();
stream.close();
this code is good for html, image file, ecc.. but it's impossible to get .torrent files, they are corrupted:
example: UBUNTU torrent,
https://torcache.net/torrent/B415C913643E5FF49FE37D304BBB5E6E11AD5101/[katproxy.com]ubuntu.14.10.desktop.64bit.iso.torrent
size of .torrent file downloaded by browser: 44920 byte
size of .torrent file downloaded by java: 44795 byte
135 byte are missing! just why??
Found the problem!
The file is GZIP compressed!!! probably, browser automatically unzip it by default... thank u very much!
Related
I look network on developer tab(google chrome) and here is the ss
I want to get data on java app.
My code is here:
String urlParameters = "tabloNo=4&yil=2017&ay=2¶Birimi=TL&taraf=10001";
byte[] postData = urlParameters.getBytes( StandardCharsets.UTF_8 );
int postDataLength = postData.length;
String request = "http://172.168.12*.15/getreport";
URL url = new URL( request );
HttpURLConnection conn= (HttpURLConnection) url.openConnection();
conn.setDoOutput(true);
conn.setInstanceFollowRedirects(false);
conn.setConnectTimeout(10000);
conn.setRequestMethod("POST");
conn.setRequestProperty("Accept", "pplication/json, text/javascript, */*; q=0.01");
// conn.setRequestProperty("Accept-Encoding:", "gzip, deflate" );
conn.setRequestProperty("Accept-Language", "tr,en;q=0.8" );
conn.setRequestProperty("Content-Length", Integer.toString(postDataLength ));
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded; charset=UTF-8");
conn.setRequestProperty("Host", "ebulten.bddk.org.tr" );
conn.setRequestProperty("Origin","http://172.168.12*.15/");
conn.setRequestProperty("Proxy-Connection", "keep-alive" );
conn.setRequestProperty("Referer", "http://172.168.12*.15/" );
conn.setRequestProperty("User-Agent","Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36" );
conn.setRequestProperty("X-Requested-With","XMLHttpRequest" );
conn.setUseCaches(true);
try(DataOutputStream wr = new DataOutputStream(conn.getOutputStream())) {
wr.write( postData );
}
it gives Connection timed out: connect
note:
it is public website and everyone read data. I want to read it with programmatically.
thanks in advance
I use HttpURLConnection to crawler https://translate.google.com/.
InetSocketAddress addr = new InetSocketAddress("127.0.0.1", 1082);
Proxy proxy = new Proxy(Proxy.Type.HTTP, addr);
url = new URL("https://translate.google.com/");
HttpURLConnection conn = (HttpURLConnection) url.openConnection(proxy);
conn.setRequestProperty("Accept-Encoding", "gzip, deflate, sdch");
conn.setRequestProperty("Connection", "keep-alive");
conn.setRequestProperty("User-Agent",
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36");
conn.setRequestProperty("Accept", "*/*");
Map<String, List<String>> reqHeaders = conn.getHeaderFields();
List<String> reqTypes = reqHeaders.get("Content-Type");
for (String ss : reqTypes) {
System.out.println(ss);
}
InputStream in = conn.getInputStream();
String s = IOUtils.toString(in, "UTF-8");
System.out.println(s.substring(0, 100));
Map<String, List<String>> resHeader = conn.getHeaderFields();
List<String> resTypes = resHeader.get("Content-Type");
for (String ss : resTypes) {
System.out.println(ss);
}
Console is
But When I change url to http://translate.google.com/.
It works well.
I know actually HttpURLConnection is HttpsURLConnection when i crawler https://translate.google.com/.
I try to use HttpsURLConnection and it still garbled.
Any suggestions?
conn.setRequestProperty("Accept-Encoding", "gzip, deflate, sdch");
The response is compressed, because the above line tells the server that the client is able to understand encodings specified in Accept-Encoding.
Try to comment this line or handle this situation.
There's a more specific implementation for HTTPS i.e. HttpsURLConnection, in case you're interested in https-specific features, e.g.:
import javax.net.ssl.HttpsURLConnection;
....
URL url = new URL("https://www.google.com/");
HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
I accept Jerry Chin's answer.Solves my problem.
My answer just recording how i resolve this problem.
If this approach is unreasonable.Let me know, I'll remove this answer.
conn.setRequestProperty("Accept-Encoding", "gzip, deflate, sdch");
And then I check response Content-Encoding.It's gzip.
So i use GZIPInputStream to receive.
InputStream in = conn.getInputStream();
GZIPInputStream gzis=new GZIPInputStream(in);
InputStreamReader reader = new InputStreamReader(gzis);
BufferedReader br = new BufferedReader(reader);
The InputStream is normal.
BTW,If you don't need Accept-Encoding,you can remove it.
And do not forget check user-agent. It's very important and different operating systems corresponding to different user-agent.
My code makes POST request to a website, but the response looks like encrypted, with weird characters. When i configure my app to use fiddler proxy, the response is valid. How can i make my app decrypt this response?
public class Login {
public Login() throws Exception {
URL url = new URL("https://account.leagueoflegends.com/auth");
HttpsURLConnection conn = (HttpsURLConnection)url.openConnection();
conn.setDoInput(true);
conn.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36");
conn.addRequestProperty("Cookie", "xxx");
conn.addRequestProperty("Content-Length", "406");
conn.addRequestProperty("Content-Type", "application/x-www-form-urlencoded; charset=UTF-8");
conn.addRequestProperty("Accept-Encoding", "gzip,deflate");
conn.addRequestProperty("Connection", "keep-alive");
conn.addRequestProperty("Referer", "xxx");
conn.setDoOutput(true);
OutputStreamWriter writer = new OutputStreamWriter(conn.getOutputStream());
writer.write("xxx");
writer.flush();
String line;
BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream()));
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
writer.close();
reader.close();
}
You are explicitly requesting a zipped stream:
conn.addRequestProperty("Accept-Encoding", "gzip,deflate");
Remove that line to remove the compression. Compression is different from encryption fortunately, it doesn't require you to supply a key.
Im trying to perform the folowing request in my android app.
The Request i send was on a Chrome Browser.
In the Request i shortend the Form Data so there is not so much code.
Request URL:http://***/
Request Method:POST
Status Code:200 OK
Request Headers
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding:gzip,deflate,sdch
Accept-Language:de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4
Cache-Control:max-age=0
Connection:keep-alive
Content-Length:424
Content-Type:application/x-www-form-urlencoded
Cookie:__utma=188893489.1646114392.1358703936.1367178892.1368783485.29; __utmz=188893489.1365594840.21.3.utmcsr=***|utmccn=(referral)|utmcmd=referral|utmcct=/
Host:***
Origin:***
Referer:**
User-Agent:Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36
Form Data
date=1375480155&mail=&dfBoot=Test
Response Headers
Connection:Keep-Alive
Content-Encoding:gzip
Content-Length:3614
Content-Type:text/html
Date:Fri, 02 Aug 2013 21:49:59 GMT
Keep-Alive:timeout=15, max=100
Server:Apache/2.2.14 (Ubuntu)
Vary:Accept-Encoding
X-Powered-By:PHP/5.3.2-1ubuntu4.20
with this code:
StrictMode.ThreadPolicy policy = new StrictMode.ThreadPolicy.Builder()
.permitAll().build();
StrictMode.setThreadPolicy(policy);
URL url = new URL("http://***/index.php");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
try {
connection.setRequestMethod("POST");
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setUseCaches(false);
connection.setRequestProperty("Content-Type","application/x-www-form-urlencoded");
connection.setRequestProperty("Content-Length",String.valueOf(post.length()));
//connection.setChunkedStreamingMode(0);//results in frezing
OutputStreamWriter writer = new OutputStreamWriter(
connection.getOutputStream());
writer.write(post);
writer.flush();
InputStream in = new BufferedInputStream(
connection.getInputStream());
while (in.available() != 0) {
in.read();
}
writer.close();
in.close();
} finally {
connection.disconnect();
}
The code is very messed up because i tryed manytime to fix my connection problem.
Please help with a better Solution.
Hm, I will show my working code with JSON, but maybe you can modify it.
URL url = new URL("http://xcxcxcxcxcx");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestProperty("Content-Type", "application/json");
connection.setRequestProperty("charset", "utf-8");
connection.setRequestProperty("Content-Length", "" + Integer.toString(jsonObject.toString().getBytes().length));
connection.setRequestMethod("POST");
connection.setDoOutput(true);
connection.connect();
DataOutputStream out = new DataOutputStream(connection.getOutputStream());
out.writeBytes(jsonObject.toString());
out.flush();
out.close();
InputStream is = connection.getInputStream();
InputStreamReader reader = new InputStreamReader(is);
BufferedReader r = new BufferedReader(reader);
StringBuilder total = new StringBuilder();
String line;
while ((line = r.readLine()) != null) {
total.append(line);
}
JSONObject jsonObjectResult = null;
if(typeOfMethod == 0){
JSONTokener tokenizer = new JSONTokener(total.toString());
jsonObjectResult = new JSONObject(tokenizer);
}
connection.disconnect();
return jsonObjectResult;
There is some differences between getting result from InputStream.
I'm having trouble reading what cookies are being sent when making a POST in java.
Here's my code:
public static void main(String[] args) throws MalformedURLException, IOException {
String urlParameters = "votebut=";
String request = "http://www.runelocus.com/toplist/vote-17648.html";
URL url = new URL(request);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setDoInput(true);
connection.setInstanceFollowRedirects(false);
connection.setRequestMethod("POST");
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0");
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
connection.setRequestProperty("charset", "utf-8");
connection.setRequestProperty("Content-Length", "" + Integer.toString(urlParameters.getBytes().length));
connection.setUseCaches(false);
DataOutputStream wr = new DataOutputStream(connection.getOutputStream());
wr.writeBytes(urlParameters);
wr.flush();
wr.close();
List<String> cookies = connection.getHeaderFields().get("Set-Cookie");
for (String cookie : cookies) {
System.out.println("Cookies: " + cookie);
}
connection.disconnect();
}
This is what it prints:
Cookies: PHPSESSID=48863f8c3adcbddf0e77e7f1b450fc0e; path=/
This is what I want it to print:
ki_u=68debd85-c1af-f1ff-2e6c-4146755c6e26; ki_t=1354418220596%3B1354418220596%3B1354422379616%3B1%3B36;
Any help please?
Thanks
It is not possible to read that cookies in that way.
In response to http://www.rune...17648.html you only get PHPSESSID cookie in response headers.
That cookies which you are looking for (ki_u and ki_t) are set by JavaScript code in this file:
http://s3.amazonaws.com/ki.js/45645/919.js
So to actually get that cookies values you need to replicate browser behavior or actually use a browser (to request the html page, parse it, download referred resources (particularly 919.js) and execute JavaScript code).