I look network on developer tab(google chrome) and here is the ss
I want to get data on java app.
My code is here:
String urlParameters = "tabloNo=4&yil=2017&ay=2¶Birimi=TL&taraf=10001";
byte[] postData = urlParameters.getBytes( StandardCharsets.UTF_8 );
int postDataLength = postData.length;
String request = "http://172.168.12*.15/getreport";
URL url = new URL( request );
HttpURLConnection conn= (HttpURLConnection) url.openConnection();
conn.setDoOutput(true);
conn.setInstanceFollowRedirects(false);
conn.setConnectTimeout(10000);
conn.setRequestMethod("POST");
conn.setRequestProperty("Accept", "pplication/json, text/javascript, */*; q=0.01");
// conn.setRequestProperty("Accept-Encoding:", "gzip, deflate" );
conn.setRequestProperty("Accept-Language", "tr,en;q=0.8" );
conn.setRequestProperty("Content-Length", Integer.toString(postDataLength ));
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded; charset=UTF-8");
conn.setRequestProperty("Host", "ebulten.bddk.org.tr" );
conn.setRequestProperty("Origin","http://172.168.12*.15/");
conn.setRequestProperty("Proxy-Connection", "keep-alive" );
conn.setRequestProperty("Referer", "http://172.168.12*.15/" );
conn.setRequestProperty("User-Agent","Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36" );
conn.setRequestProperty("X-Requested-With","XMLHttpRequest" );
conn.setUseCaches(true);
try(DataOutputStream wr = new DataOutputStream(conn.getOutputStream())) {
wr.write( postData );
}
it gives Connection timed out: connect
note:
it is public website and everyone read data. I want to read it with programmatically.
thanks in advance
Related
I use HttpURLConnection to crawler https://translate.google.com/.
InetSocketAddress addr = new InetSocketAddress("127.0.0.1", 1082);
Proxy proxy = new Proxy(Proxy.Type.HTTP, addr);
url = new URL("https://translate.google.com/");
HttpURLConnection conn = (HttpURLConnection) url.openConnection(proxy);
conn.setRequestProperty("Accept-Encoding", "gzip, deflate, sdch");
conn.setRequestProperty("Connection", "keep-alive");
conn.setRequestProperty("User-Agent",
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36");
conn.setRequestProperty("Accept", "*/*");
Map<String, List<String>> reqHeaders = conn.getHeaderFields();
List<String> reqTypes = reqHeaders.get("Content-Type");
for (String ss : reqTypes) {
System.out.println(ss);
}
InputStream in = conn.getInputStream();
String s = IOUtils.toString(in, "UTF-8");
System.out.println(s.substring(0, 100));
Map<String, List<String>> resHeader = conn.getHeaderFields();
List<String> resTypes = resHeader.get("Content-Type");
for (String ss : resTypes) {
System.out.println(ss);
}
Console is
But When I change url to http://translate.google.com/.
It works well.
I know actually HttpURLConnection is HttpsURLConnection when i crawler https://translate.google.com/.
I try to use HttpsURLConnection and it still garbled.
Any suggestions?
conn.setRequestProperty("Accept-Encoding", "gzip, deflate, sdch");
The response is compressed, because the above line tells the server that the client is able to understand encodings specified in Accept-Encoding.
Try to comment this line or handle this situation.
There's a more specific implementation for HTTPS i.e. HttpsURLConnection, in case you're interested in https-specific features, e.g.:
import javax.net.ssl.HttpsURLConnection;
....
URL url = new URL("https://www.google.com/");
HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
I accept Jerry Chin's answer.Solves my problem.
My answer just recording how i resolve this problem.
If this approach is unreasonable.Let me know, I'll remove this answer.
conn.setRequestProperty("Accept-Encoding", "gzip, deflate, sdch");
And then I check response Content-Encoding.It's gzip.
So i use GZIPInputStream to receive.
InputStream in = conn.getInputStream();
GZIPInputStream gzis=new GZIPInputStream(in);
InputStreamReader reader = new InputStreamReader(gzis);
BufferedReader br = new BufferedReader(reader);
The InputStream is normal.
BTW,If you don't need Accept-Encoding,you can remove it.
And do not forget check user-agent. It's very important and different operating systems corresponding to different user-agent.
I'm trying to send get request in java, when it's without headers as below, I get redirected to login page.
URL myURL1 = new URL("http://www.eventlister.com");
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("GET");
But when request is sent with all headers as it is sent from browser when reading response I get it encoded like this
???}isI??g)b?CZ?n?7BH?:m?u??dkzz; $P????C2?????YT??;??B3m????7/?.o~?p,?????..., even if encoding is set manually in reader.
URL myURL1 = new URL("http://www.eventlister.com");
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("GET");
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36");
con.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
con.setRequestProperty("Accept-Encoding", "gzip, deflate, sdch");
con.setRequestProperty("Accept-Language", "en-US,en;q=0.8,zh-CN;q=0.6,zh;q=0.4,ru;q=0.2,uk;q=0.2,ja;q=0.2,fr-FR;q=0.2,fr;q=0.2");
con.setRequestProperty("Connection", "keep-alive");
con.setRequestProperty("Upgrade-Insecure-Requests", "1");
con.setRequestProperty("Host", "www.eventlister.com");
BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream(), StandardCharsets.UTF_8));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
Thank you in advance.
It looks like it's gzip'ed data. Try it without "Accept-Encoding: gzip", so that server returns plain data.
I know this is a comment, not an answer. I don't have comment privilege yet :-(
i'm trying to create stand-alone webserver to programmatically search torrent file (ex. from torrentz.eu) and put on download.
I'm totally getting mad by download single torrent file, it seems server response is different using browser or java.
this is the script:
connection = (HttpURLConnection)url.openConnection();
connection.setRequestProperty("Cookie", cookies);
System.setProperty("http.agent", "");
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
connection.setRequestMethod("GET");
connection.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
connection.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
connection.setRequestProperty("Connection", "keep-alive");
connection.setRequestProperty("Content-Language", "en-US");
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
connection.setRequestProperty("Connection", "Keep-Alive");
connection.setConnectTimeout(22000);
connection.setReadTimeout(12000);
connection.setUseCaches(false);
connection.setDoInput(true);
connection.setDoOutput(true);
connection.connect();
respCode = connection.getResponseCode();
if(respCode != 200){
// do something..
return false;
}
ByteArrayOutputStream list = new ByteArrayOutputStream();
stream = connection.getInputStream();
byte[] buffer = new byte[512];
int c;
while ((c = stream.read(buffer)) != -1) {
if(c > 0){
list.write(buffer, 0, c);
}
}
list.flush();
stream.close();
this code is good for html, image file, ecc.. but it's impossible to get .torrent files, they are corrupted:
example: UBUNTU torrent,
https://torcache.net/torrent/B415C913643E5FF49FE37D304BBB5E6E11AD5101/[katproxy.com]ubuntu.14.10.desktop.64bit.iso.torrent
size of .torrent file downloaded by browser: 44920 byte
size of .torrent file downloaded by java: 44795 byte
135 byte are missing! just why??
Found the problem!
The file is GZIP compressed!!! probably, browser automatically unzip it by default... thank u very much!
My code makes POST request to a website, but the response looks like encrypted, with weird characters. When i configure my app to use fiddler proxy, the response is valid. How can i make my app decrypt this response?
public class Login {
public Login() throws Exception {
URL url = new URL("https://account.leagueoflegends.com/auth");
HttpsURLConnection conn = (HttpsURLConnection)url.openConnection();
conn.setDoInput(true);
conn.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36");
conn.addRequestProperty("Cookie", "xxx");
conn.addRequestProperty("Content-Length", "406");
conn.addRequestProperty("Content-Type", "application/x-www-form-urlencoded; charset=UTF-8");
conn.addRequestProperty("Accept-Encoding", "gzip,deflate");
conn.addRequestProperty("Connection", "keep-alive");
conn.addRequestProperty("Referer", "xxx");
conn.setDoOutput(true);
OutputStreamWriter writer = new OutputStreamWriter(conn.getOutputStream());
writer.write("xxx");
writer.flush();
String line;
BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream()));
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
writer.close();
reader.close();
}
You are explicitly requesting a zipped stream:
conn.addRequestProperty("Accept-Encoding", "gzip,deflate");
Remove that line to remove the compression. Compression is different from encryption fortunately, it doesn't require you to supply a key.
I am getting
Http 403 code
When I try to access that URL using my application
URL url = new URL(path);
url.openStream();
URL url = new URL(downloadURL);
/*URLConnection conn = url.openConnection();
conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2");*/
HttpURLConnection httpcon = (HttpURLConnection) url.openConnection();
httpcon.setRequestProperty("User-Agent","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.102 Safari/537.36");
httpcon.connect();
//url.getFile();
InputStream is = httpcon.getInputStream();
//InputStream is = conn.getInputStream();
But I can open the URL which I am using in my application ..
Can anybody please help
I can access that URL In my browser...
That URL is from some server(local) not public.
URL url = new URL(path);
URLConnection conn = url.openConnection();
conn.getInputStream();
Try this way.