We Keep Coding

HttpURLConnection with https InputStream Garbled

I use HttpURLConnection to crawler https://translate.google.com/.
InetSocketAddress addr = new InetSocketAddress("127.0.0.1", 1082);
Proxy proxy = new Proxy(Proxy.Type.HTTP, addr);
url = new URL("https://translate.google.com/");
HttpURLConnection conn = (HttpURLConnection) url.openConnection(proxy);
conn.setRequestProperty("Accept-Encoding", "gzip, deflate, sdch");
conn.setRequestProperty("Connection", "keep-alive");
conn.setRequestProperty("User-Agent",
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36");
conn.setRequestProperty("Accept", "*/*");
Map<String, List<String>> reqHeaders = conn.getHeaderFields();
List<String> reqTypes = reqHeaders.get("Content-Type");
for (String ss : reqTypes) {
System.out.println(ss);
}
InputStream in = conn.getInputStream();
String s = IOUtils.toString(in, "UTF-8");
System.out.println(s.substring(0, 100));
Map<String, List<String>> resHeader = conn.getHeaderFields();
List<String> resTypes = resHeader.get("Content-Type");
for (String ss : resTypes) {
System.out.println(ss);
}
Console is
But When I change url to http://translate.google.com/.
It works well.
I know actually HttpURLConnection is HttpsURLConnection when i crawler https://translate.google.com/.
I try to use HttpsURLConnection and it still garbled.
Any suggestions?

conn.setRequestProperty("Accept-Encoding", "gzip, deflate, sdch");
The response is compressed, because the above line tells the server that the client is able to understand encodings specified in Accept-Encoding.
Try to comment this line or handle this situation.
There's a more specific implementation for HTTPS i.e. HttpsURLConnection, in case you're interested in https-specific features, e.g.:
import javax.net.ssl.HttpsURLConnection;
....
URL url = new URL("https://www.google.com/");
HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();

I accept Jerry Chin's answer.Solves my problem.
My answer just recording how i resolve this problem.
If this approach is unreasonable.Let me know, I'll remove this answer.
conn.setRequestProperty("Accept-Encoding", "gzip, deflate, sdch");
And then I check response Content-Encoding.It's gzip.
So i use GZIPInputStream to receive.
InputStream in = conn.getInputStream();
GZIPInputStream gzis=new GZIPInputStream(in);
InputStreamReader reader = new InputStreamReader(gzis);
BufferedReader br = new BufferedReader(reader);
The InputStream is normal.
BTW,If you don't need Accept-Encoding,you can remove it.
And do not forget check user-agent. It's very important and different operating systems corresponding to different user-agent.

Java send request encoding

I'm trying to send get request in java, when it's without headers as below, I get redirected to login page.
URL myURL1 = new URL("http://www.eventlister.com");
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("GET");
But when request is sent with all headers as it is sent from browser when reading response I get it encoded like this
???}isI??g)b?CZ?n?7BH?:m?u??dkzz; $P????C2?????YT??;??B3m????7/?.o~?p,?????..., even if encoding is set manually in reader.
URL myURL1 = new URL("http://www.eventlister.com");
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("GET");
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36");
con.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
con.setRequestProperty("Accept-Encoding", "gzip, deflate, sdch");
con.setRequestProperty("Accept-Language", "en-US,en;q=0.8,zh-CN;q=0.6,zh;q=0.4,ru;q=0.2,uk;q=0.2,ja;q=0.2,fr-FR;q=0.2,fr;q=0.2");
con.setRequestProperty("Connection", "keep-alive");
con.setRequestProperty("Upgrade-Insecure-Requests", "1");
con.setRequestProperty("Host", "www.eventlister.com");
BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream(), StandardCharsets.UTF_8));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
Thank you in advance.

It looks like it's gzip'ed data. Try it without "Accept-Encoding: gzip", so that server returns plain data.
I know this is a comment, not an answer. I don't have comment privilege yet :-(

Sending post request to website

I'm trying to use java to log in to a website and then eventually retrieve the cookie in order to gain access to information on the site. It seems my post request is working but I am receiving a response code of 500. I was wondering if this is because I have formatted the post data incorrectly
When using the website it says the post is formatted as below
{userName: "dfh", password: "suyj"}
In my code I have used this
String urlParameters = "userName:dfh,Password:suyj";
con.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
wr.writeBytes(urlParameters);
wr.flush();
wr.close();
Am I right in thinking this could be the issue, and if so how can I correct my post data format?
Below is my whole code
public class post {
public static void main(String[] args) throws IOException {
final String USER_AGENT = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0";
String url = "website";
URL obj = new URL(url);
HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();
//add reuqest header
con.setRequestMethod("POST");
con.setRequestProperty("User-Agent", USER_AGENT);
con.setRequestProperty("Accept", "application/json, text/javascript, */*; q=0.01");
con.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
con.setRequestProperty("Accept-Encoding", "gzip, deflate");
con.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
con.setRequestProperty("X-Requested-With", "XMLHttpRequest");
con.setRequestProperty("Referer", "https://extbasicph05.podc.sl.edst.ibm.com/FFAMobileRelay/");
con.setRequestProperty("Content-Length", "64");
con.setRequestProperty("Connection", "keep-alive");
con.setRequestProperty("Referer", "https://extbasicph05.podc.sl.edst.ibm.com/FFAMobileRelay/");
con.setRequestProperty("Cookies", "UnicaNIODID=KGew34gcvZ5-Y2NoBQr; mmid=-1658088866%7CKAAAAAr15Tc8FgsAAA%3D%3D; mmcore.pd=1484641984%7CejEyAAoBQvXlNzwWCxPPXGheAVibftSXgNJIDwAAADoOVvoDstFIAAAAAP//////////AAZEaXJlY3QBFgwIAAYABQAAAAAAAP+MAACAigAA/4wAAAIAxDEAAABFBVUW6QsA/////wHpCxoM//83AAAAAAAAAAOSfwAAusgAAJN/AAAgyAAAlH8AACHIAAABIYAAAAIAAADVNgAAAPcCB+70CwD/////AfQLHQz//60CAAEAAAAAAX+KAAAp2gAAAoCKAABAWBAAgYoAALMAAAAAAAABRQ%3D%3D; mmcore.srv=ldnvwcgus03; IBM_W3SSO_ACCESS=w3-03.sso.ibm.com; CoreID6=32675392371714128784599&ci=51040000|IBMTEST_51040000|IBMTESTW3_50200000|IBMTESTWWW; CoreM_State=75~-1~-1~-1~-1~3~3~5~3~3~7~7~|~~|~~|~~|~||||||~|~~|~~|~~|~~|~~|~~|~~|~; CoreM_State_Content=6~|~EE9EDB09923CD77F~|~0; PrefID=222-11978519; CSList=23427744/17540903,0/0,0/0,0/0,0/0; _ga=GA1.2.233924805.1433836377; mmcore.tst=0.169; ibmSurvey=1435758135373; 50200000_clogin=v=1&l=1435758136&e=1435760174616");
String urlParameters = "userName:dfh,Password:suyj";
// Send post request
con.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
wr.writeBytes(urlParameters);
wr.flush();
wr.close();
int responseCode = con.getResponseCode();
System.out.println("\nSending 'POST' request to URL : " + url);
System.out.println("Post parameters : " + urlParameters);
System.out.println("Response Code : " + responseCode);
System.out.println(con.getErrorStream());
BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
//print result
System.out.println(response.toString());
}
}

You have already found out your problem. Just fix it...
Instead of String urlParameters = "userName:dfh,Password:suyj";, use
String urlParameters = "{userName:\"dfh\",Password:\"suyj\"}";
Though I believe your server expects JSON formatted post parameter. In that case you should use,
String urlParameters = "{\"userName\":\"dfh\",\"Password\":\"suyj\"}";
Or use a JSON parser to create the parameters.

Java HttpURLConnection trying to login with cookie

So am trying to login this website using java but for some reason its not working as expected i got the host and all that stuff but its not going to the account page with the cookie it still shows the login page and yes my account info is correct any help is great
public static void main(String[] args) {
try {
String params = "loginEmail=private#hotmail.com&loginPassword=privatepassword&Submit=Sign+In";
String urls = "http://www.filefactory.com/member/signin.php";
URL url = new URL(urls);
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type",
"application/x-www-form-urlencoded");
connection.setRequestProperty("Host", "www.filefactory.com");
connection.setRequestProperty("Referer", "http://www.filefactory.com/member/signin.php");
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.101 Safari/537.36 OPR/25.0.1614.50");
connection.setRequestProperty("Content-Length", "" +
Integer.toString(params.getBytes().length));
connection.setRequestProperty("Content-Language", "en-US");
connection.setDoInput(true);
connection.setDoOutput(true);
//Send request
DataOutputStream wr = new DataOutputStream (
connection.getOutputStream ());
wr.writeBytes (params);
wr.flush ();
wr.close ();
//Get Response
InputStream is = connection.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is));
String line;
StringBuilder response = new StringBuilder();
while((line = rd.readLine()) != null) {
response.append(line);
response.append('\r');
}
rd.close();
System.out.println(response.toString());
// get the cookie if need, for login
String cookies = connection.getHeaderField("Set-Cookie");
// open the new connnection again
connection = (HttpURLConnection) new URL("http://www.filefactory.com/account/").openConnection();
connection.setRequestProperty("Cookie", cookies);
connection.addRequestProperty("Accept-Language", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
connection.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.101 Safari/537.36 OPR/25.0.1614.50");
connection.addRequestProperty("Host", "www.filefactory.com");
System.out.println("Redirect to URL : " + "http://www.filefactory.com/account/");
BufferedReader in = new BufferedReader(
new InputStreamReader(connection.getInputStream()));
String inputLine;
StringBuilder html = new StringBuilder();
while ((inputLine = in.readLine()) != null) {
html.append(inputLine);
}
in.close();
System.out.println("URL Content... \n" + html.toString());
System.out.println("Done");
} catch (Exception e) {
e.printStackTrace();
}
}
}

You are using : String cookies = connection.getHeaderField("Set-Cookie");
Are you sure there is only one entry for that header? There could be more.
http://en.wikipedia.org/wiki/HTTP_cookie
Try using chrome or firefox and try logging in manually to capture the request and response. That may give you some hints regarding what could be wrong.
Additionally you could use a tool to view the communication between your client and the server (unless you are already doing so)

It's hard to tell, not knowing the exact way that website works, but you should note that it sends you the cookies first when it presents the login page to you. When you send in your credentials you have to already send them together with the cookies, so that it knows to associate those credentials with this cookie.

How to send post parameter which can be read by php

I have some connection to url:
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
How I can send post parameter to this url that php function: http://www.php.net/manual/en/reserved.variables.request.php can read it ? I try to according this question: Java - sending HTTP parameters via POST method easily but with no success. It just display empty array
php code:
print_r($_REQUEST,1)
java code:
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setDoInput(true);
connection.setInstanceFollowRedirects(false);
connection.setRequestMethod("POST");
connection.setRequestProperty("category_name", categoryName);
connection.setRequestProperty("complete", complete);
DataOutputStream wr = new DataOutputStream(connection.getOutputStream());
connection.getResponseCode();
wr.flush();
wr.close();
connection.disconnect();

You forgot to set the Content-Type header, and you need to write the category_name and complete values as a string to the stream.
Quoting the code from the post you mentioned:
String urlParameters = "param1=a&param2=b&param3=c";
String request = "http://example.com/index.php";
URL url = new URL(request);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setDoInput(true);
connection.setInstanceFollowRedirects(false);
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
connection.setRequestProperty("charset", "utf-8");
connection.setRequestProperty("Content-Length", "" + Integer.toString(urlParameters.getBytes().length));
connection.setUseCaches (false);
DataOutputStream wr = new DataOutputStream(connection.getOutputStream ());
wr.writeBytes(urlParameters);
wr.flush();
wr.close();
connection.disconnect();
As you can see, he sets the headers (Content-Type, Charset and Content-Length) and then writes the POST data to the stream.
The POST data is just like the GET format: key-value pairs seperated by &, and the key and value are seperated by =.
As the Content-Type is set to www-form-urlencoded, you need to 'url encode' the key and value values. You can do this using the URLEncoder.encode method.
The urlParameters for your POST data would be:
String urlParameters = "category_name=" + URLEncoder.encode(categoryName) + "&complete=" + URLEncoder.encode(complete);