We Keep Coding

HttpURLConnection with https InputStream Garbled

I use HttpURLConnection to crawler https://translate.google.com/.
InetSocketAddress addr = new InetSocketAddress("127.0.0.1", 1082);
Proxy proxy = new Proxy(Proxy.Type.HTTP, addr);
url = new URL("https://translate.google.com/");
HttpURLConnection conn = (HttpURLConnection) url.openConnection(proxy);
conn.setRequestProperty("Accept-Encoding", "gzip, deflate, sdch");
conn.setRequestProperty("Connection", "keep-alive");
conn.setRequestProperty("User-Agent",
"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36");
conn.setRequestProperty("Accept", "*/*");
Map<String, List<String>> reqHeaders = conn.getHeaderFields();
List<String> reqTypes = reqHeaders.get("Content-Type");
for (String ss : reqTypes) {
System.out.println(ss);
}
InputStream in = conn.getInputStream();
String s = IOUtils.toString(in, "UTF-8");
System.out.println(s.substring(0, 100));
Map<String, List<String>> resHeader = conn.getHeaderFields();
List<String> resTypes = resHeader.get("Content-Type");
for (String ss : resTypes) {
System.out.println(ss);
}
Console is
But When I change url to http://translate.google.com/.
It works well.
I know actually HttpURLConnection is HttpsURLConnection when i crawler https://translate.google.com/.
I try to use HttpsURLConnection and it still garbled.
Any suggestions?

conn.setRequestProperty("Accept-Encoding", "gzip, deflate, sdch");
The response is compressed, because the above line tells the server that the client is able to understand encodings specified in Accept-Encoding.
Try to comment this line or handle this situation.
There's a more specific implementation for HTTPS i.e. HttpsURLConnection, in case you're interested in https-specific features, e.g.:
import javax.net.ssl.HttpsURLConnection;
....
URL url = new URL("https://www.google.com/");
HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();

I accept Jerry Chin's answer.Solves my problem.
My answer just recording how i resolve this problem.
If this approach is unreasonable.Let me know, I'll remove this answer.
conn.setRequestProperty("Accept-Encoding", "gzip, deflate, sdch");
And then I check response Content-Encoding.It's gzip.
So i use GZIPInputStream to receive.
InputStream in = conn.getInputStream();
GZIPInputStream gzis=new GZIPInputStream(in);
InputStreamReader reader = new InputStreamReader(gzis);
BufferedReader br = new BufferedReader(reader);
The InputStream is normal.
BTW,If you don't need Accept-Encoding,you can remove it.
And do not forget check user-agent. It's very important and different operating systems corresponding to different user-agent.

Sending post request to website

I'm trying to use java to log in to a website and then eventually retrieve the cookie in order to gain access to information on the site. It seems my post request is working but I am receiving a response code of 500. I was wondering if this is because I have formatted the post data incorrectly
When using the website it says the post is formatted as below
{userName: "dfh", password: "suyj"}
In my code I have used this
String urlParameters = "userName:dfh,Password:suyj";
con.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
wr.writeBytes(urlParameters);
wr.flush();
wr.close();
Am I right in thinking this could be the issue, and if so how can I correct my post data format?
Below is my whole code
public class post {
public static void main(String[] args) throws IOException {
final String USER_AGENT = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0";
String url = "website";
URL obj = new URL(url);
HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();
//add reuqest header
con.setRequestMethod("POST");
con.setRequestProperty("User-Agent", USER_AGENT);
con.setRequestProperty("Accept", "application/json, text/javascript, */*; q=0.01");
con.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
con.setRequestProperty("Accept-Encoding", "gzip, deflate");
con.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
con.setRequestProperty("X-Requested-With", "XMLHttpRequest");
con.setRequestProperty("Referer", "https://extbasicph05.podc.sl.edst.ibm.com/FFAMobileRelay/");
con.setRequestProperty("Content-Length", "64");
con.setRequestProperty("Connection", "keep-alive");
con.setRequestProperty("Referer", "https://extbasicph05.podc.sl.edst.ibm.com/FFAMobileRelay/");
con.setRequestProperty("Cookies", "UnicaNIODID=KGew34gcvZ5-Y2NoBQr; mmid=-1658088866%7CKAAAAAr15Tc8FgsAAA%3D%3D; mmcore.pd=1484641984%7CejEyAAoBQvXlNzwWCxPPXGheAVibftSXgNJIDwAAADoOVvoDstFIAAAAAP//////////AAZEaXJlY3QBFgwIAAYABQAAAAAAAP+MAACAigAA/4wAAAIAxDEAAABFBVUW6QsA/////wHpCxoM//83AAAAAAAAAAOSfwAAusgAAJN/AAAgyAAAlH8AACHIAAABIYAAAAIAAADVNgAAAPcCB+70CwD/////AfQLHQz//60CAAEAAAAAAX+KAAAp2gAAAoCKAABAWBAAgYoAALMAAAAAAAABRQ%3D%3D; mmcore.srv=ldnvwcgus03; IBM_W3SSO_ACCESS=w3-03.sso.ibm.com; CoreID6=32675392371714128784599&ci=51040000|IBMTEST_51040000|IBMTESTW3_50200000|IBMTESTWWW; CoreM_State=75~-1~-1~-1~-1~3~3~5~3~3~7~7~|~~|~~|~~|~||||||~|~~|~~|~~|~~|~~|~~|~~|~; CoreM_State_Content=6~|~EE9EDB09923CD77F~|~0; PrefID=222-11978519; CSList=23427744/17540903,0/0,0/0,0/0,0/0; _ga=GA1.2.233924805.1433836377; mmcore.tst=0.169; ibmSurvey=1435758135373; 50200000_clogin=v=1&l=1435758136&e=1435760174616");
String urlParameters = "userName:dfh,Password:suyj";
// Send post request
con.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
wr.writeBytes(urlParameters);
wr.flush();
wr.close();
int responseCode = con.getResponseCode();
System.out.println("\nSending 'POST' request to URL : " + url);
System.out.println("Post parameters : " + urlParameters);
System.out.println("Response Code : " + responseCode);
System.out.println(con.getErrorStream());
BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
//print result
System.out.println(response.toString());
}
}

You have already found out your problem. Just fix it...
Instead of String urlParameters = "userName:dfh,Password:suyj";, use
String urlParameters = "{userName:\"dfh\",Password:\"suyj\"}";
Though I believe your server expects JSON formatted post parameter. In that case you should use,
String urlParameters = "{\"userName\":\"dfh\",\"Password\":\"suyj\"}";
Or use a JSON parser to create the parameters.

How to decrypt HttpsURLConnection

My code makes POST request to a website, but the response looks like encrypted, with weird characters. When i configure my app to use fiddler proxy, the response is valid. How can i make my app decrypt this response?
public class Login {
public Login() throws Exception {
URL url = new URL("https://account.leagueoflegends.com/auth");
HttpsURLConnection conn = (HttpsURLConnection)url.openConnection();
conn.setDoInput(true);
conn.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36");
conn.addRequestProperty("Cookie", "xxx");
conn.addRequestProperty("Content-Length", "406");
conn.addRequestProperty("Content-Type", "application/x-www-form-urlencoded; charset=UTF-8");
conn.addRequestProperty("Accept-Encoding", "gzip,deflate");
conn.addRequestProperty("Connection", "keep-alive");
conn.addRequestProperty("Referer", "xxx");
conn.setDoOutput(true);
OutputStreamWriter writer = new OutputStreamWriter(conn.getOutputStream());
writer.write("xxx");
writer.flush();
String line;
BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream()));
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
writer.close();
reader.close();
}

You are explicitly requesting a zipped stream:
conn.addRequestProperty("Accept-Encoding", "gzip,deflate");
Remove that line to remove the compression. Compression is different from encryption fortunately, it doesn't require you to supply a key.

Java HttpURLConnection trying to login with cookie

So am trying to login this website using java but for some reason its not working as expected i got the host and all that stuff but its not going to the account page with the cookie it still shows the login page and yes my account info is correct any help is great
public static void main(String[] args) {
try {
String params = "loginEmail=private#hotmail.com&loginPassword=privatepassword&Submit=Sign+In";
String urls = "http://www.filefactory.com/member/signin.php";
URL url = new URL(urls);
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type",
"application/x-www-form-urlencoded");
connection.setRequestProperty("Host", "www.filefactory.com");
connection.setRequestProperty("Referer", "http://www.filefactory.com/member/signin.php");
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.101 Safari/537.36 OPR/25.0.1614.50");
connection.setRequestProperty("Content-Length", "" +
Integer.toString(params.getBytes().length));
connection.setRequestProperty("Content-Language", "en-US");
connection.setDoInput(true);
connection.setDoOutput(true);
//Send request
DataOutputStream wr = new DataOutputStream (
connection.getOutputStream ());
wr.writeBytes (params);
wr.flush ();
wr.close ();
//Get Response
InputStream is = connection.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is));
String line;
StringBuilder response = new StringBuilder();
while((line = rd.readLine()) != null) {
response.append(line);
response.append('\r');
}
rd.close();
System.out.println(response.toString());
// get the cookie if need, for login
String cookies = connection.getHeaderField("Set-Cookie");
// open the new connnection again
connection = (HttpURLConnection) new URL("http://www.filefactory.com/account/").openConnection();
connection.setRequestProperty("Cookie", cookies);
connection.addRequestProperty("Accept-Language", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
connection.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.101 Safari/537.36 OPR/25.0.1614.50");
connection.addRequestProperty("Host", "www.filefactory.com");
System.out.println("Redirect to URL : " + "http://www.filefactory.com/account/");
BufferedReader in = new BufferedReader(
new InputStreamReader(connection.getInputStream()));
String inputLine;
StringBuilder html = new StringBuilder();
while ((inputLine = in.readLine()) != null) {
html.append(inputLine);
}
in.close();
System.out.println("URL Content... \n" + html.toString());
System.out.println("Done");
} catch (Exception e) {
e.printStackTrace();
}
}
}

You are using : String cookies = connection.getHeaderField("Set-Cookie");
Are you sure there is only one entry for that header? There could be more.
http://en.wikipedia.org/wiki/HTTP_cookie
Try using chrome or firefox and try logging in manually to capture the request and response. That may give you some hints regarding what could be wrong.
Additionally you could use a tool to view the communication between your client and the server (unless you are already doing so)

It's hard to tell, not knowing the exact way that website works, but you should note that it sends you the cookies first when it presents the login page to you. When you send in your credentials you have to already send them together with the cookies, so that it knows to associate those credentials with this cookie.

Post Request clone on Android

Im trying to perform the folowing request in my android app.
The Request i send was on a Chrome Browser.
In the Request i shortend the Form Data so there is not so much code.
Request URL:http://***/
Request Method:POST
Status Code:200 OK
Request Headers
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding:gzip,deflate,sdch
Accept-Language:de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4
Cache-Control:max-age=0
Connection:keep-alive
Content-Length:424
Content-Type:application/x-www-form-urlencoded
Cookie:__utma=188893489.1646114392.1358703936.1367178892.1368783485.29; __utmz=188893489.1365594840.21.3.utmcsr=***|utmccn=(referral)|utmcmd=referral|utmcct=/
Host:***
Origin:***
Referer:**
User-Agent:Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36
Form Data
date=1375480155&mail=&dfBoot=Test
Response Headers
Connection:Keep-Alive
Content-Encoding:gzip
Content-Length:3614
Content-Type:text/html
Date:Fri, 02 Aug 2013 21:49:59 GMT
Keep-Alive:timeout=15, max=100
Server:Apache/2.2.14 (Ubuntu)
Vary:Accept-Encoding
X-Powered-By:PHP/5.3.2-1ubuntu4.20
with this code:
StrictMode.ThreadPolicy policy = new StrictMode.ThreadPolicy.Builder()
.permitAll().build();
StrictMode.setThreadPolicy(policy);
URL url = new URL("http://***/index.php");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
try {
connection.setRequestMethod("POST");
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setUseCaches(false);
connection.setRequestProperty("Content-Type","application/x-www-form-urlencoded");
connection.setRequestProperty("Content-Length",String.valueOf(post.length()));
//connection.setChunkedStreamingMode(0);//results in frezing
OutputStreamWriter writer = new OutputStreamWriter(
connection.getOutputStream());
writer.write(post);
writer.flush();
InputStream in = new BufferedInputStream(
connection.getInputStream());
while (in.available() != 0) {
in.read();
}
writer.close();
in.close();
} finally {
connection.disconnect();
}
The code is very messed up because i tryed manytime to fix my connection problem.
Please help with a better Solution.

Hm, I will show my working code with JSON, but maybe you can modify it.
URL url = new URL("http://xcxcxcxcxcx");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestProperty("Content-Type", "application/json");
connection.setRequestProperty("charset", "utf-8");
connection.setRequestProperty("Content-Length", "" + Integer.toString(jsonObject.toString().getBytes().length));
connection.setRequestMethod("POST");
connection.setDoOutput(true);
connection.connect();
DataOutputStream out = new DataOutputStream(connection.getOutputStream());
out.writeBytes(jsonObject.toString());
out.flush();
out.close();
InputStream is = connection.getInputStream();
InputStreamReader reader = new InputStreamReader(is);
BufferedReader r = new BufferedReader(reader);
StringBuilder total = new StringBuilder();
String line;
while ((line = r.readLine()) != null) {
total.append(line);
}
JSONObject jsonObjectResult = null;
if(typeOfMethod == 0){
JSONTokener tokenizer = new JSONTokener(total.toString());
jsonObjectResult = new JSONObject(tokenizer);
}
connection.disconnect();
return jsonObjectResult;
There is some differences between getting result from InputStream.