We Keep Coding

How to print the whole response on the console

I can't print the whole response from the server on the console!
,
There are 3 ways to bypass this matter,
Add this header Connection: close
Replace HTTP/1.1 to HTTP/1.0
Add this s.close(); // Socket.close();
I can't close the connection because I want to send more than once at the same connection,
I just want to print the whole response without closing the connection.
String content = "GET /Zuck HTTP/1.1\r\nHost: www.facebook.com\r\nuser-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36\r\n\r\n";

Executing your code returns the following:
HTTP/1.1 302 Found
Location: https://www.facebook.com/zuck
Strict-Transport-Security: max-age=15552000; preload
Content-Type: text/html; charset="utf-8"
X-FB-Debug: NHDnNLmTeg5PBPiSL7++1dz/ZdRbnlnKy1gpdfBbLFkvrhbJMJT+nLJd1VYpmEkkkUtmvXsjgLvFEeML/82WUA==
Date: Thu, 18 Jun 2020 15:36:24 GMT
Alt-Svc: h3-27=":443"; ma=3600
Connection: keep-alive
Content-Length: 0
HTTP reponse status code 302 indicates a redirect to the Location: https://www.facebook.com/zuck. Either handle redirects in your code or - to get your example running - simply replace Zuck with zuck in your content string.

Since your operating on raw socket you actually cannot determine when you have received whole response. You can however do it with protocols like http in same cases.
In your example you receive Content-Length: 0 which tells the number (0) of bytes the body of message have.
You can also pass header Connection: close which closes connection after sending full response, but I think it is not what you're looking for.
You can also just do read/write operations on two separate threads.

What is the proper method to parse and send HTTP requests using sockets in Java?

I'm creating a basic local proxy server, the goal is to accept http and https traffic from my web browser, parse it for information, send and receive the requests to the proper host, and then return it to the web browser.
I currently have an open socket to my web browser. I am receiving both http and https requests from the browser like so:
HTTP:
GET http://example.com/ HTTP/1.1
Host: example.com User-Agent:
Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Upgrade-Insecure-Requests: 1
HTTPS:
CONNECT example.com:443 HTTP/1.1
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0
Proxy-Connection: keep-alive
Connection: keep-alive
Host: example.com:443
I open a socket to the "Host:" from the above with the following code:
public void sendRequest() throws IOException{
Socket socket = new Socket(host, port);
//socket.getInputStream.read();
BufferedWriter out = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream(), "UTF8"));
BufferedReader in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
for(int i = 0; i < lines.size(); i++){
out.write(lines.get(i) + "\r\n");
}
out.flush();
outputReturn(in);
}
And I receive the reply like so:
public void outputReturn(BufferedReader in){
try{
System.out.println("\n * Response");
String line;
while ((line = in.readLine()) != null) {
System.out.println(line);
}
}
catch (IOException i){
System.out.println(i);
}
}
The replies come back as so:
HTTP:
* Response
HTTP/1.1 200 OK
Content-Encoding: gzip
Accept-Ranges: bytes
Cache-Control: max-age=604800
Content-Type: text/html; charset=UTF-8
Date: Wed, 10 Apr 2019 22:53:28 GMT
Etag: "1541025663+gzip"
Expires: Wed, 17 Apr 2019 22:53:28 GMT
Last-Modified: Fri, 09 Aug 2013 23:54:35 GMT
Server: ECS (ord/4C92)
Vary: Accept-Encoding
X-Cache: HIT
Content-Length: 606
;�R�TA��0
��W�ri]��S�V #���1k��Z��$�6���q۽���#+���l�I�I��s�PzUe���Bf
�'��+�>���+�OF �I4h��^#^
�ЧA�p#�M���u����������*
<�|ԅߎP���P�-�6�O��$}�Jl)ǰ_,�4yU�rQazw�r���t
.�s���3�
z�_������2�Mel
ϋ5����%�t
뫪R���t3
��:�|�Q��]���
V-z�|�Y3*���rKp�5th��"��C���NH����v��OOyޣ�xs�����V��$��X�6�BR�b�C��PqE���K�<� �G�כ7����E(17Vx2�US��
% x��)�d�����e��O&�4/䤘���~��Oi�s�X�dW�7��#�u�"��y\$]j<�L�r�˻'�ɪ�Vg?Kr {=��΋]E��^x;�ƱX
TU��]�[�{��s+�e����9�g���]����H�4���#�KA��'�Z�����*r�
�$�G� ��4�n�8���㊄+c���E�hA��X���������L��RIt�[4\����
HTTPS:
CONNECT getpocket.cdn.mozilla.net:443 HTTP/1.1
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0
Proxy-Connection: keep-alive
Connection: keep-alive
Host: getpocket.cdn.mozilla.net:443
* Response
java.net.SocketException: Connection reset
Questions:
Why do I receive what seems like binary from the HTTP request?
Why so I receive nothing from my HTTPS request?
What SHOULD I be doing instead?
Thanks in advance.

For your HTTP request, the Content-Encoding is gzip. The binary is the gzip-compressed data.
For your HTTPS request, you're not making an SSL/TLS handshake, so the server drops the connection.
For HTTP, I don't think you need to do anything, the browser should handle it for you. There's no feasible way to proxy an HTTPS/SSL/TLS using the method you described.

Rebuild HTTP flow with incubated Java 10 HttpClient

I'm trying to rebuild the session setup to a web server by a Java HttpClient application. I have chosen the incubated HttpClient provided with Java 9 and Java 10.
With Chrome I captured this headers from a single request:
General
Request URL: https://<some_url>?user_id=1176&onlyDirectUserItems=true&onlyAssignedToUser=true&show=Unresolved&itemsFilter=0
Request Method: GET
Status Code: 302 Found
Remote Address: <theProxy>:8000
Referrer Policy: no-referrer-when-downgrade
Response Headers
Connection: Keep-Alive
Content-Length: 164
Content-Type: text/html; charset=iso-8859-1
Date: Fri, 08 Jun 2018 14:33:16 GMT
Keep-Alive: timeout=300, max=100
Location: https://<another_url>:443/nesp/app/plogin?agAppNa=app_me_company_ext&c=secure/name/password/uri&target=%22https://<another-usr>/browseIssues.spr?user_id=1176&onlyDirectUserItems=true&onlyAssignedToUser=true&show=Unresolved&itemsFilter=0%22
P3p: CP="NOI"
Server: Apache
Set-Cookie: IPCZQX03224bfb75=030003000000000000000000000000008f7aed69; path=/; domain=.me.de
Via: 1.1 <host> (Access Gateway-ag-7169149846802036-13837511)
Request Headers
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
Connection: keep-alive
Cookie: org.ditchnet.jsp.tabs.wiki=wiki-wysiwyg; ZNPCQ003-31393000=6c2f99a3; ZNPCQ003-32323200=cd188fdd
DNT: 1
Host: <host>
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
Query String Parameters
user_id: 1176
onlyDirectUserItems: true
onlyAssignedToUser: true
show: Unresolved
itemsFilter: 0
What can be seen the Response Header provides a URL (header-key: "location") which I need to grab and call next. But with my http client I fail with status-code 400 and get almost nothing
This is my code
url = "https://<some_url>?user_id=1176&onlyDirectUserItems=true&onlyAssignedToUser=true&show=Unresolved&itemsFilter=0";
HttpClient client = HttpClient.newBuilder()
.proxy(ProxySelector.of(new InetSocketAddress("<theProxy>", 8000)))
.cookieHandler(new CookieManager(null, CookiePolicy.ACCEPT_ALL))
.followRedirects(HttpClient.Redirect.SAME_PROTOCOL)
.build();
HttpRequest request = HttpRequest.newBuilder()
.header("User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0")
.header("Upgrade-Insecure-Requests", "1")
// .header("Host", "<host>")
.header("Connection", "keep-alive")
.header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8")
.header("Accept-Language", "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7")
.header("Accept-Encoding", "gzip, deflate, br")
.uri(new URI(url))
.build();
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandler.asString());
HttpHeaders headers = response.headers();
Map<String, List<String>> headerMap= headers.map();
for (String key : headerMap.keySet()) {
System.out.println(">"+key+"<");
for (String value : headerMap.get(key)) {
System.out.println(" " + value);
}
}
System.out.println(response.statusCode());
System.out.println(response.body());
I have no clue what might be wrong and how to proceed to get this done. I hope someone can tell me what to ty next.
What I also do not understand: I had to remove the header "Host" - because I got the response: "Your browser sent a request that this server could not understand."
The very same header as can be found in the Chrome-listing

I could get it run with Apache HttpClient. I cannot tell what is wrong with the incubated HttpClient - but there is for sure a reason that it is not a fully integrated part of the Java 9/10 delivery

Getting 500 error from HttpClient, works in browser

I'm using Apache HttpClient to try to submit some post data to a server. Unfortunately, I don't have access to the server to get any log information so that won't be possible.
If I go through this process with Firefox, it works fine. (I do get a 302 warning on this particular page)
I have matched the Request headers of both Firefox and my program.
Firefox Request Headers:
Host: server ip
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://server ip/
Content-Type: application/x-www-form-urlencoded
Content-Length: 407
Cookie: sessionId=blahblah
Connection: keep-alive
Upgrade-Insecure-Requests: 1
My Programs Request Headers shown from context.getRequest().getAllHeaders();
Host: server ip
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:55.0) Gecko/20100101 Firefox/55.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://server ip/
Content-Type: application/x-www-form-urlencoded
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Content-Length: 406
Cookie: sessionId=blahblah
I have matched the body of the request by comparing the output of EntityUtils.toString(httpPost.getEntity(), "UTF-8"); and the built in tool for Firefox's tool to look at the request body, and they match almost character for character. (Just a slight difference in the session id which is expected as it's not using the same session.)
I'm not sure what else to check. What could be causing the server to behave differently between the Browser and the program?
Below is my code for the POST request.
HttpPost httpPost = new HttpPost("https://" + getIp() + "");
List<NameValuePair> params = new ArrayList<NameValuePair>();
params.add(new BasicNameValuePair("FTPUsername", "blah"));
params.add(new BasicNameValuePair("FTPPassword", "blah"));
params.add(new BasicNameValuePair("FormButtonSubmit", "OK"));
httpPost.setEntity(new UrlEncodedFormEntity(params));
httpPost.setHeader("Host", ip);
httpPost.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:55.0) Gecko/20100101 Firefox/55.0");
httpPost.setHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
httpPost.setHeader("Accept-Language", "en-US,en;q=0.5");
httpPost.setHeader("Accept-Encoding", "gzip, deflate, br");
httpPost.setHeader("Referer", referer);
httpPost.setHeader("Content-Type", "application/x-www-form-urlencoded");
httpPost.setHeader("Connection", "keep-alive");
httpPost.setHeader("Upgrade-Insecure-Requests", "1");
//Response
HttpResponse response = getHttpClient().execute(httpPost, LoginRequest.context);
int statusCode = response.getStatusLine().getStatusCode();
httpPost.releaseConnection();
I realize this could probably be many things since 500 is a server error, but it's got to be something I'm submitting wrong or I'm missing something as it works perfectly in the browser.

302 "warning" is actually a redirect. HTTP Client does do redirect automatically, you must flag the RedirectStrategy, For HttpClient 4.3:
HttpClient instance = HttpClientBuilder.create()
.setRedirectStrategy(new LaxRedirectStrategy()).build();
see examples in answer and w3 docs:
If the 302 status code is received in response to a request other than
GET or HEAD, the user agent MUST NOT automatically redirect the
request unless it can be confirmed by the user

Do you work with Windows machine? or Linux machine?
If you use a windows machine, have you tried working with WAMP server for Linux use LAMP server, so if you install it, you won't get those errors, that's how I fixed my error. Once if you install these two servers, change the port number in skype by logging into Skype and change the port number or uninstall your skype. It should work.

Copy Contents of HttpMethod to another HttpMethod

I have been tryig to handle a redirect(302) in java code and now that I am done doing it by my code. I ran into an other problem in which after login, on click on any link I get logged out. So I checked my TCP Stream through wireshark and found that there are few HeaderRequests missing. After implementation of my code, Http Header are as follows :
GET /index.php/ HTTP/1.1
Host: 10.28.161.31
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.24) Gecko/20111109 CentOS/3.6-3.el5.centos Firefox/3.6.24
Cookie: PHPSESSID=d488eea5e85afc8ec526c1a749e7ab20; path=/
Referrer: http://10.28.161.31
Cookie: $Version=0; PHPSESSID=d488eea5e85afc8ec526c1a749e7ab20; $Path=/ ???
and original Http Headers are as follows :
GET / HTTP/1.1
Host: 10.28.161.31
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.24) Gecko/20111109 CentOS/3.6-3.el5.centos Firefox/3.6.24
Referer: http://10.28.161.31/index.php
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Cookie: PHPSESSID=978ee1e3b3696743c5c8f507a2ec7212
According to my observation, I did not copied the Header's content properly and that's why it is logging out quickly. So my question is that, how can I copy the complete content of HttpMethod to another HttpMethod? If any one can provide a code snippet or an example/tutorial would be great or If any one can give me a heads up on where I am doing things wrong, that would be appreciable.
My implementation is right here :
private HttpMethod loadHttp302Request(HttpMethod method, HttpClient client,
int status, String urlString) throws HttpException, IOException {
if (status!=302)
return null;
String[] url = urlString.split("/");
HttpMethod theMethod = new GetMethod(urlString + method.getResponseHeader("Location").getValue());
theMethod.setRequestHeader("Cookie", method.getResponseHeader("Set-Cookie")
.getValue());
theMethod.setRequestHeader("Referrer",url[0]+"//"+url[2]);
int _status = client.executeMethod(theMethod);
return theMethod;
}

HttpClient can automatically handle the redirects if you set the strategy. Follow this post on usage example Httpclient 4, error 302. How to redirect?