Rebuild HTTP flow with incubated Java 10 HttpClient - java

I'm trying to rebuild the session setup to a web server by a Java HttpClient application. I have chosen the incubated HttpClient provided with Java 9 and Java 10.
With Chrome I captured this headers from a single request:
General
Request URL: https://<some_url>?user_id=1176&onlyDirectUserItems=true&onlyAssignedToUser=true&show=Unresolved&itemsFilter=0
Request Method: GET
Status Code: 302 Found
Remote Address: <theProxy>:8000
Referrer Policy: no-referrer-when-downgrade
Response Headers
Connection: Keep-Alive
Content-Length: 164
Content-Type: text/html; charset=iso-8859-1
Date: Fri, 08 Jun 2018 14:33:16 GMT
Keep-Alive: timeout=300, max=100
Location: https://<another_url>:443/nesp/app/plogin?agAppNa=app_me_company_ext&c=secure/name/password/uri&target=%22https://<another-usr>/browseIssues.spr?user_id=1176&onlyDirectUserItems=true&onlyAssignedToUser=true&show=Unresolved&itemsFilter=0%22
P3p: CP="NOI"
Server: Apache
Set-Cookie: IPCZQX03224bfb75=030003000000000000000000000000008f7aed69; path=/; domain=.me.de
Via: 1.1 <host> (Access Gateway-ag-7169149846802036-13837511)
Request Headers
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
Connection: keep-alive
Cookie: org.ditchnet.jsp.tabs.wiki=wiki-wysiwyg; ZNPCQ003-31393000=6c2f99a3; ZNPCQ003-32323200=cd188fdd
DNT: 1
Host: <host>
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
Query String Parameters
user_id: 1176
onlyDirectUserItems: true
onlyAssignedToUser: true
show: Unresolved
itemsFilter: 0
What can be seen the Response Header provides a URL (header-key: "location") which I need to grab and call next. But with my http client I fail with status-code 400 and get almost nothing
This is my code
url = "https://<some_url>?user_id=1176&onlyDirectUserItems=true&onlyAssignedToUser=true&show=Unresolved&itemsFilter=0";
HttpClient client = HttpClient.newBuilder()
.proxy(ProxySelector.of(new InetSocketAddress("<theProxy>", 8000)))
.cookieHandler(new CookieManager(null, CookiePolicy.ACCEPT_ALL))
.followRedirects(HttpClient.Redirect.SAME_PROTOCOL)
.build();
HttpRequest request = HttpRequest.newBuilder()
.header("User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0")
.header("Upgrade-Insecure-Requests", "1")
// .header("Host", "<host>")
.header("Connection", "keep-alive")
.header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8")
.header("Accept-Language", "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7")
.header("Accept-Encoding", "gzip, deflate, br")
.uri(new URI(url))
.build();
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandler.asString());
HttpHeaders headers = response.headers();
Map<String, List<String>> headerMap= headers.map();
for (String key : headerMap.keySet()) {
System.out.println(">"+key+"<");
for (String value : headerMap.get(key)) {
System.out.println(" " + value);
}
}
System.out.println(response.statusCode());
System.out.println(response.body());
I have no clue what might be wrong and how to proceed to get this done. I hope someone can tell me what to ty next.
What I also do not understand: I had to remove the header "Host" - because I got the response: "Your browser sent a request that this server could not understand."
The very same header as can be found in the Chrome-listing

I could get it run with Apache HttpClient. I cannot tell what is wrong with the incubated HttpClient - but there is for sure a reason that it is not a fully integrated part of the Java 9/10 delivery

Related

Cannot use Fileupload Multipart on Weblogic server

I'm having a blocking exception while requesting the servlet parts in Weblogic 12.2.1.4-dev on docker.(please note that this is working on Wildfly server)
The Java code i'm using is:
import javax.servlet.http.*;
protected void doPut(HttpServletRequest request, HttpServletResponse response) throws IOException {
...
Collection<Part> parts = request.getParts();
...
}
the error i'm having in the server.log:
<Oct 2, 2020 9:17:59,302 AM GMT> <Warning> <HTTP> <BEA-101394> <The exception "The request content-type is not a multipart/form-data" occurred when processing getParameter or getParameterValues from a multipart value of a ServletRequest.>
javax.servlet.ServletException: The request content-type is not a multipart/form-data
at weblogic.servlet.utils.fileupload.Multipart.getParts(Multipart.java:158)
at weblogic.servlet.internal.ServletRequestImpl$RequestParameters.getParts(ServletRequestImpl.java:2497)
at weblogic.servlet.internal.ServletRequestImpl$RequestParameters.access$3000(ServletRequestImpl.java:2181)
at weblogic.servlet.internal.ServletRequestImpl.getParts(ServletRequestImpl.java:3652)
at javax.servlet.http.HttpServletRequestWrapper.getParts(HttpServletRequestWrapper.java:375)
And the Http request details on google chrome:
// GENERAL
Request URL: http://172.21.1.1:8310/backoffice/service
Request Method: PUT
Status Code: 400 Bad Request
Remote Address: 172.1.1.1:8310
Referrer Policy: strict-origin-when-cross-origin
// REQUEST HEADERS
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-BE,en;q=0.9,fr-FR;q=0.8,fr;q=0.7,en-US;q=0.6,es;q=0.5,it;q=0.4
Cache-Control: no-cache
Connection: keep-alive
Content-Length: 647482
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryoCBB96YNI9S3vXob
Cookie: JSESSIONID=XIjomkiRAVxEtEn0qwlILe46arjsphNNibL00t2dHEhj75oc167A!-481127499
Host: 172.1.1.1:8310
Origin: http://172.1.1.1:8310
Pragma: no-cache
Referer: http://172.1.1.1:8310/backoffice/products/edit?id=MA01
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36
X-Requested-With: XMLHttpRequest
// FORM DATA
subside-velo-pedelec25.pdf: (binary)
actionId: 98c6e900-0289-4bb6-8e98-2b967b2ee363
windowId: YbVZr0XiCFHxU7K4hUlmBJmLoXwzY6
I faced the same issue.
FileUpload's servlet implementation from WebLogic core libs only allow multipart requests under "POST" method:
private boolean isMultipart() {
if (!this.request.getMethod().toLowerCase().equals("post"))
return false;
String contentType = this.request.getContentType();
if (contentType == null)
return false;
if (contentType.toLowerCase().startsWith("multipart/form-data"))
return true;
return false;
}
If this validation fails, it will throw an exception including a message that does not have any relation with the actual error:
if (!isMultipart())
throw new ServletException("The request content-type is not a multipart/form-data");
So I switched my method from PUT to POST.

jsoup request returns wrong status code

I have a Twitter shortened URL (t.co) and I'm trying to use jsoup to send a request and parse its response. There should be three redirect hops before reaching the final URL. This is not the case when using jsoup, even after setting followRedirects to true.
My code:
public static void main(String[] args) {
try {
Response response = Jsoup.connect("https://t. co/sLMy6zi4Yw").followRedirects(true).execute(); // Space intentional to avoid SOF shortened errors
System.out.println(response.statusCode()); // prints 200
} catch (IOException e) {
System.out.println(e.getMessage());
}
}
However, using Python's Request library, I can get the right response:
response = requests.get('https://t. co/sLMy6zi4Yw', allow_redirects=False)
print(response.status_code)
301
I'm using jsoup version 1.11.2 and Requests version 2.18.4 with Python 3.5.2.
Anybody have any insight on the matter?
To overcome this special case you can remove the User-Agent header which Jsoup sets by default (for some unknown/undocument reason)
Connection connection = Jsoup.connect(url).followRedirects(true);
connection.request().removeHeader("User-Agent");
Let's examine the raw requests & view the server behavior
Request with user agent (to simulate a browser) returns
status code 200
Meta refresh which is a method of instructing a web browser to automatically refresh the current web page or frame after a given time interval, this case 0 seconds and url http://bit. ly/2n3VDpo
Javascript code which replaces location to the same url (google "meta refresh is depercated" / "drawbacks using meta refresh")
Curl example
curl --include --raw "https://t. co/sLMy6zi4Yw" --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
Response
Chrome/63.0.3239.132 Safari/537.36"
HTTP/1.1 200 OK
cache-control: private,max-age=300
content-length: 257
content-security-policy: referrer always;
content-type: text/html; charset=utf-8
referrer-policy: unsafe-url
server: tsa_b
strict-transport-security: max-age=0
vary: Origin
x-response-time: 20
x-xss-protection: 1; mode=block; report=https://twitter.com/i/xss_report
<head><meta name="referrer" content="always"><noscript><META http-equiv="refresh" content="0;URL=http://bit. ly/2n3VDpo"></noscript><title>http://bit. ly/2n3VDpo</title></head><script>window.opener = null;location.replace("http:\/\/bit. ly\/2n3VDpo")</script>
Request without user agent returns
status code 301
header "location" with the redirect url
Curl example
curl --include --raw "https://t. co/sLMy6zi4Yw"
HTTP/1.1 301 Moved Permanently
cache-control: private,max-age=300
content-length: 0
location: http://bit. ly/2n3VDpo
server: tsa_b
strict-transport-security: max-age=0
vary: Origin
x-response-time: 9

Getting 500 error from HttpClient, works in browser

I'm using Apache HttpClient to try to submit some post data to a server. Unfortunately, I don't have access to the server to get any log information so that won't be possible.
If I go through this process with Firefox, it works fine. (I do get a 302 warning on this particular page)
I have matched the Request headers of both Firefox and my program.
Firefox Request Headers:
Host: server ip
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://server ip/
Content-Type: application/x-www-form-urlencoded
Content-Length: 407
Cookie: sessionId=blahblah
Connection: keep-alive
Upgrade-Insecure-Requests: 1
My Programs Request Headers shown from context.getRequest().getAllHeaders();
Host: server ip
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:55.0) Gecko/20100101 Firefox/55.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://server ip/
Content-Type: application/x-www-form-urlencoded
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Content-Length: 406
Cookie: sessionId=blahblah
I have matched the body of the request by comparing the output of EntityUtils.toString(httpPost.getEntity(), "UTF-8"); and the built in tool for Firefox's tool to look at the request body, and they match almost character for character. (Just a slight difference in the session id which is expected as it's not using the same session.)
I'm not sure what else to check. What could be causing the server to behave differently between the Browser and the program?
Below is my code for the POST request.
HttpPost httpPost = new HttpPost("https://" + getIp() + "");
List<NameValuePair> params = new ArrayList<NameValuePair>();
params.add(new BasicNameValuePair("FTPUsername", "blah"));
params.add(new BasicNameValuePair("FTPPassword", "blah"));
params.add(new BasicNameValuePair("FormButtonSubmit", "OK"));
httpPost.setEntity(new UrlEncodedFormEntity(params));
httpPost.setHeader("Host", ip);
httpPost.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:55.0) Gecko/20100101 Firefox/55.0");
httpPost.setHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
httpPost.setHeader("Accept-Language", "en-US,en;q=0.5");
httpPost.setHeader("Accept-Encoding", "gzip, deflate, br");
httpPost.setHeader("Referer", referer);
httpPost.setHeader("Content-Type", "application/x-www-form-urlencoded");
httpPost.setHeader("Connection", "keep-alive");
httpPost.setHeader("Upgrade-Insecure-Requests", "1");
//Response
HttpResponse response = getHttpClient().execute(httpPost, LoginRequest.context);
int statusCode = response.getStatusLine().getStatusCode();
httpPost.releaseConnection();
I realize this could probably be many things since 500 is a server error, but it's got to be something I'm submitting wrong or I'm missing something as it works perfectly in the browser.
302 "warning" is actually a redirect. HTTP Client does do redirect automatically, you must flag the RedirectStrategy, For HttpClient 4.3:
HttpClient instance = HttpClientBuilder.create()
.setRedirectStrategy(new LaxRedirectStrategy()).build();
see examples in answer and w3 docs:
If the 302 status code is received in response to a request other than
GET or HEAD, the user agent MUST NOT automatically redirect the
request unless it can be confirmed by the user
Do you work with Windows machine? or Linux machine?
If you use a windows machine, have you tried working with WAMP server for Linux use LAMP server, so if you install it, you won't get those errors, that's how I fixed my error. Once if you install these two servers, change the port number in skype by logging into Skype and change the port number or uninstall your skype. It should work.

How to retrieve an NTLM Challenge

I am currently trying to Authenticate with an IIS/6.0 Data Server. With the code below, how do I retrieve the challenge from the server. Currently what I am doing is sending the first GET request to the server
//Part 1: The Request
pw.println("GET /dashboard/ HTTP/1.1");
pw.println("Host: MyServer.net");
pw.println("User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0");
pw.println("Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
pw.println("Accept-Language: en-US,en;q=0.5");
pw.println("Accept-Encoding: gzip, deflate");
pw.println("Connection: keep-alive");
pw.println("WWW-Authenticate: Negotiate");
pw.println();
pw.flush();
//Part 1: The Response
HTTP/1.1 401 Unauthorized
Content-Length: 1656
Content-Type: text/html
Server: Microsoft-IIS/6.0
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
X-Powered-By: ASP.NET
Date: Mon, 14 Sep 2015 19:28:16 GMT
Then I send the next request
//Part 2: The Request
pw.println("GET /dashboard/ HTTP/1.1");
pw.println("Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
pw.println("Referer: http://MyServer.net/dashboard/");
pw.println("Accept-Language: en-US,en;q=0.5");
pw.println("User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0");
pw.println("Accept-Encoding: gzip, deflate");
pw.println("Host: MyServer.net");
pw.println("Connection: keep-alive");
pw.println("Authorization: Negotiate");
pw.println();
pw.flush();
//Part 2: The Response
HTTP/1.1 401 Unauthorized
Content-Length: 1539
Content-Type: text/html
Server: Microsoft-IIS/6.0
WWW-Authenticate: Negotiate YF0GBisGAQUFAqBTMFGgMDAuBgkqhkiC9xIBAgIGCSqGSIb3EgECAgYKKoZIhvcSAQICAwYKKwYBBAGCNwICCqMdMBugGRsXcGFlbXMxOTYkQFNUQVJCVUNLUy5ORVQ=
X-Powered-By: ASP.NET
Date: Mon, 14 Sep 2015 19:28:16 GMT
There are two things wrong that I think I have done here.
The WWW-Authenticate Header field appears to be wrong in "Part 2: The Response" I think it is because I am not using NTLM (Which is what I want to use)
I have not sent My Active Directory credentials yet. I do not know what I need to do next.
Currently I found a really helpful document Responding to the Challenge which helps explain how to encode the Active Directory credentials
What steps do I need to take in order to completely authenticate with the server so that I can poll data from it?

Copy Contents of HttpMethod to another HttpMethod

I have been tryig to handle a redirect(302) in java code and now that I am done doing it by my code. I ran into an other problem in which after login, on click on any link I get logged out. So I checked my TCP Stream through wireshark and found that there are few HeaderRequests missing. After implementation of my code, Http Header are as follows :
GET /index.php/ HTTP/1.1
Host: 10.28.161.31
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.24) Gecko/20111109 CentOS/3.6-3.el5.centos Firefox/3.6.24
Cookie: PHPSESSID=d488eea5e85afc8ec526c1a749e7ab20; path=/
Referrer: http://10.28.161.31
Cookie: $Version=0; PHPSESSID=d488eea5e85afc8ec526c1a749e7ab20; $Path=/ ???
and original Http Headers are as follows :
GET / HTTP/1.1
Host: 10.28.161.31
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.24) Gecko/20111109 CentOS/3.6-3.el5.centos Firefox/3.6.24
Referer: http://10.28.161.31/index.php
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Cookie: PHPSESSID=978ee1e3b3696743c5c8f507a2ec7212
According to my observation, I did not copied the Header's content properly and that's why it is logging out quickly. So my question is that, how can I copy the complete content of HttpMethod to another HttpMethod? If any one can provide a code snippet or an example/tutorial would be great or If any one can give me a heads up on where I am doing things wrong, that would be appreciable.
My implementation is right here :
private HttpMethod loadHttp302Request(HttpMethod method, HttpClient client,
int status, String urlString) throws HttpException, IOException {
if (status!=302)
return null;
String[] url = urlString.split("/");
HttpMethod theMethod = new GetMethod(urlString + method.getResponseHeader("Location").getValue());
theMethod.setRequestHeader("Cookie", method.getResponseHeader("Set-Cookie")
.getValue());
theMethod.setRequestHeader("Referrer",url[0]+"//"+url[2]);
int _status = client.executeMethod(theMethod);
return theMethod;
}
HttpClient can automatically handle the redirects if you set the strategy. Follow this post on usage example Httpclient 4, error 302. How to redirect?

Categories