We are using HttpHead to get the info from our customer's website, but for some reason we are getting cookie in the response as well. Is it expected? Is there a way to set to not return cookie?
The following is the code we have
HttpClient httpclient = new DefaultHttpClient();
// the time it takes to open TCP connection.
httpclient.getParams().setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, this.timeout);
// timeout when server does not send data.
httpclient.getParams().setParameter(CoreConnectionPNames.SO_TIMEOUT, this.timeout);
// the head method
HttpHead httphead = new HttpHead(url);
HttpResponse response = httpclient.execute(httphead);
And we are getting the following warning, indicating that there was cookie returned with response as well.
[WARN] ResponseProcessCookies - Cookie rejected: "[version: 0][name: DXFXFSG][value: AUR][domain: ...omitted...][path: /][expiry: null]". Illegal domain attribute "...omitted...". Domain of origin: "...omitted..."
Yes it is expected; you should get the same response as for the equivalent GET except that there is no body. If the GET would include a cookie, you should see it.
As an aside, I believe the warning you are seeing, from the redacted message you gave, is that the server is trying to set a cookie for a different domain.
Related
I'm trying to read a JSON response from a RESTful webserver running on an IoT module (Advantech WISE-4012). According to the documentation, any GET request should be made in this form
GET /ai_value/slot_0/ch_0
Any Java implementation of GET requests (Java libraries, Apache etc.), anyway, append to the end of the request the protocol signature HTTP/1.1. E.g:
GET http://192.168.0.14/ai_value/slot_0/ch_0 HTTP/1.1
Because of this (probably) i'm getting Error 400 (Bad request) on every client i tried so far. The only working method i've discovered was sending a simple request through the address bar on Google Chrome browser (sometimes i get a response, sometimes a get a bad request error either). How can i write a java implementation of a GET request plain and simple as described by the documentation? How can i test a custom GET request without HTTP/1.1 at the end? Every chrome extension i tried (Advanced REST Client, Postman) add the protocol version at the end, so i haven't had the chance to verify if that's why i'm getting a bad request error.
EDIT:
This is the response header from Advanced REST client
Connection: close
Content-Type: application/json
Server: WISE-4000/8.1.0020
While the source message is the following one:
GET /ai_value/slot_0/ch_0 HTTP/1.1
HOST: 192.168.0.14
The only mismatch between the documentation is the HTTP/1.1 signature as mentioned before. Adding the "accept: application/json" makes no difference either
After a bit of digging into the documentation, it looks like the default timeout (i.e. 720 seconds) is the one causing an issue. There doesn't seem to be any way to work it around (ideally, the system should reset the time after a successful request and we should only get 400 - or 403 ideally after 720 seconds of inactivity).
A couple of points I would like to recommend to the API developers for WISE-4012 (if they are in touch with you):
Add brief documentation for authentication and timeout (probably, more response codes and error messages with each error response)
Enable OAuth for API Access
As far as current implentation is conerned, I guess you need to do a basic auth and pass username/password with every request, Or add Authentication header with every API request to get successful response without any 400s.
Check if this helps.
HttpClient httpClient = HttpClients.createDefault();
URI reqUri = new URI(<uri>);
RequestBuilder requestBuilder = RequestBuilder.create("GET");
requestBuilder.setUri(reqUri);
requestBuilder.setHeader(<headerKey>, <headerValue>);
requestBuilder.setEntity(<entity_data>);
HttpUriRequest httpRequest = requestBuilder.build();
httpResponse = httpClient.execute(httpRequest);
I am using the Apache HttpClient to send requests to our internal API servers. The servers require authentication and need a cookie to be set with an auth token.
Up to HttpClient 4.3.6 this has been working fine, but on 4.4 and above it has stopped sending the cookies on requests. My cookie domain is set to .subdomain.mycompany.com, which works for 4.3.6, but not 4.4 and above. If I'm more specific and give the full host as the cookie domain, i.e. host.subdomain.mycompany.com it works, but this is not a solution.
Here's a code snippet similar to what I'm doing:
public CloseableHttpResponse execute(CloseableHttpClient httpClient) throws IOException {
BasicClientCookie cookie = new BasicClientCookie("cookieName", "myAuthtoken");
cookie.setPath("/");
cookie.setDomain(".subdomain.mycompany.com");
cookie.setSecure(false);
HttpContext localContext = new BasicHttpContext(parentContext);
CookieStore cookieStore = new BasicCookieStore();
cookieStore.addCookie(cookie);
localContext.setAttribute(HttpClientContext.COOKIE_STORE, cookieStore);
return httpClient.execute(target, request, localContext);
}
The httpClient is already constructed and passed into this code which sets the auth cookie.
I saw this, which is similar Cookies getting ignored in Apache httpclient 4.4, but in my case the cookies aren't being sent to the server.
After turning on wire logging in the HttpClient I can see the following in 4.3.6, but not in 4.4 and above:
DEBUG [org.apache.http.client.protocol.RequestAddCookies] Cookie [version: 0][name: cookieName][value: authToken][domain: .subdomain.mycompany.com][path: /][expiry: Wed Jul 15 16:07:05 IST 2015] match [host.subdomain.mycompany.com:80/myApi]
Which leads me to think it's something to do with cookie domain matching. Anyone have any ideas? Thanks.
I have debugged the example code. The problem is at BasicDomainHandler.match(Cookie, CookieOrigin) line: 129 as it expects org.apache.http.cookie.ClientCookie.DOMAIN_ATTR to be set in order to match full host name from URL to cookie domain. So you need to add the following line to your code, after you set the domain:
cookie.setAttribute(ClientCookie.DOMAIN_ATTR, "true");
The change was added with revision 1646864 on 12/19/14, 10:59 PM:
RFC 6265 compliant cookie spec
As suggested by the other answer, setting something like this should resolve:
cookie.setAttribute(ClientCookie.DOMAIN_ATTR, ".subdomain.mycompany.com");
The necessity of setting ClientCookie.DOMAIN_ATTR is is documented in HTTP Components Chapter 3. HTTP state management:
Here is an example of creating a client-side cookie object:
BasicClientCookie cookie = new BasicClientCookie("name", "value");
// Set effective domain and path attributes
cookie.setDomain(".mycompany.com");
cookie.setPath("/");
// Set attributes exactly as sent by the server
cookie.setAttribute(ClientCookie.PATH_ATTR, "/");
cookie.setAttribute(ClientCookie.DOMAIN_ATTR, ".mycompany.com");
I am trying to use httpclient to verify if a specific endpoint is reachable. It seems that it is only possible to check if the server is up but cannot verify if the actual resource is available.
Here is my code:
HttpClient client = new Default HttpClient();
client.execute(new HttpOptions(url)).getStatusLine().getStatusCode();
According to the protocol (http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html) it should have been checking the status of the specific resource but I always get a 200 response as long the the actual host is reachable. What am I doing wrong here?
Thank you.
I'm trying to parse an XML response within an Android app. The technique of parsing itself is not the problem, but the process of receiving the XML makes it difficult to do it the common way.
More in detail:
I request a xhtml website with the apache httpclient (in Android). The website is located on a Java EE Application Server (AS). I give two GET parameters with the request (username, password).
The website is located in a secure area on the AS, so first of all the AS forwards me to the login page. The loginpage takes the username and password (from the GET parameter) and logs me in automatically. If the login credentials are valid I'll get redirected to the requested XHTML page. This is the site I want to parse with the android SAX parser.
But when I try to do this, the only respose I'm able to parse is the login page, not the page. I'm redirected to after successfull login. Can anyone tell me how to instruct the android apache http client to take the response of the redirected page (for later parsing) after the automatical login process?
The logged-in user is stored in the HTTP session which is identified by a cookie with the name JSESSIONID. You need to ensure that you pass the obtained cookie back on every subsequent request, also on the redirects. Otherwise the server will consider the redirected request as unauthorized and redirect you once again back to the login page.
Managing the obtained cookies can be done with help of the CookieStore which you need to set in the HttpContext which you in turn need to pass on every HttpClient#execute() call.
HttpClient httpClient = new DefaultHttpClient();
CookieStore cookieStore = new BasicCookieStore();
HttpContext httpContext = new BasicHttpContext();
httpContext.setAttribute(ClientContext.COOKIE_STORE, cookieStore);
// ...
HttpResponse response1 = httpClient.execute(yourMethod1, httpContext);
// ...
HttpResponse response2 = httpClient.execute(yourMethod2, httpContext);
// ...
Found my solution with the help of this blog entry:
http://ginger-space.blogspot.com/2007/04/httpclient-for-form-based.html
Just updated the Code to the current Apache Client andere IT worked.
I'm trying to use the Apache/Jakarta HttpClient 4.1.1 to connect to an arbitrary web page using the given credentials. To test this, I have a minimal install of IIS 7.5 on my dev machine running where only one authentication mode is active at a time. Basic authentication works fine, but Digest and NTLM return 401 error messages whenever I try to log in. Here is my code:
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpContext localContext = new BasicHttpContext();
HttpGet httpget = new HttpGet("http://localhost/");
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(AuthScope.ANY,
new NTCredentials("user", "password", "", "localhost"));
if (!new File(System.getenv("windir") + "\\krb5.ini").exists()) {
List<String> authtypes = new ArrayList<String>();
authtypes.add(AuthPolicy.NTLM);
authtypes.add(AuthPolicy.DIGEST);
authtypes.add(AuthPolicy.BASIC);
httpclient.getParams().setParameter(AuthPNames.PROXY_AUTH_PREF,
authtypes);
httpclient.getParams().setParameter(AuthPNames.TARGET_AUTH_PREF,
authtypes);
}
localContext.setAttribute(ClientContext.CREDS_PROVIDER, credsProvider);
HttpResponse response = httpclient.execute(httpget, localContext);
System.out.println("Response code: " + response.getStatusLine());
The one thing I've noticed in Fiddler is that the hashes sent by Firefox versus by HttpClient are different, making me think that maybe IIS 7.5 is expecting stronger hashing than HttpClient provides? Any ideas? It'd be great if I could verify that this would work with NTLM. Digest would be nice too, but I can live without that if necessary.
I am not an expert on the subject but during the NTLM authentication using http components I have seen that the client needs 3 attempts in order to connect to an NTML endpoint in my case. It is kinda described here for Spnego but it is a bit different for the NTLM authentication.
For NTLM in the first attempt client will make a request with Target auth state: UNCHALLENGED and Web server returns HTTP 401 status and a header: WWW-Authenticate: NTLM
Client will check for the configured Authentication schemes, NTLM should be configured in client code.
Second attempt, client will make a request with Target auth state: CHALLENGED, and will send an authorization header with a token encoded in base64 format: Authorization: NTLM TlRMTVNTUAABAAAAAYIIogAAAAAoAAAAAAAAACgAAAAFASgKAAAADw==
Server again returns HTTP 401 status but the header: WWW-Authenticate: NTLM now is populated with encoded information.
3rd Attempt Client will use the information from WWW-Authenticate: NTLM header and will make the final request with Target auth state: HANDSHAKE and an authorisation header Authorization: NTLM which contains more information for the server.
In my case I receive an HTTP/1.1 200 OK after that.
In order to avoid all this in every request documentation at chapter 4.7.1 states that the same execution token must be used for logically related requests. For me it did not worked.
My code:
I initialize the client once in a #PostConstruct method of an EJB
PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
cm.setMaxTotal(18);
cm.setDefaultMaxPerRoute(6);
RequestConfig requestConfig = RequestConfig.custom()
.setSocketTimeout(30000)
.setConnectTimeout(30000)
.setTargetPreferredAuthSchemes(Arrays.asList(AuthSchemes.NTLM))
.setProxyPreferredAuthSchemes(Arrays.asList(AuthSchemes.BASIC))
.build();
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY,
new NTCredentials(userName, password, hostName, domainName));
// Finally we instantiate the client. Client is a thread safe object and can be used by several threads at the same time.
// Client can be used for several request. The life span of the client must be equal to the life span of this EJB.
this.httpclient = HttpClients.custom()
.setConnectionManager(cm)
.setDefaultCredentialsProvider(credentialsProvider)
.setDefaultRequestConfig(requestConfig)
.build();
Use the same client instance in every request:
HttpPost httppost = new HttpPost(endPoint.trim());
// HttpClientContext is not thread safe, one per request must be created.
HttpClientContext context = HttpClientContext.create();
response = this.httpclient.execute(httppost, context);
Deallocate the resources and return the connection back to connection manager, at the #PreDestroy method of my EJB:
this.httpclient.close();
I had the same problem with HttpClient4.1.X After upgrading it to
HttpClient 4.2.6 it woked like charm. Below is my code
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpContext localContext = new BasicHttpContext();
HttpGet httpget = new HttpGet("url");
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(AuthScope.ANY,
new NTCredentials("username", "pwd", "", "domain"));
List<String> authtypes = new ArrayList<String>();
authtypes.add(AuthPolicy.NTLM);
httpclient.getParams().setParameter(AuthPNames.TARGET_AUTH_PREF,authtypes);
localContext.setAttribute(ClientContext.CREDS_PROVIDER, credsProvider);
HttpResponse response = httpclient.execute(httpget, localContext);
HttpEntity entity=response.getEntity();
The easiest way troubleshoot such situations I found is Wireshark. It is a very big hammer, but it really will show you everything. Install it, make sure your server is on another machine (does not work with Localhost) and start logging.
Run your request that fails, run one that works. Then, filter by http (just put http in the filter field), find the first GET request, find the other GET request and compare. Identify meaningful difference, you now have specific keywords or issues to search code/net for. If not enough, narrow down to first TCP conversation and look at full request/response. Same with the other one.
I solved an unbelievable number of problems with that approach. And Wireshark is very useful tool to know. Lots of super-advanced functions to make your network debugging easier.
You can also run it on either client or server end. Whatever will show you both requests to allow you to compare.
I had a similar problem with HttpClient 4.1.2. For me, it was resolved by reverting to HttpClient 4.0.3. I could never get NTLM working with 4.1.2 using either the built-in implementation or using JCIFS.
Updating our application to use the jars in the httpcomponents-client-4.5.1 resolved this issue for me.
I finally figured it out. Digest authentication requires that if you use a full URL in the request, the proxy also needs to use the full URL. I did not leave the proxy code in the sample, but it was directed to "localhost", which caused it to fail. Changing this to 127.0.0.1 made it work.