We Keep Coding

Apache HttpClient not receiving entire response

Update: If I use System.out.println(EntityUtils.toString(response.getEntity())); the output is what appears to be the missing lines of HTML (including the closing body and html tags). However, printing to a file still only gives me the first 2000 odd lines missing the last 1000.
I am using the following code to perform a http post request:
public static String Post(CloseableHttpClient httpClient, String url, Header[] headers,
List<NameValuePair> data, HttpClientContext context) throws IOException
{
HttpPost httpPost = new HttpPost(url);
httpPost.setEntity(new UrlEncodedFormEntity(data));
httpPost.setHeaders(headers);
CloseableHttpResponse response = httpClient.execute(httpPost, context);
if (response.getEntity() == null)
throw new NullPointerException("Unable to get html for: " + url);
// Get the data then close the response object
String responseData = EntityUtils.toString(response.getEntity());
EntityUtils.consume(response.getEntity());
response.close();
return responseData;
}
However I am not receiving the full response entity. I am missing about 1000 lines of html (including the closing body and html tags. I think this is because the data is being sent in chunks although I am not entirely sure.
Here are the response headers:
Cache-Control:max-age=0, no-cache, no-store
Connection:Transfer-Encoding
Connection:keep-alive
Content-Encoding:gzip
Content-Type:text/html; charset=utf-8
Date:Sat, 04 Jul 2015 15:14:58 GMT
Expires:Sat, 04 Jul 2015 15:14:58 GMT
Pragma:no-cache
Server:Microsoft-IIS/7.5
Transfer-Encoding:chunked
Vary:User-Agent
Vary:Accept-Encoding
X-Content-Type-Options:nosniff
X-Frame-Options:SAMEORIGIN
How can I ensure that I receive the full response entity?

To gather all the essence of the comments. There is nothing wrong with your code here - using EntityUtils is the recomended way to deal with all kinds of responses. You have error in code that stores your response to the file.

I've got a similar problem, and solved it by ensuring lcosing the connection like :
} finally {
try {
EntityUtils.consume(entity);
try {
response.getOutputStream().flush();
} catch (IOException e) {
logger.warn("Error while flushing the response output connection. It will ensure to close the connection.", e);
}
if (null != httpResponse) {
httpResponse.close();
}
} catch (IOException ignore) {
}
}
Or event better with try-resources :
try(CloseableHttpResponse response = httpClient.execute(httpPost, context)){
if (response.getEntity() == null){
throw new NullPointerException("Unable to get html for: " + url);
}
String responseData = EntityUtils.toString(response.getEntity());
EntityUtils.consume(response.getEntity());
}

Add Path+ Parameters to Response Fields

I´m making a Web Service, using Java and Glassfish as server.
I´m also using Apache Server for Processing HTTP requests, i.e, when I make a request, I´m able to get the standard informations, like:
HTTP/1.1 200 OK[\r][\n]
Server: GlassFish Server Open Source Edition 4.1 [\r][\n]"
X-Powered-By: Servlet/3.1 JSP/2.3 (GlassFish Server Open Source Edition 4.1 Java/Oracle Corporation/1.8)[\r][\n]
Set-Cookie: JSESSIONID=efc5aa919b55321d3aeaf2c9b3b6; Path=/context; HttpOnly[\r][\n]
Content-Type: text/xml; charset=utf-8[\r][\n]
Date: Thu, 07 May 2015 15:26:40 GMT[\r][\n]
Transfer-Encoding: chunked[\r][\n]
WWW-Authenticate: Basic realm="file"[\r][\n]
Content-Language: [\r][\n]
Content-Type: text/html[\r][\n]
Content-Length: 1090[\r][\n]
SOAPAction: ""[\r][\n]
Host: localhost:8080[\r][\n]
Connection: Keep-Alive[\r][\n]
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)[\r][\n]
One Example of a Web Operation that I developed is:
#WebMethod(operationName = "someoperation")
#Produces(MediaType.APPLICATION_XML)
public void makeHappen(#WebParam(name = "req") Object obj,
#WebParam(name = "resp", mode = WebParam.Mode.OUT) Holder<String> response) {
To List of information that I get, I want to add own specifications, like:
--> OperationName: someOperation
HTTP/1.1 200 OK[\r][\n]
Server: GlassFish Server Open Source Edition 4.1 [\r][\n]"
X-Powered-By: Servlet/3.1 JSP/2.3 (GlassFish Server Open Source Edition 4.1 Java/Oracle Corporation/1.8)[\r][\n]
Set-Cookie: JSESSIONID=efc5aa919b55321d3aeaf2c9b3b6; Path=/context; HttpOnly[\r][\n]
Content-Type: text/xml; charset=utf-8[\r][\n]
Date: Thu, 07 May 2015 15:26:40 GMT[\r][\n]
Transfer-Encoding: chunked[\r][\n]
WWW-Authenticate: Basic realm="file"[\r][\n]
Content-Language: [\r][\n]
Content-Type: text/html[\r][\n]
Content-Length: 1090[\r][\n]
SOAPAction: ""[\r][\n]
Host: localhost:8080[\r][\n]
Connection: Keep-Alive[\r][\n]
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)[\r][\n]

For WebSphere Application Server:
Refer to the documentation and examples in Sending transport headers with JAX-WS
Here is a short programming example that illustrates how request
transport headers are sent by a JAX-WS Web services client
application:
public class MyApplicationClass {
// Inject an instance of the service's port-type.
#WebServiceRef(EchoService.class)
private EchoPortType port;
// This method will invoke the web service operation and send transport headers on the request.
public void invokeService() {
// Set up the Map that will contain the request headers.
Map<String, Object> requestHeaders = new HashMap<String, Object>();
requestHeaders.put(“MyHeader1”, “This is a string value”);
requestHeaders.put(“MyHeader2”, new Integer(33));
requestHeaders.put(“MyHeader3”, new Boolean(true));
// Set the Map as a property on the RequestContext.
BindingProvider bp = (BindingProvider) port;
bp.getRequestContext().put(com.ibm.websphere.webservices.Constants.REQUEST_TRANSPORT_PROPERTIES, requestHeaders);
// Invoke the web services operation.
String result = port.echoString(“Hello, world!”);
}
}
Here is a short programming example that illustrates how response
transport headers are sent by a JAX-WS Web services endpoint
implementation class:
#WebService
public class EchoServiceImpl implements EchoServicePortType {
// Inject an instance of WebServiceContext so we can retrieve
// the MessageContext for each invocation of this endpoint.
#Resource
WebServiceContext ctxt;
/**
* Default constructor.
*/
public EchoServiceImpl() {
....
}
public String echoString(String input) {
String result = “Echo result: “ + input;
// Retrieve the MessageContext from the injected WebServiceContext.
MessageContext mc = ctxt.getMessageContext();
// Send some headers back in the response message.
Map<String, Object> responseHeaders = new HashMap<String, Object>();
responseHeaders.put("MyHeader1", "This is a string response value");
responseHeaders.put("MyHeader2", new Integer(33));
responseHeaders.put("MyHeader3”, new Boolean(false));
// Set the response header Map on the MessageContext.
mc.put(com.ibm.websphere.webservices.Constants.RESPONSE_TRANSPORT_PROPERTIES, responseHeaders);
return result;
}
}
For GlassFish Application Server:
You can get the javax.xml.ws.WebServiceContext and from it javax.xml.ws.handler.MessageContext. Then add to the MessageContext your headers, something like this:
...
Map<String, List<String>> headers = new HashMap<String, List<String>>();
headers.put("OperationName", someOperation);
messageContext.put(MessageContext.HTTP_REQUEST_HEADERS, headers)
...
Also you can try to append the HTTP header to the request by using this approach:
...
Dispatch<SOAPMessage> dispatch =
service.createDispatch(portName, SOAPMessage.class, Service.Mode.MESSAGE);
Map<String, List<String>> headers =
new HashMap<String, List<String>>();
headers.put("OperationName", someOperation);
dispatch.getRequestContext().put(MessageContext.HTTP_REQUEST_HEADERS,
headers);
...
See Also:
How to modify request headers in a J2EE web application
How to add HTTP header to SOAP Webservice Glassfish

Https request returns 403

When I make a request to : " https://www.btcturk.com/api/orderbook " via browser or curl I get the response as expected.
When I make the same request via jersey or java libraries such as HttpsURLConnection, I get a 403 forbidden response.
I can use the same methods to make requests to any other urls running under https. An example method can be found below :
public static RestResponse getRestResponse(String url)
{
String jsonString;
try
{
Client client = Client.create();
WebResource webResource = client.resource(url);
ClientResponse response = webResource.accept("application/json")
.get(ClientResponse.class);
if (response.getStatus() != 200) {
return RestResponse.createUnsuccessfulResponse(response.getStatusInfo().getReasonPhrase());
}
jsonString = response.getEntity(String.class);
}
catch(Exception e)
{
return RestResponse.createUnsuccessfulResponse(e);
}
return RestResponse.createSuccessfulResponse(jsonString);
}
The above code is just to give the idea. The whole thing can be found at: https://github.com/cgunduz/btcenter/tree/master/src/main/java/com/cemgunduz/web
My network knowledge is very limited, so any directions towards where I should start would be helpful. Cheers.

You probably have to provide some credentials. Depending on the server configuration you have to provide either a user/password combination or a valid certificate. Try the solutions provided here:
Using HTTPS with REST in Java

How to invoke Google Drive REST API with Jersey?

I am trying to use Java Jersey instead of Google client libraries to access the Google File API, but I keep getting returned a response status of "401 Unauthorized". Prior to invoking the call, I have obtained an access token from Google, using Oauth:
public static String getGoogleFileResource(final String fileId,
final String accessToken) {
//projection
ClientConfig cc = new DefaultClientConfig();
cc.getProperties().put(ClientConfig.PROPERTY_FOLLOW_REDIRECTS, true);
Client client = Client.create(cc);
String url = String
.format("https://www.googleapis.com/drive/v2/files/%s?fields=downloadUrl&key=%s",
fileId, GoogleClientConstants.GOOGLE_api_key);
WebResource webResource = client.resource(url);
String response = webResource
.accept(MediaType.APPLICATION_JSON_TYPE,
MediaType.APPLICATION_XML_TYPE)
.header("Authorization", "Bearer " + accessToken)
.get(String.class);
logger.info("Authorization - " + "Bearer " + accessToken);
logger.info(" reponse " + response);
return response;
}
What am I doing wrong ?

Turns out you need to add your access token to the http request header.
In the Google developer's guide: "uploading a file"
https://developers.google.com/drive/manage-uploads
You need to send a post request like this:
POST /upload/drive/v2/files?uploadType=media HTTP/1.1
Host: www.googleapis.com
Content-Type: image/jpeg
Content-Length: number_of_bytes_in_JPEG_file
Authorization: your_auth_token
JPEG data
note that in the header you need to put a your_auth_token

HttpClient 4.x how to use cookies?

I'm trying to use the cookies I get in a response to my post method using HttpClient 4.0.3;
Here is my code:
public void generateRequest()
{
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpPost httppost = new HttpPost("http://mysite.com/login");
httpclient.getParams().setParameter("http.useragent", "Custom Browser");
httpclient.getParams().setParameter(CoreProtocolPNames.PROTOCOL_VERSION,
HttpVersion.HTTP_1_1);
httpclient.getParams().setParameter(ClientPNames.COOKIE_POLICY,
CookiePolicy.BROWSER_COMPATIBILITY);
CookieStore cookieStore = new BasicCookieStore();
HttpContext localContext = new BasicHttpContext();
localContext.setAttribute(ClientContext.COOKIE_STORE, cookieStore);
try
{
LOG.info("Status Code: sending");
List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>(2);
nameValuePairs.add(new BasicNameValuePair("email", "john%40gmail.com"));
nameValuePairs.add(new BasicNameValuePair("password", "mypassword"));
httppost.setEntity(new UrlEncodedFormEntity(nameValuePairs));
httppost.setHeader("ContentType", "application/x-www-form-urlencoded");
HttpResponse response = httpclient.execute(httppost, localContext);
HttpEntity entity = response.getEntity();
if (entity != null)
{
entity.consumeContent();
}
iterateCookies(httpclient);
}
catch (ClientProtocolException e)
{
LOG.error("ClientProtocolException", e);
}
catch (IOException e)
{
LOG.error("IOException", e);
}
}
private void iterateCookies(DefaultHttpClient httpclient)
{
List<Cookie> cookies = httpclient.getCookieStore().getCookies();
if (cookies.isEmpty())
{
System.out.println("No cookies");
}
else
{
for (Cookie c : cookies)
{
System.out.println("-" + c.toString());
}
}
}
But I keep getting the No cookies logged out even though when I use web-sniffer.net, I get this response:
Status: HTTP/1.1 302 Found
Cache-Control: private, no-store
Content-Type: text/html; charset=utf-8
Location: http://www.mysite.com/loginok.html
Server: Microsoft-IIS/7.0
X-AspNet-Version: 2.0.50727
Set-Cookie: USER=DDA5FF4E1C30661EC61CFA; domain=.mysite.com; expires=Tue, 08-Jan-2013 18:39:53 GMT; path=/
Set-Cookie: LOGIN=D6CC13A23DCF56AF81CFAF; domain=.mysite.com; path=/ Date: Mon, 09 Jan 2012 18:39:53 GMT
Connection: close
Content-Length: 165
All the examples I've found online that make any sort of sense refer to HttpClient 3.x where you can set the CookiePolicy to IGNORE and handle the Set-Cookie header manually. I can't understand why this is so difficult in 4.x. I need access to the USER hash for a number of reasons. Can anyone please tell me how in the hell I can get access to it?
UPDATE
I have found the following C# code which does the same thing and works correctly.
private static string TryGetCookie(string user, string pass, string baseurl)
{
string body = string.Format("email={0}&password={1}", user, pass);
byte[] bodyData = StringUtils.StringToASCIIBytes(body);
HttpWebRequest req = WebRequest.Create(baseurl) as HttpWebRequest;
if (null != req.Proxy)
{
req.Proxy.Credentials = CredentialCache.DefaultCredentials;
}
req.AllowAutoRedirect = false;
req.Method = "Post";
req.ContentType = "application/x-www-form-urlencoded";
req.ContentLength = bodyData.Length;
using (Stream reqBody = req.GetRequestStream())
{
reqBody.Write(bodyData, 0, bodyData.Length);
reqBody.Close();
}
HttpWebResponse resp1 = req.GetResponse() as HttpWebResponse;
string cookie = resp1.Headers["Set-Cookie"];
if( string.IsNullOrEmpty(cookie))
{
if (0 < resp1.ContentLength)
{
// it's probably not an event day, and the server is returning a singlecharacter
StreamReader stringReader = new StreamReader(resp1.GetResponseStream());
return stringReader.ReadToEnd();
}
return null;
}
return ParseCookie(cookie);
}
I believe my java code is not forming the post request correctly because when I use a URLConnection and print the request header from web-sniffer.net below:
POST /reg/login HTTP/1.1[CRLF]
Host: live-timing.formula1.com[CRLF]
Connection: close[CRLF]
User-Agent: Web-sniffer/1.0.37 (+http://web-sniffer.net/)[CRLF]
Accept-Charset: ISO-8859-1,UTF-8;q=0.7,*;q=0.7[CRLF]
Cache-Control: no-cache[CRLF]
Accept-Language: de,en;q=0.7,en-us;q=0.3[CRLF]
Referer: http://web-sniffer.net/[CRLF]
Content-type: application/x-www-form-urlencoded[CRLF]
Content-length: 53[CRLF]
[CRLF]
email=john%40gmail.com&password=mypassword
I get a response from the server that contains the set-cookies header. Is my java code not generating the request the same as web-sniffer.net?
I have seen a post method generated using this code:
PostMethod authPost = new PostMethod("http://localhost:8000/webTest/j_security_check");
// authPost.setFollowRedirects(false);
NameValuePair[] data = {
new NameValuePair("email", "john%40gmail.com"),
new NameValuePair("password", "mypassword")
};
authPost.setRequestBody(data);
status = client.executeMethod(authPost);
The main difference here being that the NameValuePair data is set in the request body rather than set as the entity. Does this make a difference? Would this produce the correct request header?

Both cookies look suspicious. Both use outdated Netscape cookie draft format. Both have invalid domain attribute value. The LOGIN appears malformed (semicolon is missing after the path attribute) on top of that. So, most likely both cookies got rejected by HttpClient.
You can find out whether this is the case by running HttpClient with the context logging turned on as described here:
http://hc.apache.org/httpcomponents-client-ga/logging.html
One last remark. Generally one should not meddle with cookie policies when using HttpClient 4.x. The default BEST_MATCH policy will automatically delegate processing of cookies to a particular cookie spec implementation based on the composition of the Set-Cookie header value. In order to disable cookie processing entirely one should remove cookie processing protocol interceptors from the protocol processing chain.
Hope this helps.

I believe the problem is that you are mixing two "styles" here: on one hand you create your own BasicCookieStore and put that in your HttpContext; on the other hand, when print the cookies, you loop over the cookie store in the DefaultHttpClient.
So either change the iterateCookies to use your local cookie store, or just use the one provided by the DefaultHttpClient. As you can see in the javadoc of DefaultHttpClient, it should automatically add response cookies to its internal cookie store.

It's always a simple answer! After debugging the C# and another C program I found. I was jumping the gun and doing my own encoding on the email address to remove the # character. This was the problem!!! Nothing else seemed to make a difference whether it was there or not! The code now looks like:
public void postData(final String email, final String password)
{
DefaultHttpClient client = new DefaultHttpClient();
HttpPost post = new HttpPost(LOGIN_URL);
client.getParams().setParameter("http.useragent", "Custom Browser");
client.getParams().setParameter(CoreProtocolPNames.PROTOCOL_VERSION, HttpVersion.HTTP_1_1);
try
{
List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>(2);
nameValuePairs.add(new BasicNameValuePair("email", email));
nameValuePairs.add(new BasicNameValuePair("password", password));
UrlEncodedFormEntity entity = new UrlEncodedFormEntity(nameValuePairs, HTTP.UTF_8);
entity.setContentType("application/x-www-form-urlencoded");
post.setEntity(entity);
printHeaders(client.execute(post));
printCookies(client.getCookieStore());
}
catch (ClientProtocolException e)
{
LOG.error("ClientProtocolException", e);
}
catch (IOException e)
{
LOG.error("IOException", e);
}
}
and the output now looks like:
Response Headers:
-Cache-Control: private, no-store
-Content-Type: text/html; charset=utf-8
-Server: Microsoft-IIS/7.0
-X-AspNet-Version: 2.0.50727
-Set-Cookie: USER=3D907C0EB817FD7...92F79616E6E96026E24; domain=.mysite.com; expires=Thu, 10-Jan-2013 20:22:16 GMT; path=/
-Set-Cookie: LOGIN=7B7028DC2DA82...5CA6FB6CD6C2B1; domain=.mysite.com; path=/
-Date: Wed, 11 Jan 2012 20:22:16 GMT
-Content-Length: 165
-Cookie:: [version: 0][name: USER][value: 3D907C0E...E26E24][domain: .mysite.com][path: /][expiry: Thu Jan 10 20:22:16 GMT 2013]
-Cookie:: [version: 0][name: LOGIN][value: 7B7028D...D6C2B1][domain: .mysite.com][path: /][expiry: null]