We Keep Coding

Apache httpClient retrying POST request while ignoring timeouts

I'm currently trying to build an OSGi service that sends a POST request to a defined API. This API is used to virus-scan a file which is contained in the request body (JSON) as Base64 string.
For this, I am using Apache HttpClient contained in Adobe AEM uberjar v6.4.0
My current implementation works fine for smaller files (<2 MB), but as filesize gets bigger, the behaviour gets strange:
When I upload a 9 MB file, the request executes for ~1 minute, then gets a HTTP400 as response and afterwards retrys the request 7 times.
I tried to use a timeout with the request. If the timeout is below 60.000ms, a TimeoutException is thrown, if it's greater than 60.000ms, I get a HTTP400 Bad Request. I guess the latter is the APIs fault which I need to clarify.
However, in both cases after the exception is thrown, httpClient retries the request and I have not been able to prevent that since. I'm struggeling with many deprecated "HowTo's" on the web and now I'm here.
I have shortened the code a bit, as it's somehow big (mostly removing debug messages and some "if... return false" at the beginning). My Code:
public boolean isAttachmentClean(InputStream inputStream) throws IOException, JSONException, ServiceUnavailableException {
//prevent httpClient from retrying in case of an IOException
final HttpRequestRetryHandler retryHandler = new DefaultHttpRequestRetryHandler(0, false);
HttpClient httpClient = HttpClients.custom().setRetryHandler(retryHandler).build();
HttpPost httpPost = new HttpPost(serviceUrl);
httpPost.setHeader("accept", "application/json");
//set some more headers...
//set timeout for POST from OSGi Config
RequestConfig timeoutConfig = RequestConfig.custom()
.setConnectionRequestTimeout(serviceTimeout)
.setConnectTimeout(serviceTimeout)
.setSocketTimeout(serviceTimeout)
.build();
httpPost.setConfig(timeoutConfig);
//create request body data
String requestBody;
try {
requestBody = buildDataJson(inputStream);
} finally {
inputStream.close();
}
HttpEntity requestBodyEntity = new ByteArrayEntity(requestBody.getBytes("UTF-8"));
httpPost.setEntity(requestBodyEntity);
//Execute and get the response.
HttpResponse response = httpClient.execute(httpPost);
if (response.getStatusLine().getStatusCode() != HttpServletResponse.SC_OK){
httpPost.abort();
throw new ServiceUnavailableException("API not available, Response Code was "+ response.getStatusLine().getStatusCode());
}
HttpEntity entity = response.getEntity();
boolean result = false;
if (entity != null) {
InputStream apiResult = entity.getContent();
try {
// check the response from the API (Virus yes or no)
result = evaluateResponse(apiResult);
} finally {
apiResult.close();
}
}
return result;
}
"buildDataJson()" simply reads the InputStream and creates a JSON needed for the API call.
"evaluateResponse()" also reads the InputStream, transforms it into a JSON and checks for a property named "Status:" "Clean".
I'd appreciate any tipps on why this request is retried over and over again.
/edit: So far, I found that Apache httpClient has some default mechanism that retries a request in case of an IOException - which is what I get here. Still, I have not found a solution on how to deactivate these retries.

Java Proxy: How to extract Destination Host and Port from the HttpRequest?

I am working on a Http Proxy in java. Basically I have 3 applications:
a client application, where I just submit a request to a server VIA a proxy
a proxy that captures the request, modifies it and then forwards it to the web server
the web server
Here is my code for the Client (taken from the apache httpcore examples, but works well):
public class ClientExecuteProxy () {
public static void main(String[] args)throws Exception {
HttpHost proxy = new HttpHost("127.0.0.1", 8080, "http");
DefaultHttpClient httpclient = new DefaultHttpClient();
try {
httpclient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
HttpHost target = new HttpHost("issues.apache.org", 443, "https");
HttpGet req = new HttpGet("/");
System.out.println("executing request to " + target + " via " + proxy);
HttpResponse rsp = httpclient.execute(target, req);
HttpEntity entity = rsp.getEntity();
System.out.println("----------------------------------------");
System.out.println(rsp.getStatusLine());
Header[] headers = rsp.getAllHeaders();
for (int i = 0; i<headers.length; i++) {
System.out.println(headers[i]);
}
System.out.println("----------------------------------------");
if (entity != null) {
System.out.println(EntityUtils.toString(entity));
}
} finally {
// When HttpClient instance is no longer needed,
// shut down the connection manager to ensure
// immediate deallocation of all system resources
httpclient.getConnectionManager().shutdown();
}
}
}
If I do a direct execution of the request to the server (if I comment the line "httpclient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);"), it works without any problem. But if I leave it like that, it will pass by the proxy. Here is the part that I do not know how to handle for the proxy:
The proxy listens for the requests, reads its content and verifies if it respects certain policies. If OK it will forward it to the server, else it will drop the request and it will send a HttpResponse with an error. The problem is when the request is OK and it needs to be forwarded. How does the proxy know to what address to forward it? My question is: How do I get the information from the request entered at the line "HttpHost target = new HttpHost("issues.apache.org", 443, "https");"?
I've googled for a couple of hours but found nothing. Can anybody help me please?

When you define an HTTP proxy to an application or browser, either:
There will be a preceding CONNECT request to form a tunnel, that tells you the target host:port, or
The entire target URL is placed in the middle of the GET/POST/... request line. Normally, without a proxy, this is just a relative URL, relative to the host:port of the TCP connection.

make https post request

I am making an http post request to one of the xml gateways, but as per their rule I must post https post request, here is my code and i am getting a custom error code which indicates in their manual that the post request must be https, can you help me modifying the following code.
public class PostXML {
public static void main(String args[]) throws FileNotFoundException {
// Get target URL
String strURL = "http://xmlgw.companieshouse.gov.uk/v1-0/xmlgw/Gateway" ;
// Get file to be posted
String strXMLFilename = "F:\\12-8\\CompanyFormation\\CompanyFormation\\web\\file.xml";
File input = new File(strXMLFilename);
// Prepare HTTP post
PostMethod post = new PostMethod(strURL);
// Request content will be retrieved directly
// from the input stream
// Per default, the request content needs to be buffered
// in order to determine its length.
// Request body buffering can be avoided when
// content length is explicitly specified
post.setRequestEntity(new InputStreamRequestEntity(new FileInputStream(input), input.length()));
// Specify content type and encoding
// If content encoding is not explicitly specified
// ISO-8859-1 is assumed
post.setRequestHeader("Content-type", "text/xml; charset=ISO-8859-1");
// Get HTTP client
HttpClient httpclient = new HttpClient();
// Execute request
try {
int result = httpclient.executeMethod(post);
// Display status code
System.out.println("Response status code: " + result);
// Display response
System.out.println("Response body: ");
System.out.println(post.getResponseBodyAsString());
} catch (Exception e) {
e.printStackTrace();
} finally {
// Release current connection to the connection pool
// once you are done
post.releaseConnection();
}
}
}

As you are implementing the HttpClient you will need to add make sure that you can get signed certificates from the server.
see http://hc.apache.org/httpclient-3.x/sslguide.html and maybe the Customizing SSL in HttpClient section.
As a validation check make sure that access is OK via a browser first.

Authenticating to sharepoint with kerberos from a java HttpClient

I have a linux\java6 client that will authenticate to sharepoint2010 with NTLM and then send HTTP REST web services using Apache Commons HttpClient.
I can do this with NTLM , but I want to use the same REST API to access sharepoint 2010 that uses kerberos auth.
Any examples how to authenticate and send REST over HTTP with a kerberos sharepoint?
(preferably using HttpClient)
p.s.
I dont have access to sharepoint code, but i do have access to sharepoint admin configurations.
This is roughly how I authenticate with NTLM:
HttpClient httpClient = new HttpClient(new SimpleHttpConnectionManager(true));
AuthPolicy.registerAuthScheme(AuthPolicy.NTLM, JCIFS_NTLMScheme.class);
String localHostName = Inet4Address.getLocalHost().getHostName();
authscope = new AuthScope(uri.getHost(), AuthScope.ANY_PORT);
httpClient.getState().setCredentials(authscope,new NTCredentials(
getUsername(),getPassword(),localHostName,getDomain()));
// after the initial ntlm auth I can call my REST service with "httpClient.executeMethod"
int status = httpClient.executeMethod(new GetMethod(accessURI + "/sitecollection/info"));

Please confirm that your environment is correctly setup for Kerberos, this can be achieved by running kinit. If this fails you will need to ensure that your krb5.ini (windows) or krb5.conf (linux) are setup to point to your domain controller correctly.
Once you have confirmed that Kerberos is functional you can use the example code from HttpClient as pasted below.
Please note that there are many issues that can cause Kerberos to fail, such as time synchronisation, supported encryption types, trust relationships across domain forests and it's also worth ensuring that your client is on a seperate box to the server.
Here is the example code which is available in the HttpClient download, you will need to ensure your JAAS configuration and krb5.conf or ini are correct!
public class ClientKerberosAuthentication {
public static void main(String[] args) throws Exception {
System.setProperty("java.security.auth.login.config", "login.conf");
System.setProperty("java.security.krb5.conf", "krb5.conf");
System.setProperty("sun.security.krb5.debug", "true");
System.setProperty("javax.security.auth.useSubjectCredsOnly","false");
DefaultHttpClient httpclient = new DefaultHttpClient();
try {
httpclient.getAuthSchemes().register(AuthPolicy.SPNEGO, new SPNegoSchemeFactory());
Credentials use_jaas_creds = new Credentials() {
public String getPassword() {
return null;
}
public Principal getUserPrincipal() {
return null;
}
};
httpclient.getCredentialsProvider().setCredentials(
new AuthScope(null, -1, null),
use_jaas_creds);
HttpUriRequest request = new HttpGet("http://kerberoshost/");
HttpResponse response = httpclient.execute(request);
HttpEntity entity = response.getEntity();
System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
System.out.println("----------------------------------------");
if (entity != null) {
System.out.println(EntityUtils.toString(entity));
}
System.out.println("----------------------------------------");
// This ensures the connection gets released back to the manager
EntityUtils.consume(entity);
} finally {
// When HttpClient instance is no longer needed,
// shut down the connection manager to ensure
// immediate deallocation of all system resources
httpclient.getConnectionManager().shutdown();
}
}
}

Java Httpurlconnection DNS resolution with multiple IP addresses

I'm using Java's HttpUrlConnection to hit foo.com
foo.com has multiple A-Records that point to different IP addresses (1.1.1.1 and 1.1.1.2)
If my first connect call resolves to 1.1.1.1, but then that machine goes down, will a subsequent connect call recognize this and try to connect on 1.1.1.2 instead?
Or do I need to implement this sort of logic myself using the INetAddress api?

I was able to resolve this by using Apache Commons HttpClient, see the code snippet below.
Like I feared, the URLConnection provided by java.net is a very simplistic implementation and will only try the first IP address from the resolved list. If you really are not allowed to use another library, you will have to write your own error handling. It's kinda messy, since you will need to resolve all IPs before hand using InetAddress, and connect to each IP passing the "Host: domain.name" header to the HTTP stack yourself until one of the IPs responds.
The Apache library is greatly more robust and allows for a great deal of customization. You can control how many times it will retry and, most importantly, it will automatically try all IP addresses resolved to the same name until one of them responds successfully.
HttpRequestRetryHandler myRetryHandler = new HttpRequestRetryHandler() {
#Override
public boolean retryRequest(IOException exception, int count, HttpContext context) {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
}
return count < 30;
}
};
ConnectionKeepAliveStrategy keepAlive = new ConnectionKeepAliveStrategy() {
#Override
public long getKeepAliveDuration(HttpResponse response, HttpContext context) {
return 500;
}
};
DefaultHttpClient httpclient = new DefaultHttpClient();
httpclient.getParams().setParameter("http.socket.timeout", new Integer(2000));
httpclient.getParams().setParameter("http.connection.timeout", new Integer(2000));
httpclient.setHttpRequestRetryHandler(myRetryHandler);
httpclient.setKeepAliveStrategy(keepAlive);
HttpGet httpget = new HttpGet("http://remotehost.com");
HttpResponse httpres = httpclient.execute(httpget);
InputStream is = httpres.getEntity().getContent();
I hope this helps!