We Keep Coding

How to fix HTTP 504 Gateway timeout exception

I am attempting a GET request to another API to get a json response.
When I make the request with the code below, I am getting HTTP 504 Error (Gateway timeout error).
However, when I tried it through rest client tool, the request does not throw any error.
How do I increase the time gap in my code to avoid the timeout error?
This is how the call looks like:
HttpClient httpClient = getBasicAuthDefaultHttpClient();
String url= "http://XXXXX";
HttpGet httpGet = new HttpGet(url);
httpGet.addHeader("id", id);
httpGet.addHeader("secret", secret);
httpGet.addHeader("network_val", networkval);
HttpResponse response = httpClient.execute(httpGet);
HttpEntity entity = response.getEntity();
if (entity != null && response.getStatusLine().getStatusCode() == HttpStatus.OK.value()) {
ObjectMapper objectMapper = new ObjectMapper();
restInfo = objectMapper.readValue(entity.getContent(), MyClass.class);
} else {
logger.error("Call to API failed: response code = {}", response.getStatusLine().getStatusCode());
}
Please note:
Could this be something to do with 'https'?
When I try with 'http' through my Insomnia REST Client I get the ERROR: error: Failure when receiving data from the peer.
https works fine without any error (https://XXXXX)

This is what I tried.
public HttpClient getBasicAuthDefaultHttpClient() {
CredentialsProvider provider = new BasicCredentialsProvider();
UsernamePasswordCredentials creds = new UsernamePasswordCredentials(user,
password);
provider.setCredentials(AuthScope.ANY, creds);
//Fix to avoid HTTP 504 ERROR (GATEWAY TIME OUT ERROR)
RequestConfig.Builder requestBuilder = RequestConfig.custom();
requestBuilder.setConnectTimeout(30 * 1000);
requestBuilder.setConnectionRequestTimeout(30 * 1000);
HttpClientBuilder builder = HttpClientBuilder.create();
builder.setDefaultRequestConfig(requestBuilder.build());
builder.setDefaultCredentialsProvider(provider).build();
return builder.build();
}

HttpClient and Connection Timeout

I send http requests over Apache HttpClient and my code is here:
HttpHost proxy = new HttpHost("78.1.1.222", 80);
DefaultProxyRoutePlanner routePlanner = new DefaultProxyRoutePlanner(proxy);
httpClient = HttpClients.custom()
.setRoutePlanner(routePlanner)
.build();
HttpGet httpGet = new HttpGet(url);
httpGet.addHeader("Authorization","Basic " + encoding);
httpGet.addHeader("Cache-Control", "no-cache");
httpResponse = httpClient.execute(httpGet);
responseCode=httpResponse.getStatusLine().getStatusCode();
........
........(code continue..)
My question is that how can I add connection timeout time to this code?
Note that I must use proxy with that and I use HttpClient 4.4 .

http://www.baeldung.com/httpclient-timeout explains various ways to set the connection timeout.

post data using Sharepoint 2010 REST API using Java http client

I am trying to post a SharePoint list item to a SharePoint 2010 list using Java client.
I am able to read the SP list items using below code but I am not sure how to write to the SharePoint.
For the post operation, I am getting http 401 error.
Any guidance will be greatly appreciated.
GET Operation:
String url = "https://organization.company.com/sites/Ateam/_vti_bin/ListData.svc/TableLoadInfo"; DefaultHttpClient client = new DefaultHttpClient();
NTCredentials credentials = new NTCredentials(username, password, hostname, domain);
AuthScope scope = new AuthScope(hostname, 443);
SSLContext sslcontext = SSLContext.getInstance("TLS");
sslcontext.init(null, new TrustManager[] { trustmanager }, null);
SSLSocketFactory sf = new SSLSocketFactory(sslcontext, SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
Scheme https = new Scheme("https", 443, sf);
client.getConnectionManager().getSchemeRegistry().register(https);
client.getCredentialsProvider().setCredentials(scope, credentials);
HttpGet request = new HttpGet(url);
//request.addHeader("Accept", "application/xml");
request.addHeader("Accept", "application/json;odata=verbose");
HttpResponse response = client.execute(request);
POST Operation:
String url = "https://organization.company.com/sites/Ateam/_vti_bin/ListData.svc/TableLoadInfo";
DefaultHttpClient client = new DefaultHttpClient();
NTCredentials credentials = new NTCredentials(username, password, hostname, domain);
AuthScope scope = new AuthScope(hostname, 443);
SSLContext sslcontext = SSLContext.getInstance("TLS");
sslcontext.init(null, new TrustManager[] { trustmanager }, null);
SSLSocketFactory sf = new SSLSocketFactory(sslcontext, SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
Scheme https = new Scheme("https", 443, sf);
client.getConnectionManager().getSchemeRegistry().register(https);
client.getCredentialsProvider().setCredentials(scope, credentials);
// POST a data
HttpPost request = new HttpPost(url);
request.addHeader("Accept", "application/json;odata=verbose");
request.addHeader("Content-type", "application/json;odata=verbose");
// request.addHeader("X-RequestDigest", FormDigestValue);
request.addHeader("X-HTTP-Method", "POST");
request.addHeader("If-Match", "*");
JSONStringer json = (JSONStringer) new JSONStringer().object().key("Table_Name").value("TableName 1")
.key("Load_Frequency").value("Weekly").key("Cycle").value("CURRENT").endObject();
StringEntity se = new StringEntity(json.toString());
request.setEntity(se);
HttpResponse response = client.execute(request);

When you post data to SharePoint you have to add "FormDigestValue" inside the header.
Please Note you have commented the line "request.addHeader("X-RequestDigest", FormDigestValue);".
That is the reason you get 401 Error.
First you have to get the "FormDigestValue" value form your current site.
Then add that value inside the header.
Following Code shows the way to get the FormDigestValue.
String digestqueryURL = serverurl + subsite + "/" + "_api/contextinfo";
HttpPost httpPost = new HttpPost(digestquery);
httpPost.addHeader("Accept", "application/json;odata=verbose");
httpPost.addHeader("X-ClientService-ClientTag", "SDK-JAVA");
HttpResponse response = httpClient.execute(httpPost);
byte[] content = EntityUtils.toByteArray(response.getEntity());
String jsonString = new String(content, "UTF-8");
JSONObject json = new JSONObject(jsonString);
String FormDigestValue = json.getJSONObject("d")
.getJSONObject("GetContextWebInformation")
.getString("FormDigestValue");
Then add the FormDigestValue to the header when you do the post operations.
request.addHeader("X-RequestDigest", FormDigestValue);

Apache Http Client 4.0.1 SSL Proxy

I am using Apache Http client 4.0.1 for communicating with the server. I already have a secure/non secure client code that works just fine.
Recently the new addition being to add proxy to this code, so i added the following piece of code to do that (currently non secure proxy),
HttpHost proxy = new HttpHost("localhost", 5555);
httpClient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
This has worked fine with a non secure request. However i am having trouble with a secure (https) request with the same code.
Get the below exception (it tries a few time before failing),
Mar 12, 2014 11:14:27 AM org.apache.http.impl.client.DefaultRequestDirector tryConnect
INFO: I/O exception (org.apache.http.NoHttpResponseException) caught when connecting to the target host: The target server failed to respond
Mar 12, 2014 11:14:27 AM org.apache.http.impl.client.DefaultRequestDirector tryConnect
INFO: Retrying connect
org.apache.http.NoHttpResponseException: The target server failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:95)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at org.apache.http.impl.client.DefaultRequestDirector.createTunnelToTarget(DefaultRequestDirector.java:899)
at org.apache.http.impl.client.DefaultRequestDirector.establishRoute(DefaultRequestDirector.java:818)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:644)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at com.poc.test.SSLTest.main(SSLTest.java:88)
Tried following things,
For https requests, i added both "http" as well as "https" to the schema registry, using the same SSLFactory as the one used for "https".
Changed the proxy to,
HttpHost proxy = new HttpHost("localhost", 5555, "https");
However in both cases it failed with,
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:397)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:572)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at com.poc.test.SSLTest.main(SSLTest.java:89)
Note - I am running a non secure proxy on my localhost via tcpmon.
EDIT: Here is the code i am using for the SSL with proxy communication,
DefaultHttpClient httpClient = new DefaultHttpClient();
try {
SSLContext ctx = SSLContext.getInstance("TLSv1.1");
TrustManager[] trustManagers = getTrustManagers("jks", new FileInputStream(new File("C:\\SSLKeyStore.ks")), "changeit");
ctx.init(null, trustManagers, new SecureRandom());
HttpGet httpget = new HttpGet("https://localhost:8844/Channels/HTTP/getData");
System.out.println("executing request" + httpget.getRequestLine());
SSLSocketFactory factory = new SSLSocketFactory(ctx);
factory.setHostnameVerifier(SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
ClientConnectionManager manager = httpClient.getConnectionManager();
manager.getSchemeRegistry().register(new Scheme("https", 443, factory));
manager.getSchemeRegistry().register(new Scheme("http", 80, PlainSocketFactory.getSocketFactory()));
HttpHost proxy = new HttpHost("localhost", 5555, "http");
httpClient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
HttpResponse response = httpClient.execute(httpget);
HttpEntity entity = response.getEntity();
System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
if (entity != null) {
System.out.println("Response content length: " + entity.getContentLength());
}
EntityUtils.consume(entity);
} catch (Exception exception) {
exception.printStackTrace();
} finally {
httpClient.getConnectionManager().shutdown();
}
Any ideas of what is happening, what am i missing with respect to https and proxy.
Latest EDIT - Even tried their example code (ClientExecuteProxy.java) as is, that too failed with proxy. Is this functionality broken?
HttpHost proxy = new HttpHost("127.0.0.1", 8080, "http");
DefaultHttpClient httpclient = new DefaultHttpClient();
try {
httpclient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
HttpHost target = new HttpHost("issues.apache.org", 443, "https");
HttpGet req = new HttpGet("/");
System.out.println("executing request to " + target + " via " + proxy);
HttpResponse rsp = httpclient.execute(target, req);
HttpEntity entity = rsp.getEntity();
System.out.println("----------------------------------------");
System.out.println(rsp.getStatusLine());
Header[] headers = rsp.getAllHeaders();
for (int i = 0; i<headers.length; i++) {
System.out.println(headers[i]);
}
System.out.println("----------------------------------------");
if (entity != null) {
System.out.println(EntityUtils.toString(entity));
}
} finally {
// When HttpClient instance is no longer needed,
// shut down the connection manager to ensure
// immediate deallocation of all system resources
httpclient.getConnectionManager().shutdown();
}
}
Thanks,
Vicky

Off the top of my head I'd say that you are dealing with a certificate trust issue.
Without seeing how you are setting up the connection specifically I can say with no certainty though.
The "peer not authenticated" means that somewhere along the line the verification of the certificates as presented by one or more of the servers cannot be verified.
EDIT
Since the proxy is under your control you have a load of flexibility at this time.
Please see this SO article, it may suit your needs.

NTLM authentication java via HttpClient

My problem is i'm trying to get into scopus using a crawler but it requires my crawler to enter the site through my school proxy server. I tried authenticating but it keep responding with 401 status.
public void testConnection() throws ClientProtocolException, IOException {
DefaultHttpClient httpclient = new DefaultHttpClient();
List<String> authpref = new ArrayList<String>();
authpref.add(AuthPolicy.NTLM);
httpclient.getParams().setParameter(AuthPNames.TARGET_AUTH_PREF, authpref);
NTCredentials creds = new NTCredentials("username","password","ezlibproxy1.ntu.edu.sg","ntu.edu.sg");//this is correct
httpclient.getCredentialsProvider().setCredentials(AuthScope.ANY, creds);
HttpHost target = new HttpHost("ezlibproxy1.ntu.edu.sg", 443, "https");//this is correct
// Make sure the same context is used to execute logically related requests
HttpContext localContext = new BasicHttpContext();
// Execute a cheap method first. This will trigger NTLM authentication
HttpGet httpget = new HttpGet("http://www-scopus-com.ezlibproxy1.ntu.edu.sg/authid/detail.url?authorId=14831850700");
HttpResponse response = httpclient.execute(target, httpget, localContext);
HttpEntity entity = response.getEntity();
System.out.println(EntityUtils.toString(entity));
int statusCode = response.getStatusLine().getStatusCode();
System.out.println("Status Code:" + statusCode);
}
The status code respond is 401 (unauthorised).
Any suggestion on this?