We Keep Coding

HttpClientBuilder not working

I have a code in my java application that connects to a url with a post request and get response.
The code worked for a long time, and it still does for most people. I have one person that the code works for him most of the times, but sometimes I am getting timeout exception.
The code:
HttpClient client = HttpClientBuilder.create().build();
HttpPost post = new HttpPost("http://example.com/link-to-api");
post.addHeader( "Data1", "my_data" );
HttpResponse response = null;
BufferedReader rd = null;
try {
List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>(1);
nameValuePairs.add(new BasicNameValuePair("post_key1","post_data1"));
post.setEntity(new UrlEncodedFormEntity(nameValuePairs));
response = client.execute(post);
} catch (IOException e) {
System.out.println("error: "+ e);
}
The exception that I get is:
org.apache.http.conn.HttpHostConnectException: Connect to example.com:80 [example.com/__IP__] failed: Connection timed out: connect
I sniffed the network, and I can see the packets when the connection is ok, but every few minutes, when I get the above exception, I don't even see the attempt connection in the sniffer.
I did ping and traceroute to the server while the timeout is occuring, and both are ok (I get response from the server)
So it seems to me that the problem is in the java application, not even attempting the connection.
I am using httpclient 4.4.1, if it is of any importance.
Why I am getting these timeouts?

I solved it . At least in my case, all I did was reset the router on my computer, and the problem disappeared.
I have read on the internet many suggestions to change server's tcp/ip parameters. So if this happens to you, before you do anything, I suggest that you check your hardware as well.

Connection Reset error in HttpClient

This is scenario I am testing. The steps are as follows
Create an instance of HttpClient(org.apache.commons.httpclient.HttpClient)3.1
Use the instance to get response from a url (http://www.example.com/submitData)
Bring down the server hosting the url mentioned in step 2.
Bring up the url
Wait till the url is up
Use the instance of HttpClient created in step 1 to get response from the url (http://www.example.com/getData)
I get an error - Connection Reset.
Can anyone help me understand.
I have used the following params in the HttpClient
httpClient.getParams().setParameter("http.socket.timeout", new Integer(0));
httpClient.getParams().setParameter("http.connection.stalecheck", new Boolean(true));
Edited after comments from Peter
This is how I create the HttpClient
HttpClient httpClient = new HttpClient();
AuthScope authScope = new AuthScope("www.example.com", 80, AuthScope.ANY_REALM);
Credentials defaultcreds = new UsernamePasswordCredentials("username", "pwd");
httpClient.getState().setCredentials(authScope, defaultcreds);
httpClient.getParams().setParameter("http.socket.timeout", new Integer(0));
httpClient.getParams().setParameter("http.connection.stalecheck", new Boolean(true));
To establish connection, I use the following
GetMethod getMethod = new GetMethod("http://"+httpClient.getHostConfiguration().getHost()+"/getData");
int statusCode = httpClient.executeMethod(getMethod);
I want the httpclient instance to preserve the hostname and port and credentials.
This is exact execption message that I get - "Connection reset".
More info - I am executing the test from eclipse. If I run the test case in debug mode, I dont get this exception.

I think the behaviour is reasonable because when the server goes down the client socket on the server goes away too. The client still holds old socket but this socket is gone on the server. You should reconnect from client.
AFAIK the stale parameter allows the client to close the connection the clean way (without Exception) but not prevent from disconnection.

This looks suspicious
httpClient.getParams().setParameter("http.socket.timeout", new Integer(0));
Having a timeout set to 0ms would likely terminate the request prematurely?

Java simple code: java.net.SocketException: Unexpected end of file from server

I wrote some simple code in Java, the method should connect to the website and return the BufferedReader.
private BufferedReader getConnection(String url_a) {
URL url;
try {
System.out.println("getting connection");
url = new URL(url_a);
HttpURLConnection urlConnection = (HttpURLConnection)
url.openConnection();
urlConnection.addRequestProperty("User-Agent",
"Mozilla/5.0 (X11; U; Linux i586; en-US; rv:1.7.3) Gecko/20040924"
+ "Epiphany/1.4.4 (Ubuntu)");
inStream = new InputStreamReader(urlConnection.getInputStream());
return new BufferedReader(inStream);
} catch (Exception ex) {
Logger.getLogger(Reader.class.getName()).log(Level.SEVERE, null, ex);
}
return null;
}
When I use it on my PC, it works fine but when I put .jar file on the server I get this error:
java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:718)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:579)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:715)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:579)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1322)
at dataconverter.Reader.getConnection(Reader.java.260)
Problem is quite strange because the exception isn't thrown each time, sometimes everything is OK and program works fine.
Has anybody got any ideas?

"Unexpected end of file" implies that the remote server accepted and closed the connection without sending a response. It's possible that the remote system is too busy to handle the request, or that there's a network bug that randomly drops connections.
It's also possible there is a bug in the server: something in the request causes an internal error, and the server simply closes the connection instead of sending a HTTP error response like it should. Several people suggest this is caused by missing headers or invalid header values in the request.
With the information available it's impossible to say what's going wrong. If you have access to the servers in question you can use packet sniffing tools to find what exactly is sent and received, and look at logs to of the server process to see if there are any error messages.

Summary
This exception is encountered when you are expecting a response, but the socket has been abruptly closed.
Detailed Explanation
Java's HTTPClient, found here, throws a SocketException with message "Unexpected end of file from server" in a very specific circumstance.
After making a request, HTTPClient gets an InputStream tied to the socket associated with the request. It then polls that InputStream repeatedly until it either:
Finds the string "HTTP/1."
The end of the InputStream is reached before 8 characters are read
Finds a string other than "HTTP/1."
In case of number 2, HTTPClient will throw this SocketException if any of the following are true:
The HTTP method is CONNECT
The HTTP method is POST and the client is set to streaming mode
Why would this happen
This indicates that the TCP socket has been closed before the server was able to send a response. This could happen for any number of reasons, but some possibilities are:
Network connection was lost
The server decided to close the connection
Something in between the client and the server (nginx, router, etc) terminated the request
Note: When Nginx reloads its config, it forcefully closes any in-flight HTTP Keep-Alive connections (even POSTs), causing this exact error.

I do get this error when I do not set the Authentication header or I set wrong credentials.

I would suggest using wire shark to trace packets. If you are using Ubuntu, sudo-apt get wireshark. Like Joni stated the only way to figure out whats going wrong is to follow the GET requests and their associated responses.
http://www.wireshark.org/download.html

In my case it was solved just passing proxy to connection. Thanks to #Andreas Panagiotidis.
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("<YOUR.HOST>", 80)));
HttpsURLConnection con = (HttpsURLConnection) url.openConnection(proxy);

In my case url contained wrong chars like spaces . Overall log your url and in some cases use browser.

Most likely the headers you are setting is incorrect or not acceptable.
Example:
connnection.setRequestProperty("content-type", "application/json");

I got this exception too. MY error code is below
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod(requestMethod);
connection.setRequestProperty("Content-type", "JSON");
I found "Content-type" should not be "JSON",is wrong!
I solved this exception by update this line to below
connection.setRequestProperty("Content-type", "application/json");
you can check up your "Content-type"

Is it possible for a site having URL starting with "http://" using HTTPS protocol

I have a site whose URL starts with "http://" but that is giving me an exception with the message - Unsupported protocol: https. Is it possible that the site is using HTTPS protocol still its URL is starting with "http://" and not "https://".
public ActionForward executeAction(ActionMapping mapping, ActionForm form,
HttpServletRequest request, HttpServletResponse response)
throws Exception {
ActionForward forward = mapping.findForward(Constants.SUCCESS);
String link = "http://abc.fgh.jkl.mno";
URL thisURL;
HttpURLConnection conn = null;
try {
thisURL = new URL(link);
conn = (HttpURLConnection) thisURL.openConnection();
System.out.println(conn.getResponseCode());
System.out.println(conn.getResponseMessage());
} catch (Exception ex) {
ex.printStackTrace();
}
return forward;
}
Stack trace
java.net.ProtocolException: Unsupported protocol: https'
at weblogic.net.http.HttpClient.openServer(HttpClient.java:342)
at weblogic.net.http.HttpClient.New(HttpClient.java:238)
at weblogic.net.http.HttpURLConnection.connect(HttpURLConnection.java:172)
at weblogic.net.http.HttpURLConnection.followRedirect(HttpURLConnection.java:643)
at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:422)
at weblogic.net.http.SOAPHttpURLConnection.getInputStream(SOAPHttpURLConnection.java:36)
at weblogic.net.http.HttpURLConnection.getResponseCode(HttpURLConnection.java:947)
at com.cingular.cscape.da.struts.action.thisAction.executeAction(thisAction.java:56)
at com.cingular.cscape.da.struts.action.BaseAction.execute(BaseAction.java:300)
at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:226)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:397)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
at org.extremecomponents.table.filter.AbstractExportFilter.doFilter(AbstractExportFilter.java:53)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3496)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(Unknown Source)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

Did you notice the unmatched apostrophe in the exception message?
java.net.ProtocolException: Unsupported protocol: https'
^
Update: looks like this apostrophe is just a quirk of how the WebLogic HttpClient prints this exception.
Asker's root problem was identified in a chat session: he's accessing an http:// URL that redirects him to an https:// URL. The webserver at that https address is serving up a certificate that his JRE/HttpClient does not trust.
The actual exception should be a SSLKeyException. I think the WebLogic HttpClient is misreporting this problem as an unsupported protocol issue. I think the core issue is:
<Warning> <Security> <BEA-090477> javax.net.ssl.SSLKeyException: [Security:090477] Certificate chain received from www.X.com - nnn.nnn.nnn.nnn was not trusted causing SSL handshake failure.
This is the message that the Asker sees when accessing the https:// URL directly (instead of through a redirect chain).
By default Java's Http[s]URLConnection follows redirects automatically and quietly. If you are curious about where you are being redirected to, try this code:
connection.setInstanceFollowRedirects(false);
String location = connection.getHeaderField("Location");
System.out.println("Redirected to: " + location);
Note that the URL you get redirected to may also redirect you somewhere else and on and on, up to http.maxRedirects times. Redirects may "chain" in this way. If they do chain, you will need to keep following the redirects until you reach a URL that does not issue a redirect. That is where the URL connection eventually finalizes when setInstanceFollowRedirects(true).
Also, I found some code in sun.net.www.protocol.http.HttpURLConnection that seems to indicate that HttpURLConnection may not support switching protocols (HTTP -> HTTPS) as part of its automatic redirect following logic:
private boolean followRedirect() throws IOException {
// ... [snip] ...
if (!url.getProtocol().equalsIgnoreCase(locUrl.getProtocol())) {
return false;
// ... [snip] ...
WebLogic has its own (different) implementation of HttpURLConnection, but it may contain similar logic to prevent protocol switching. So, even if the Asker resolves his certificate trust issues, he still may not be able to use HttpURLConnection to automatically follow a redirect chain that goes from HTTP to HTTPS. A workaround would be to use setInstanceFollowRedirects(false) and follow redirects manually. Or access the HTTPS site directly.

HTTP is a protocol that runs on TCP/IP. HTTPS is HTTP on secure socket (SSL or newer TLS). Now, what is a URL? It's a string that identify a resource, and looks roughly like:
scheme://host:port/path/to/resource
Note that sometimes we don't specify the port number, because some protcols have an associated well-known port number. For HTTP is 80, and for HTTPS is 443, so these numbers are implied. However, note that you can bind your server socket on whatever port you want to: if you tell your HTTP server program to bind its socket to the port 8273 on localhost, it will happily do so.
Said that, in your case the message says "unsupported protocol: https". We can just guess here, however I think the string you pass to the URL is not the one you think.

Another possibility is the HTTP address responding with a 3xx redirect response to an HTTPS URL, which your client tries to follow, but does not support.
You can very by capturing the network traffic with tcpdump or wireshark.
There are so many Java HTTP client libraries that do support HTTPS, why not switch to one of those?

Why does HttpClient throw a SocketTimeOutException when executing POST

I have code similar to the following:
try {
HttpPost post = new HttpPost(httpsUrl);
setHeaders(post);
HttpEntity entity = new StringEntity(request, "UTF-8");
post.setEntity(entity);
HttpResponse response = httpclient.execute(post);
String result = EntityReader.readContent(response.getEntity());
checkAnswer(result);
return result;
} catch (Exception e) {
throw new ZapException("Error executing the http post request: "+e.getMessage(), e);
}
It sends the content of request to a server via POST using a httpclient instance that might have already been used before (it has persistent connections turned on, since we're sending quite some requests to the same server...).
This sometimes fails with a SocketTimeoutException with "Read timed out" as the message.
It's not clear to us, why it only fails at some times, when most times it doesn't. What gives?

In the following, I assume you are using Apache Commons HttpClient (org.apache.commons.httpclient.HttpClient).
Maybe you get thrown a SocketTimeoutException simply because, occasionally, the host your HttpClient instance is communicating with takes too long to respond, triggering HttpClient's cancellation routine.
You can increase the connection timeout and the socket timeout with the following
HttpConnectionParams params = httpclient.getHttpConnectionManager().getParams();
params.setConnectionTimeout(20000);
params.setSoTimeout(15000);
Aditionally, if you still face timeouts despite increasing the timeout limits, it is a good practice to handle the SocketTimeoutException gracefully - for example by retrying the connection a second and third time.