I am debugging one problem of connection reset and need some help.
Here is the background
Using java version 8, apache httpClient 4.5.2
I have a following program, which runs successfully on windows 10, 7 but end up with connection reset on Azure windows server 2016 VM.
import java.io.IOException;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import org.apache.commons.codec.binary.Base64;
import org.apache.http.Header;
import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import org.apache.http.util.EntityUtils;
public class TestConnectionReset
{
static PoolingHttpClientConnectionManager connManager = new PoolingHttpClientConnectionManager();
static {
connManager.setMaxTotal(10);
connManager.setDefaultMaxPerRoute(2);
}
public static void main(String[] args) throws ClientProtocolException, IOException, InterruptedException {
while (true) {
HttpClientBuilder clientBuilder = HttpClientBuilder.create();
RequestConfig config = RequestConfig.custom().setConnectTimeout(1800000).setConnectionRequestTimeout(1800000)
.setSocketTimeout(1800000).build();
clientBuilder.setDefaultRequestConfig(config);
clientBuilder.setConnectionManager(connManager);
String userName = "xxxxx";
String password = "xxxxx";
String userNamePasswordPair = String.valueOf(userName) + ":" + password;
String authenticationData = "Basic " + new String((new Base64()).encode(userNamePasswordPair.getBytes()));
HttpPost post = new HttpPost("https://url/rest/oauth/token");
Map<String, String> requestBodyMap = new HashMap<String, String>();
requestBodyMap.put("grant_type", "client_credentials");
String req = getFormUrlEncodedBodyFromMap(requestBodyMap);
StringEntity stringEntity = new StringEntity(req);
post.setEntity(stringEntity);
post.setHeader("Authorization", authenticationData);
post.setHeader("Content-Type", "application/x-www-form-urlencoded");
CloseableHttpClient closeableHttpClient = clientBuilder.build();
HttpResponse response = closeableHttpClient.execute(post);
Header[] hs = response.getAllHeaders();
for (Header header : hs) {
System.out.println(header.toString());
}
System.out.println(EntityUtils.toString(response.getEntity()));
Thread.sleep(10*60*1000L);
}
}
public static String getFormUrlEncodedBodyFromMap(Map<String, String> formData) {
StringBuilder requestBody = new StringBuilder();
Iterator<Map.Entry<String, String>> itrFormData = formData.entrySet().iterator();
while (itrFormData.hasNext()) {
Map.Entry<?, ?> entry = (Map.Entry)itrFormData.next();
requestBody.append(entry.getKey()).append("=").append(entry.getValue());
if (itrFormData.hasNext()) {
requestBody.append("&");
}
}
return requestBody.toString();
}
}
I am using pooling httpclient connection manager. 1st request in 1st time loop execution succeeded but subsequent iteration of for loop with next request fails.
My findings
If we see underlying socket connection on windows 10, after 1st request socket goes into CLOSE_WAIT state and next request executes with closing the existing connection and creating new connection.
Actually server closes the connection in duration of 5 minutes. But windows 10 able to detect it and re-initiate the connection when next request is triggered.
Now, on windows server 2016, I can see that netstat shows socket ESTABLISHED state. Means connection is ready to use and in that, it picks up the same connection and finally server has already closed it so results into connection reset error.
I suspect its an environmental issue, where server 2016 is keeping socket ESTABLISHED even after server has terminated it, but on windows 10 socket status changed to CLOSE_WAIT.
Help on this is much appreciated
Finally got the root cause,
Its issue with microsoft azure. They are using SNAT and closing outbound TCP connections after 4 minute idle time. This wasted my 5 days to figureout.
Means if you are connected with server with keep-alive and hope that you can reuse the connection till server time out and sends FIN. But before that if idle period reaches to 4 minutes, azure kills it. BOOM!!. Worst part is, it is not even notifying server or client with RST, means violating TCP and questioning its reliability.
clientBuilder.setKeepAliveStrategy(new ConnectionKeepAliveStrategy() {
#Override
public long getKeepAliveDuration(HttpResponse response, HttpContext context) {
// TODO Auto-generated method stub
return 3*60*1000;
}
});
Using above code, I managed to close connection on 3 minute expiry and close it before azure kills it.
Related
I have a WSDL file which I've turned into Java code by using WSDL2Java inside SoapUI, it works fine, but now I need to add my company's proxy to it, so every SOAP http request would go through it (but not other http requests).
I've looked through multiple threads concerning the same issue and found these options:
system wide proxy by adding
System.getProperties().put("proxySet", "true");
System.getProperties().put("https.proxyHost", "10.10.10.10");
System.getProperties().put("https.proxyPort", "8080");
which doesn't work for me, since it affect the whole jvm.
adding the following code
HelloService hello = new HelloService();
HelloPortType helloPort = cliente.getHelloPort();
org.apache.cxf.endpoint.Client client = ClientProxy.getClient(helloPort);
HTTPConduit http = (HTTPConduit) client.getConduit();
http.getClient().setProxyServer("proxy");
http.getClient().setProxyServerPort(8080);
http.getProxyAuthorization().setUserName("user proxy");
http.getProxyAuthorization().setPassword("password proxy");
which I don't get how to use. My generated code doesn't have any traces of org.apache.cxf, only javax.xml.ws.
Adding this to my port configuration:
((BindingProvider) port).getRequestContext().put("http.proxyHost", "proxy#example.com");
((BindingProvider) port).getRequestContext().put("http.proxyPort", "80");
Here I use a random non-existing proxy and expect to get an error of any sort(timeout, invalid proxy, etc.), but instead it goes through without any errors.
Here is an example without using 3rd party libraries.
https://github.com/schuch/jaxws-proxy-example/blob/master/jaxws-client-with-proxy/src/main/java/ch/schu/example/helloworld/Client.java
package ch.schu.example.helloworld;
import java.net.ProxySelector;
import ch.schu.example.hello.HelloImpl;
import ch.schu.example.hello.HelloImplService;
public class Client {
public static void main(String[] args) {
ProxySelector.setDefault(new MyProxySelector());
HelloImplService service = new HelloImplService();
HelloImpl hello = service.getHelloImplPort();
System.out.println(hello.sayHello("Howard Wollowitz"));
}
}
https://github.com/schuch/jaxws-proxy-example/blob/master/jaxws-client-with-proxy/src/main/java/ch/schu/example/helloworld/MyProxySelector.java
package ch.schu.example.helloworld;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.*;
import java.util.*;
public class MyProxySelector extends ProxySelector {
#Override
public List<Proxy> select(URI uri)
{
System.out.println("select for " + uri.toString());
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("localhost", 9999));
ArrayList<Proxy> list = new ArrayList<Proxy>();
list.add(proxy);
return list;
}
#Override
public void connectFailed(URI uri, SocketAddress sa, IOException ioe) {
System.err.println("Connection to " + uri + " failed.");
}
}
I'm using OpenJDK 11 on Linux and I need to make sure all my web requests done with HttpURLConnection are properly closed and do not keep any file descriptors open.
Oracle's manual tells to use close on the InputStream and Android's manual tells to use disconnect on the HttpURLConnection object.
I also set Connection: close and http.keepAlive to false to avoid pooling of connections.
This seems to work with plain http requests but not encrypted https requests whose response is sent with non-chunked encoding. Only a GC seems to clean up the closed connections.
This example code:
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.stream.Stream;
public class Test {
private static int printFds() throws IOException {
int cnt = 0;
try (Stream<Path> paths = Files.list(new File("/proc/self/fd").toPath())) {
for (Path path : (Iterable<Path>)paths::iterator) {
System.out.println(path);
++cnt;
}
}
System.out.println();
return cnt;
}
public static void main(String[] args) throws IOException, InterruptedException {
System.setProperty("http.keepAlive", "false");
for (int i = 0; i < 10; i++) {
// Must be a https endpoint returning non-chunked response
HttpURLConnection conn = (HttpURLConnection) new URL("https://www.google.com/").openConnection();
conn.setRequestProperty("Connection", "close");
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
while (in.readLine() != null) {
}
in.close();
conn.disconnect();
conn = null;
in = null;
}
Thread.sleep(1000);
int numBeforeGc = printFds();
System.gc();
Thread.sleep(1000);
int numAfterGc = printFds();
System.out.println(numBeforeGc == numAfterGc ? "No socket leaks" : "Sockets were leaked");
}
}
prints this output:
/proc/self/fd/0
/proc/self/fd/1
/proc/self/fd/2
/proc/self/fd/3
/proc/self/fd/4
/proc/self/fd/5
/proc/self/fd/9
/proc/self/fd/6
/proc/self/fd/7
/proc/self/fd/8
/proc/self/fd/10
/proc/self/fd/11
/proc/self/fd/12
/proc/self/fd/13
/proc/self/fd/14
/proc/self/fd/15
/proc/self/fd/16
/proc/self/fd/17
/proc/self/fd/18
/proc/self/fd/19
/proc/self/fd/0
/proc/self/fd/1
/proc/self/fd/2
/proc/self/fd/3
/proc/self/fd/4
/proc/self/fd/5
/proc/self/fd/9
/proc/self/fd/6
/proc/self/fd/7
/proc/self/fd/8
Sockets were leaked
Changing to a http URL makes the sockets close correctly as expected without GC:
/proc/self/fd/0
/proc/self/fd/1
/proc/self/fd/2
/proc/self/fd/3
/proc/self/fd/4
/proc/self/fd/5
/proc/self/fd/6
/proc/self/fd/0
/proc/self/fd/1
/proc/self/fd/2
/proc/self/fd/3
/proc/self/fd/4
/proc/self/fd/5
/proc/self/fd/6
No socket leak
Tested with both OpenJDK 11 and 12. Did I miss something or is this a bug?
Turns out to be a bug after all: https://bugs.openjdk.java.net/browse/JDK-8216326
shutdownInput is now replaced by close in the latest builds of JDK 11 and 13 (but not 12).
I need to set source IP address (for something like IP Spoofing) before sending out an http request. Class used for setting up http connection is HTTPURLConnection. I found below link on stackoverflow which is really useful.
Registering and using a custom java.net.URL protocol
As in the post, I have already created 3 classes extending URLConnection , URLStreamHandler and implementing URLStreamHandlerFactory. This looks to be working fine; however I am getting exception which I think is because I have not implemented getInputStream for URLConnection as was mentioned in above post.
I have couple of questions
1> I am extending custom URLConnection class from HTTPURLConnection, so what's the need of implementing getInputStream as anyway it's a virtual method
2> If I have to do it, can someone provide sample implementation of this method?
JMeter already provides the IP Spoofing feature.
In Http Request Defaults, select (in version 3.0 of JMeter) advanced tab :
See http://jmeter.apache.org/usermanual/component_reference.html#HTTP_Request_parms1:
Source address field
[Only for HTTP Request with HTTPClient implementation]
This property is used to enable IP Spoofing. It overrides the default local IP address for this sample. The JMeter host must have multiple IP addresses (i.e. IP aliases, network interfaces, devices). The value can be a host name, IP address, or a network interface device such as "eth0" or "lo" or "wlan0".
If the property httpclient.localaddress is defined, that is used for all HttpClient requests.
Based on UBIK LOAD PACK's answer. Here is the same code using Aapche http client 3.x. (The variable for setting source IP is args[1])
import org.apache.commons.httpclient.*;
import org.apache.commons.httpclient.methods.*;
import org.apache.commons.httpclient.protocol.Protocol;
public class HC3Test {
public static void main(String[] args) throws Exception {
String url = args[0];
java.net.URL uri = new java.net.URL(url);
HostConfiguration hc = new HostConfiguration();
hc.setHost(uri.getHost(), uri.getPort(), Protocol.getProtocol(uri.getProtocol()));
hc.setLocalAddress(java.net.InetAddress.getByName(args[1]));//for pseudo 'ip spoofing'
HttpClient client = new HttpClient(new SimpleHttpConnectionManager());
client.setHostConfiguration(hc);
GetMethod method = new GetMethod(url);
client.executeMethod(method);
method.releaseConnection();
}
}
Sample code for http client 4.x:
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.conn.params.ConnRoutePNames;
public class HC4Test {
public static void main(String[] args) throws Exception {
DefaultHttpClient httpClient = new DefaultHttpClient();
org.apache.http.params.HttpParams params = httpClient.getParams();
params.setParameter(ConnRoutePNames.LOCAL_ADDRESS,
java.net.InetAddress.getByName(args[1]));//for pseudo 'ip spoofing'
httpClient.execute(new HttpGet(args[0]));
}
}
To accomplish to have a different source address via an http request you could use a local proxy.
import java.io.IOException;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.MalformedURLException;
import java.net.Proxy;
import java.net.URL;
import java.net.URLConnection;
import java.net.UnknownHostException;
public class Huhu {
public static void main(String[] args) throws IOException {
URL url = new URL("http://google.com");
Proxy proxy = new Proxy(Proxy.Type.DIRECT,
new InetSocketAddress(
InetAddress.getByAddress(
new byte[]{your, ip, interface, here}), yourTcpPortHere));
URLConnection conn = url.openConnection(proxy);
}
}
You do not have to override anything this way.
How can I find out what the actual timeout values are for HttpClient 4.3? I know how to set them explicitly but I'd like to know which values are used implicitly if I don't overwrite them.
Example
import org.apache.http.client.config.RequestConfig;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.DefaultHttpRequestRetryHandler;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.params.HttpParams;
public class HttpClientDefaults
{
public static void main(String[] args)
{
RequestConfig config = RequestConfig.custom().build();
System.out.println("ConnectionRequestTimeout: " + config.getConnectionRequestTimeout());
System.out.println("ConnectTimeout: " + config.getConnectTimeout());
System.out.println("SocketTimeout: " + config.getSocketTimeout());
CloseableHttpClient client = HttpClientBuilder.create().setRetryHandler(new DefaultHttpRequestRetryHandler()).build();
HttpParams params = client.getParams();
System.out.println(params.getParameter("http.socket.timeout"));
}
}
Output
ConnectionRequestTimeout: -1
ConnectTimeout: -1
SocketTimeout: -1
Exception in thread "main" java.lang.UnsupportedOperationException
client.getParams() throws an java.lang.UnsupportedOperationException exception. Does somebody know a way to read out the actual values that are going to be used?
The default value for socket, connect, and connection request timeouts is 0 (no timeout)
First to say I'm n00b in Java. I can understand most concepts but in my situation I want somebody to help me. I'm using JBoss Netty to handle simple http request and using MemCachedClient check existence of client ip in memcached.
import org.jboss.netty.channel.ChannelHandler;
import static org.jboss.netty.handler.codec.http.HttpHeaders.*;
import static org.jboss.netty.handler.codec.http.HttpHeaders.Names.*;
import static org.jboss.netty.handler.codec.http.HttpResponseStatus.*;
import static org.jboss.netty.handler.codec.http.HttpVersion.*;
import com.danga.MemCached.*;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;
import org.jboss.netty.buffer.ChannelBuffer;
import org.jboss.netty.buffer.ChannelBuffers;
import org.jboss.netty.channel.ChannelFuture;
import org.jboss.netty.channel.ChannelFutureListener;
import org.jboss.netty.channel.ChannelHandlerContext;
import org.jboss.netty.channel.ExceptionEvent;
import org.jboss.netty.channel.MessageEvent;
import org.jboss.netty.channel.SimpleChannelUpstreamHandler;
import org.jboss.netty.handler.codec.http.Cookie;
import org.jboss.netty.handler.codec.http.CookieDecoder;
import org.jboss.netty.handler.codec.http.CookieEncoder;
import org.jboss.netty.handler.codec.http.DefaultHttpResponse;
import org.jboss.netty.handler.codec.http.HttpChunk;
import org.jboss.netty.handler.codec.http.HttpChunkTrailer;
import org.jboss.netty.handler.codec.http.HttpRequest;
import org.jboss.netty.handler.codec.http.HttpResponse;
import org.jboss.netty.handler.codec.http.HttpResponseStatus;
import org.jboss.netty.handler.codec.http.QueryStringDecoder;
import org.jboss.netty.util.CharsetUtil;
/**
* #author The Netty Project
* #author Andy Taylor (andy.taylor#jboss.org)
* #author Trustin Lee
*
* #version $Rev: 2368 $, $Date: 2010-10-18 17:19:03 +0900 (Mon, 18 Oct 2010) $
*/
#SuppressWarnings({"ALL"})
public class HttpRequestHandler extends SimpleChannelUpstreamHandler {
private HttpRequest request;
private boolean readingChunks;
/** Buffer that stores the response content */
private final StringBuilder buf = new StringBuilder();
protected MemCachedClient mcc = new MemCachedClient();
private static SockIOPool poolInstance = null;
static {
// server list and weights
String[] servers =
{
"lcalhost:11211"
};
//Integer[] weights = { 3, 3, 2 };
Integer[] weights = {1};
// grab an instance of our connection pool
SockIOPool pool = SockIOPool.getInstance();
// set the servers and the weights
pool.setServers(servers);
pool.setWeights(weights);
// set some basic pool settings
// 5 initial, 5 min, and 250 max conns
// and set the max idle time for a conn
// to 6 hours
pool.setInitConn(5);
pool.setMinConn(5);
pool.setMaxConn(250);
pool.setMaxIdle(21600000); //1000 * 60 * 60 * 6
// set the sleep for the maint thread
// it will wake up every x seconds and
// maintain the pool size
pool.setMaintSleep(30);
// set some TCP settings
// disable nagle
// set the read timeout to 3 secs
// and don't set a connect timeout
pool.setNagle(false);
pool.setSocketTO(3000);
pool.setSocketConnectTO(0);
// initialize the connection pool
pool.initialize();
// lets set some compression on for the client
// compress anything larger than 64k
//mcc.setCompressEnable(true);
//mcc.setCompressThreshold(64 * 1024);
}
#Override
public void messageReceived(ChannelHandlerContext ctx, MessageEvent e) throws Exception {
HttpRequest request = this.request = (HttpRequest) e.getMessage();
if(mcc.get(request.getHeader("X-Real-Ip")) != null)
{
HttpResponse response = new DefaultHttpResponse(HTTP_1_1, OK);
response.setHeader("X-Accel-Redirect", request.getUri());
ctx.getChannel().write(response).addListener(ChannelFutureListener.CLOSE);
}
else {
sendError(ctx, NOT_FOUND);
}
}
private void writeResponse(MessageEvent e) {
// Decide whether to close the connection or not.
boolean keepAlive = isKeepAlive(request);
// Build the response object.
HttpResponse response = new DefaultHttpResponse(HTTP_1_1, OK);
response.setContent(ChannelBuffers.copiedBuffer(buf.toString(), CharsetUtil.UTF_8));
response.setHeader(CONTENT_TYPE, "text/plain; charset=UTF-8");
if (keepAlive) {
// Add 'Content-Length' header only for a keep-alive connection.
response.setHeader(CONTENT_LENGTH, response.getContent().readableBytes());
}
// Encode the cookie.
String cookieString = request.getHeader(COOKIE);
if (cookieString != null) {
CookieDecoder cookieDecoder = new CookieDecoder();
Set<Cookie> cookies = cookieDecoder.decode(cookieString);
if(!cookies.isEmpty()) {
// Reset the cookies if necessary.
CookieEncoder cookieEncoder = new CookieEncoder(true);
for (Cookie cookie : cookies) {
cookieEncoder.addCookie(cookie);
}
response.addHeader(SET_COOKIE, cookieEncoder.encode());
}
}
// Write the response.
ChannelFuture future = e.getChannel().write(response);
// Close the non-keep-alive connection after the write operation is done.
if (!keepAlive) {
future.addListener(ChannelFutureListener.CLOSE);
}
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, ExceptionEvent e)
throws Exception {
e.getCause().printStackTrace();
e.getChannel().close();
}
private void sendError(ChannelHandlerContext ctx, HttpResponseStatus status) {
HttpResponse response = new DefaultHttpResponse(HTTP_1_1, status);
response.setHeader(CONTENT_TYPE, "text/plain; charset=UTF-8");
response.setContent(ChannelBuffers.copiedBuffer(
"Failure: " + status.toString() + "\r\n",
CharsetUtil.UTF_8));
// Close the connection as soon as the error message is sent.
ctx.getChannel().write(response).addListener(ChannelFutureListener.CLOSE);
}
}
When I try to send request like http://127.0.0.1:8090/1/2/3
I'm getting
java.lang.NoClassDefFoundError: com/danga/MemCached/MemCachedClient
at httpClientValidator.server.HttpRequestHandler.<clinit>(HttpRequestHandler.java:66)
I believe it's not related to classpath. May be it's related to context in which mcc doesn't exist.
Any help appreciated
EDIT:
Original code http://docs.jboss.org/netty/3.2/xref/org/jboss/netty/example/http/snoop/package-summary.html
I've modified some parts to fit my needs.
Why do you think this is not classpath related? That's the kind of error you get when the jar you need is not available. How do you start your app?
EDIT
Sorry - i loaded and tried the java_memcached-release_2.5.2 bundle in eclipse and found no issue so far. Debugging the class loading revealed nothing unusual. I can't help besides some more hints to double check:
make sure your download is correct. download and unpack again. (are the com.schooner.* classes available?)
make sure you use > java 1.5
make sure your classpath is correct and complete. The example you have shown does not include netty. Where is it.
I'm not familiar with interactions stemming from adding a classpath to the manifest. Maybe revert to plain style, add all jars needed (memcached, netty, yours) to the classpath and reference the main class to start, not a startable jar file