I have developed a standalone Javase client which performs an EJB Lookup to a remote server and executes its method.The Server application is in EJB 3.0
Under some strange magical but rare situations my program hangs indefinetly, on looking inside the issue it seems that while looking up the ejb on the server, I never get the response from the server and it also never times out.
I would like to know if there is a property or any other way through which we can setup the lookup time in client or at the server side.
There is a very nice article that discusses ORB configuration best practices at DeveloperWorks here. I'm quoting the three different settings that can be configured at client (you, while doing a lookup and executing a method at a remote server);
Connect timeout: Before the client ORB can even send a request to a server, it needs to establish an IIOP connection (or re-use an
existing one). Under normal circumstances, the IIOP and underlying TCP
connect operations should complete very fast. However, contention on
the network or another unforeseen factor could slow this down. The
default connect timeout is indefinite, but the ORB custom property
com.ibm.CORBA.ConnectTimeout (in seconds) can be used to change the
timeout.
Locate request timeout: Once a connection has been established and a client sends an RMI request to the server, then LocateRequestTimeout
can be used to limit the time for the CORBA LocateRequest (a CORBA
“ping”) for the object. As a result, the LocateRequestTimeout should
be less than or equal to the RequestTimeout because it is a much
shorter operation in terms of data sent back and forth. Like the
RequestTimeout, the LocateRequestTimeout defaults to 180 seconds.
Request timeout: Once the client ORB has an established TCP connection to the server, it will send the request across. However, it
will not wait indefinitely for a response, by default it will wait for
180 seconds. This is the ORB request timeout interval. This can
typically be lowered, but it should be in line with the expected
application response times from the server.
You can try the following code, which performs task & then waits at most the time specified.
Future<Object> future = executorService.submit(new Callable<Object>() {
public Object call() {
return lookup(JNDI_URL);
}
});
try {
Object result = future.get(20L, TimeUnit.SECONDS); //- Waiting for at most 20 sec
} catch (ExecutionException ex) {
logger.log(LogLevel.ERROR,ex.getMessage());
return;
}
Also, the task can be cancelled by future.cancel(true).
Remote JNDI uses the ORB, so the only option available is com.ibm.CORBA.RequestTimeout, but that will have an affect on all remote calls. As described in the 7.0 InfoCenter, the default value is 180 (3 minutes).
Related
I have used apache httpclient 4.5 in production for a while now, but recently, with the addition of a new use case, the system started failing.
We have multiple services that communicate through REST webservices, the client is a wrapper around apache httpclient 4.5.
Say i have service A communicating with service B. The communication works correctly until I restart service B. The next call I initiate from service A to service B fails, due to time out. After doing some research I found that the underlying TCP connection is reused for performance reasons (no more 2 way handshake etc). Since the server has been restarted, the underlying TCP connection is stale.
After reading the documentation, I found out that I can expire my connection after n seconds. Say I restart service B, then the call will fail the first n seconds, but after that the connection is rebuild. This is the keepAliveStrategy I implemented
connManager = new PoolingHttpClientConnectionManager();
connManager.setMaxTotal(100);
connManager.setDefaultMaxPerRoute(10);
ConnectionKeepAliveStrategy keepAliveStrategy = new DefaultConnectionKeepAliveStrategy() {
public long getKeepAliveDuration(HttpResponse response, HttpContext context) {
long keepAliveDuration = super.getKeepAliveDuration(response, context);
if (keepAliveDuration == -1) {
keepAliveDuration = 45 * 1000; // 45 seconds
}
return keepAliveDuration;
}
};
CloseableHttpClient closeableHttpClient = HttpClients.custom()
.setConnectionManager(connManager)
.setKeepAliveStrategy(keepAliveStrategy)
.build();
I am just wondering if this is correct usage of this library. I this the way it is meant to work or am I making everything overly complex?
Not sure it's 100% the same scenario, but here's my 2 cents:
We had a similar issues (broken connections in pool after a period of inactivity). When we were using an older version of HttpClient (3.X), we used the http.connection.stalecheck manager parameter, taking a minor performance hit over the possibility to get a IOException when a connection has been used that was closed server-side.
After upgrading to 4.4+ this approach was deprecated and started using setValidateAfterInactivity, which is a middle ground between per-call validation and runtime-error scenario:
PoolingHttpClientConnectionManager poolingConnManager = new PoolingHttpClientConnectionManager();
poolingConnManager.setValidateAfterInactivity(5000);
void o.a.h.i.c.PoolingHttpClientConnectionManager.setValidateAfterInactivity(int ms)
Defines period of inactivity in milliseconds after which persistent connections must be re-validated prior to being leased to the consumer. Non-positive value passed to this method disables connection validation. This check helps detect connections that have become stale (half-closed) while kept inactive in the pool.
If you're also controlling the consumed API, you can adapt the keep-alive strategy to the timing your client uses. We're using AWS Cloudfront + ELB's with connection draining for deregistered instances to ensure the kept-alive connections are fully closed, when performing a rolling upgrade. I guess as long as the connections are guaranteed to be kept alive for, say 30 seconds, any value passed to the connection manager below that will always ensure the validity check will mitigate any runtime I/O errors which are purely related to stale/expired connections.
I've been looking a bit to Flume's HttpSource internals, trying yo figure out how the Jetty server is used.
I've seen a single element list of Connectors is used; this Connector will listen for incoming Http connections on configured Http host and port. Then a Context is created for the root path, and a HttpServlet is added to this Context containing the logic to be executed when a connection is received. Finally, the Jetty server is started.
Connector[] connectors = new Connector[1];
if (sslEnabled) {
SslSocketConnector sslSocketConnector = new HTTPSourceSocketConnector(excludedProtocols);
...
connectors[0] = sslSocketConnector;
} else {
SelectChannelConnector connector = new SelectChannelConnector();
...
connectors[0] = connector;
}
connectors[0].setHost(host);
connectors[0].setPort(port);
srv.setConnectors(connectors);
try {
org.mortbay.jetty.servlet.Context root = new org.mortbay.jetty.servlet.Context(srv, "/", org.mortbay.jetty.servlet.Context.SESSIONS);
root.addServlet(new ServletHolder(new FlumeHTTPServlet()), "/");
HTTPServerConstraintUtil.enforceConstraints(root);
srv.start();
...
My question is, seen the above implementation: does such a Jetty server create a thread for each incoming Http connection? Or does a unique HttpServlet serve all the requests, one by one, sequentially?
Thanks for helping!
First of note: org.mortbay.jetty means you are using a very old version of Jetty. Probably Jetty 5 or Jetty 6. Those have been EOL (End of Life'd) way back in 2010 (and earlier).
Back in the Jetty 6 days, there was a ThreadPool that was used on-demand, and depending on your Connector type it would either result in a thread per connection (known as blocking connectors), or a thread per nio selection (in this case your 1 connections have many threads over the lifetime of the connection, but never more than 1 active per connection).
Starting with Jetty 8, and Servlet async, this threading model was refactored to favor async behavior of request processing more.
With Jetty 9, all blocking connectors were dropped in favor of supporting fully async processing of the request, its inputstreams, and its outputstreams.
The current model is for a ThreadPool of threads to be used, on demand, only when needed by a connection (this could be for processing of the request, or the response, or reading the request body content, or writing the response body content, or active websocket streaming, etc...)
This model is preferred for SPDY and HTTP/2 based support, where you have multiple requests per physical connection. But know that in those models its quite possible to have multiple active threads per physical connection, depending on behavior of your servlets.
Also, the web application itself can choose to spin up more threads for its own processing, such as via the servlet async processing behaviors, or to initiate outgoing requests to other services, or to process other tasks that are unrelated to a specific request / response context.
I though Spymemcached does attempt to reestablish connection to the server when this connection get lost.
But I am seeing something different; wondering what I misunderstand or what I do wrong. Here is some sample code:
MemcachedClient c=new MemcachedClient(AddrUtil.getAddresses("server:11211"));
while(true)
try {
System.out.println(c.get("hatsts"));
Thread.sleep(10000);
} catch(Exception e) {
e.printStackTrace();
}
It runs initially without problem. Then I pull the network plug. Subsequently, the client detects a network failure and throws following exception:
net.spy.memcached.OperationTimeoutException: Timeout waiting for value
But then, when i re-establish the network, the client does not recover and continues throwing the exception; even after 5 min. I tried SpyMemcached 2.10.6 and 2.9.0.
What am I missing?
The problem here is that because you pulled the network cable the tcp socket on you client still thinks the connection is valid. The tcp keepalive varies from operating system to operating system and can be as high as 30 minutes. As a result the application (in this case Spymemcached) is not notified that the connection is no longer valid by the tcp layer and therefore Spymemcached doesn't try to reconnect.
The way Spymemcached detects this situation is by counting the amount of consecutive operation timeouts. The last time I checked the default value was 99. Once this many ops time out then Spymemcached will reconnect. You can change this value in the ConnectionFactory if you want to set it to some other value. There's a function called getContinuousTimeout() which is where the Spymemcached gets 99 from by default. You can construct your own ConnectionFactory with the ConnectionFactoryBuilder.
Hopefully this is enough information to answer your question and get you going in the right direction. If not let me know and I can add some more details.
Looking at JDK 1.7 API. I cannot seem to be able to set a connection timeout on an AsynchonousSocketChannel. Is there anyway I can setup a connection timeout on such a channel?
Thanks.
The answer is: you can't.
The first thing to understand is how a TCP connect works. The kernel is sending SYN packets, backing off the time between each retry. This can be tuned via kernel parameters. An article covering this in detail (for linux) can be found here
To give you an idea of what's involved to implement your own shorter timeout for a socket connect is to put the socket in nonblocking mode, throw it in a select() with a timeout, then use getsockopt() to see what happened. This StackOverflow answer shows how this works.
With NIO.2, the connection process is handled for you using threads that you don't have access to. Unfortunately there's no way to tell it you want a shorter timeout on connects; it just calls your completion handler / notifies the Future when the connection either succeeds of fails (including timing out).
You do have the option of calling get(timeout, unit) on the returned Future, then cancelling the Futureif it times out ... but that means if you want the connect to be async you have to add another layer of threading / callbacks and may as well just implement your own async thing with nio.
One last thing worth mentioning since you're looking at async network stuff is that Netty does give this to you (using Nio):
Bootstrap bootstrap = new Bootstrap()
.group(new NioEventLoopGroup())
.channel(NioSocketChannel.class)
.remoteAddress(new InetSocketAddress(remoteAddress, port))
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectionTimeout);
ChannelFuture f = bootstrap.connect();
And you can register a listener with that ChannelFuture for a callback.
I'm issuing a socket connection, using the following snippet
Socket socket = new Socket();
InetSocketAddress endPoint = new InetSocketAddress("localhost", 1234);
try
{
socket.connect(endPoint, 30000);
}
catch (IOException e)
{
e.printStackTrace();
// Logging
}
The endpoint it is trying to connect to is offline, what I want it to do is to attempt to connect, and using the 30000ms timeout, wait for that period of time before it concludes a result
Currently, that 30000 parameter doesn't seem to be applied, as from the timestamp on my logging it appears that it is determining within 1 second that a connection failed.
How can I force the connect to wait for a set amount of time before giving up?
13:13:57,685 6235 DEBUG [Thread-7] - Unable to connect to [localhost:1234]
13:13:58,685 7235 DEBUG [Thread-7] - Unable to connect to [localhost:1234]
13:13:59,695 8245 DEBUG [Thread-7] - Unable to connect to [localhost:1234]
13:14:00,695 9245 DEBUG [Thread-7] - Unable to connect to [localhost:1234]
EDIT : The API does state Connects this socket to the server with a specified timeout value. A timeout of zero is interpreted as an infinite timeout. The connection will then block until established or an error occurs. however it appears I'm not experiencing such behaviour, or am not catering to it, most likely the latter
What you're getting here is correct. connect won't sit on a socket waiting until it sees a server, it will attempt to connect and wait for a response. if there is nothing to connect to, it returns. if there is something to connect to, it will wait timeout seconds for a response and fail if none is received.
You need to distinguish among several possible exception conditions.
ConnectException with the text 'connection refused', which means the host was up and reachable and nothing was listening at the port. This happens very quickly and cannot be subjected to a timeout.
NoRouteToHostException: this indicates a connectivity issue. Again it happens immediately and cannot be subjected to a timeout.
UnknownHostException: the host names cannot be resolved via DNS. This happens immediately, or rather after a generally short DNS delay, and cannot be subjected to a timeout.
ConnectException with any other text: this can indicate a failure to respond by the target system. Usually happens when firewalls are present. Can be subjected to a timeout.
You are doing the correct thing by calling Socket.connect() with a timeout parameter. If you don't do this, or if you specify a zero timeout, the default system timeout is used, which is of the order of 60-75 seconds depending on the platform. This is contrary to the Javadoc's statement about an 'infinite timeout', which is not correct. Also you cannot increase the timeout beyond this limit via Socket.connect() witha a timeout parameter. Alternatively you can use java.nio socket channels in non-blocking mode with a select() to administer the timeout for you, but you still can't increase the timeout beyond the platform default via this or any other method.
When the timeout occurs, a SocketTimeoutException exception is thrown which you do not catch and log. The IOException is fired when "an error occurs during the connection". The timeout is never applied because there's an error beforehand.
Edit: Just to clarify: TCP/IP as a suite has many specifics that could prevent a packet from reaching it's desired outcome (a SYN/ACK packet). If a computer responds to your SYN packet by an informing your application that the port is closed (i.e. there's no application running/listening there), it would fire an exception telling you that it is impossible to connect to that port. If you wish to send and re-send SYN packets either way with the knowledge that an application will come online listening on that port, this is done on a different network layer (and, as far as I know, is not accessible with Java out-of-the-box).
Try scocket.setSoTimeout(timeout) before connecting.