I'm adding a throttling feature in my application, which needs to close a channel before the SSL handshake to mitigate CPU usage when the incoming request rate exceeds the threshold. Now I use Netty SslHandler with server mode to do the handshake. My current implementation is adding a ChannelInboundHandlerAdapter before the SslHandler and rewrite the channelActive method:
#Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
if (!limiter.tryAcquire()) {
ctx.close();
return;
}
ctx.pipeline().remove(this);
ctx.fireChannelActive();
}
In this way the channel can be closed when it becomes active. However, I will receive SslHandshakeCompletionEvent with handshake failure. I read the source code of Netty SslHandler. It will set handshake failure in channelInactive method when the channelInactive event is triggered. So I wonder whether there is a better way to close the channel without firing the handshake failure event, since the handshake process has not started when my throttling close the channel.
You're attacking the problem at the wrong end. The idea is not to close channels that are being accepted too quickly, but to not accept them too quickly, which will be done in the accept handler, not the channel handler.
Your approach is almost right, however, it would be correct to override channelRegistered method instead of channelActive.
#Override
public void channelRegistered(ChannelHandlerContext ctx) throws Exception {
if (isGlobalLimitReached()) {
ctx.close();
} else {
ctx.pipeline().remove(this);
ctx.pipeline().addFirst(sslCtx.newHandler(ctx.alloc()));
ctx.fireChannelRegistered();
}
}
Also, you need to add SslHandler at the beginning of pipeline on the fly and only when your condition is met (no limit reached).
In that case, you'll not get SslHandshakeCompletionEvent. As handshake is started when SslHandler is added to pipeline and
channel.isActive() return true (your case in above code).
channelRegistered is most recent point where you can refuse/close connection.
Related
Netty complains about uncaught DecoderException exceptions, and even though I catch some of these with my handler, I obviously do not catch all:
2020-09-30 19:55:38.952 [Netty-NioEventLoop-Thread-1] WARN i.n.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
My setup is a bit complex, but here are the main gists
The channel setup in SecuredChannelInitializer sets up an SSL handler on the pipeline
#Override
protected void initChannel(final SocketChannel socketChannel) {
final var pipeline = socketChannel.pipeline();
pipeline.addLast(sslCtx.newHandler(socketChannel.alloc()));
pipeline.addLast(createProtocolNegotiationHandler());
}
This calls out to
private ApplicationProtocolNegotiationHandler createProtocolNegotiationHandler() {
return new ApplicationProtocolNegotiationHandler(HTTP_1_1) {
#Override
protected void configurePipeline(ChannelHandlerContext ctx, String protocol) {
final var pipeline = ctx.pipeline();
addProtocolHandlers(protocol, pipeline);
applicationPipelineInitializer.addHandlersToPipeline(pipeline);
}
This last step adds a bunch of handlers
#Override
public void addHandlersToPipeline(final ChannelPipeline pipeline) {
pipeline.addLast(
new HttpLoggerHandler(),
connectionErrorHandler,
...
So this basically boils down to a pipeline that consists of
ssl handler
protocol handlers (http 1 / 2)
http logger
connection error handler (implements exceptionCaught)
The connection error handler looks like this:
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
if (isDevelopment && (cause instanceof SSLHandshakeException || cause instanceof DecoderException)) {
logger.debug("SSL handshake exceptions. Ignored in development, as self-signed certificates are common");
} else {
ctx.fireExceptionCaught(cause);
}
}
This does get called, but very seldom! There are quite a few errors like above buzzing by, but every now and then this might get called, so it seems like two different things. It almost seems like we are talking about two different pipelines, but I do not see how that is possible. When setting a breakpoing in DefaultChannelPipeline, I can see that it is for the same port as I set up above. I tried inspecting the objects for the pipeline connected to DefaultChannelPipeline in the debugger, but could not really tell if the pipeline was the same as the one I had setup.
There are multiple flows within a pipeline. An inbound and outbound flow. Does your error handler handle both inbound and outbound errors that aren't handled by anything else?
Are you sure that your pipeline ordering is correct? The error handler must be at the end of both inbound and outbound processing. See https://netty.io/4.0/api/io/netty/channel/ChannelPipeline.html#addLast-io.netty.channel.ChannelHandler
For details of "How an event flows in a pipeline"
What's the correct way of handling a websocket error besides logging it?
Regarding onError(), the Endpoint documentation states that:
Developers may implement this method when the web socket session
creates some kind of error that is not modeled in the web socket
protocol. This may for example be a notification that an incoming
message is too big to handle, or that the incoming message could not
be encoded.
There are a number of categories of exception that this method is
(currently) defined to handle:
connection problems, for example, a socket failure that occurs before the web socket connection can be formally closed. These are modeled as SessionExceptions
runtime errors thrown by developer created message handlers calls.
conversion errors encoding incoming messages before any message handler has been called. These are modeled as DecodeExceptions
Are all of these types of exceptions fatal, causing the websocket to close?
Should the onError() method close the websocket (call Session.close()) if an error occurs?
So far, I assumed that it's my responsibility to cleanly close the session, informing the client about the close reason. This is why my onError() tried invoking session.close() if session.isOpen() returned true, but this caused tomcat (8.0.15) to throw a NullPointerException:
...
Caused by: java.lang.NullPointerException
at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.onWritePossible(WsRemoteEndpointImplServer.java:96)
at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.doWrite(WsRemoteEndpointImplServer.java:81)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.writeMessagePart(WsRemoteEndpointImplBase.java:444)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.startMessage(WsRemoteEndpointImplBase.java:335)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.startMessageBlock(WsRemoteEndpointImplBase.java:264)
at org.apache.tomcat.websocket.WsSession.sendCloseMessage(WsSession.java:536)
at org.apache.tomcat.websocket.WsSession.doClose(WsSession.java:464)
at org.apache.tomcat.websocket.WsSession.close(WsSession.java:441)
at my.package.MyEndpoint.onWebSocketError(MyEndpoint.java:229)
... 18 more
Is this a tomcat bug, a misunderstanding on my part, or both?
Edit: It seems that the Java EE websocket example dukeeetf2 assumes that errors are fatal; and that there's no need to close the session. The errors are logged, and the session is removed:
#OnError
public void error(Session session, Throwable t) {
/* Remove this connection from the queue */
queue.remove(session);
logger.log(Level.INFO, t.toString());
logger.log(Level.INFO, "Connection error.");
}
#OnError method invocation does not mean that Session will be closed; You can do whatever you want, it depends in the contract specified by your application.
stacktrace from tomcat implementation seems like a bug.
ad dukeeetf2 sample - seems like this code contains other assumptions - Endpoints does not throw an exception, so everything caught here is from underlying WebSocket framework implementation. That does not really mean that there is an "Connection Error"; I would maybe do close right away (if this is how I wan't my application to handle errors); this implementation could result in opened connections without any messages.
I saw this is a bit dated but ended up here today when looking for this info.
Depending on how you rely on the state of the websocket, you need to close the session manually, at least for the javax.websocket implementation.
In my case, the error happening was causing a problem for the websession client administration implementation, so I closed the session as in the above example.
I think it depends on what you need, but it certainly does not do a close session in this implementation.
My server uses a AsynchronousServerSocketChannel that listens for client connections using a CompletionHandler. When a client connection is accepted, the AsynchronousSocketChannel is read, again using a CompletionHandler to receive the data with no timeout.
So far so good, my client connects, writes data that is read by the server, which is able to respond sending data back to the client via the same socket.
When my client terminates, it calls AsynchronousSocketChannel.close(), to close the socket. When this call is made the server is waiting to read data from the socket.
I had expected the call to AsynchronousSocketChannel.close() on the client to translate into a callback to CompletionHandler.completed with a read length of -1 on the server, indicating the socket had been closed, however the callback is to CompletionHandler.failed with the following exception:
java.io.IOException: The specified network name is no longer available.
at sun.nio.ch.Iocp.translateErrorToIOException(Iocp.java:309)
at sun.nio.ch.Iocp.access$700(Iocp.java:46)
at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:399)
at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:744)
How should a client close a socket, so that it is not seen as an error on the server?
The documentation on close says that it causes
AsynchronousCloseException or ClosedChannelException on the other side.
To cause completed(-1) the client should call shutdownInput.
However, I would treat AsynchronousCloseException and ClosedChannelException as normal shutdown, along with completed(-1).
Looking at stack traces and the implementation sources, you might notice that the exception is thrown by the internal sun.nio.ch.UnixAsynchronousSocketChannelImpl#finish() method which checks for pending read/write operations. So the only way to avoid this exception is to prevent new asynchronous read() and write() calls at some point, and that should be a part of the application's logic.
I've dealt with this exception quite a lot, and in most cases the root problem was in the completion handler's unconditional "continue listening" calls:
conn.read(src, attachment, new CompletionHandler<Integer, T>() {
#Override
public void completed(Integer result, T attachment) {
// Some business logic here
// Below is the problem - unconditional read() call
conn.read(src, attachment, this);
}
#Override
public void failed(Throwable t, T attachment) {
// Exception handling
}
});
To gracefully close the connection, there should be no unconditional async read/write calls. To achieve that, one might need to send an additional message which would mean that no new async data is expected and it's safe to close the connection. And the correct pseudo-code would look something like this:
conn.read(src, attachment, new CompletionHandler<Integer, T>() {
#Override
public void completed(Integer result, T attachment) {
// Some business logic here
if(continueListening()) {
conn.read(src, attachment, this);
}
}
// ...
});
I have a webservice which is accepting a POST method with XML. It is working fine then at some random occasion, it fails to communicate to the server throwing IOException with message The target server failed to respond. The subsequent calls work fine.
It happens mostly, when i make some calls and then leave my application idle for like 10-15 min. the first call which I make after that returns this error.
I tried couple of things ...
I setup the retry handler like
HttpRequestRetryHandler retryHandler = new HttpRequestRetryHandler() {
public boolean retryRequest(IOException e, int retryCount, HttpContext httpCtx) {
if (retryCount >= 3){
Logger.warn(CALLER, "Maximum tries reached, exception would be thrown to outer block");
return false;
}
if (e instanceof org.apache.http.NoHttpResponseException){
Logger.warn(CALLER, "No response from server on "+retryCount+" call");
return true;
}
return false;
}
};
httpPost.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, retryHandler);
but this retry never got called. (yes I am using right instanceof clause). While debugging this class never being called.
I even tried setting up HttpProtocolParams.setUseExpectContinue(httpClient.getParams(), false); but no use. Can someone suggest what I can do now?
IMPORTANT
Besides figuring out why I am getting the exception, one of the important concerns I have is why isn't the retryhandler working here?
Most likely persistent connections that are kept alive by the connection manager become stale. That is, the target server shuts down the connection on its end without HttpClient being able to react to that event, while the connection is being idle, thus rendering the connection half-closed or 'stale'. Usually this is not a problem. HttpClient employs several techniques to verify connection validity upon its lease from the pool. Even if the stale connection check is disabled and a stale connection is used to transmit a request message the request execution usually fails in the write operation with SocketException and gets automatically retried. However under some circumstances the write operation can terminate without an exception and the subsequent read operation returns -1 (end of stream). In this case HttpClient has no other choice but to assume the request succeeded but the server failed to respond most likely due to an unexpected error on the server side.
The simplest way to remedy the situation is to evict expired connections and connections that have been idle longer than, say, 1 minute from the pool after a period of inactivity. For details please see the 2.5. Connection eviction policy of the HttpClient 4.5 tutorial.
Accepted answer is right but lacks solution. To avoid this error, you can add setHttpRequestRetryHandler (or setRetryHandler for apache components 4.4) for your HTTP client like in this answer.
HttpClient 4.4 suffered from a bug in this area relating to validating possibly stale connections before returning to the requestor. It didn't validate whether a connection was stale, and this then results in an immediate NoHttpResponseException.
This issue was resolved in HttpClient 4.4.1. See this JIRA and the release notes
Solution: change the ReuseStrategy to never
Since this problem is very complex and there are so many different factors which can fail I was happy to find this solution in another post: How to solve org.apache.http.NoHttpResponseException
Never reuse connections:
configure in org.apache.http.impl.client.AbstractHttpClient:
httpClient.setReuseStrategy(new NoConnectionReuseStrategy());
The same can be configured on a org.apache.http.impl.client.HttpClientBuilder builder:
builder.setConnectionReuseStrategy(new NoConnectionReuseStrategy());
Although accepted answer is right, but IMHO is just a workaround.
To be clear: it's a perfectly normal situation that a persistent connection may become stale. But unfortunately it's very bad when the HTTP client library cannot handle it properly.
Since this faulty behavior in Apache HttpClient was not fixed for many years, I definitely would prefer to switch to a library that can easily recover from a stale connection problem, e.g. OkHttp.
Why?
OkHttp pools http connections by default.
It gracefully recovers from situations when http connection becomes stale and request cannot be retried due to being not idempotent (e.g. POST). I cannot say it about Apache HttpClient (mentioned NoHttpResponseException).
Supports HTTP/2.0 from early drafts and beta versions.
When I switched to OkHttp, my problems with NoHttpResponseException disappeared forever.
Nowadays, most HTTP connections are considered persistent unless declared otherwise. However, to save server ressources the connection is rarely kept open forever, the default connection timeout for many servers is rather short, for example 5 seconds for the Apache httpd 2.2 and above.
The org.apache.http.NoHttpResponseException error comes most likely from one persistent connection that was closed by the server.
It's possible to set the maximum time to keep unused connections open in the Apache Http client pool, in milliseconds.
With Spring Boot, one way to achieve this:
public class RestTemplateCustomizers {
static public class MaxConnectionTimeCustomizer implements RestTemplateCustomizer {
#Override
public void customize(RestTemplate restTemplate) {
HttpClient httpClient = HttpClientBuilder
.create()
.setConnectionTimeToLive(1000, TimeUnit.MILLISECONDS)
.build();
restTemplate.setRequestFactory(
new HttpComponentsClientHttpRequestFactory(httpClient));
}
}
}
// In your service that uses a RestTemplate
public MyRestService(RestTemplateBuilder builder ) {
restTemplate = builder
.customizers(new RestTemplateCustomizers.MaxConnectionTimeCustomizer())
.build();
}
This can happen if disableContentCompression() is set on a pooling manager assigned to your HttpClient, and the target server is trying to use gzip compression.
Same problem for me on apache http client 4.5.5
adding default header
Connection: close
resolve the problem
Use PoolingHttpClientConnectionManager instead of BasicHttpClientConnectionManager
BasicHttpClientConnectionManager will make an effort to reuse the connection for subsequent requests with the same route. It will, however, close the existing connection and re-open it for the given route.
I have faced same issue, I resolved by adding "connection: close" as extention,
Step 1: create a new class ConnectionCloseExtension
import com.github.tomakehurst.wiremock.common.FileSource;
import com.github.tomakehurst.wiremock.extension.Parameters;
import com.github.tomakehurst.wiremock.extension.ResponseTransformer;
import com.github.tomakehurst.wiremock.http.HttpHeader;
import com.github.tomakehurst.wiremock.http.HttpHeaders;
import com.github.tomakehurst.wiremock.http.Request;
import com.github.tomakehurst.wiremock.http.Response;
public class ConnectionCloseExtension extends ResponseTransformer {
#Override
public Response transform(Request request, Response response, FileSource files, Parameters parameters) {
return Response.Builder
.like(response)
.headers(HttpHeaders.copyOf(response.getHeaders())
.plus(new HttpHeader("Connection", "Close")))
.build();
}
#Override
public String getName() {
return "ConnectionCloseExtension";
}
}
Step 2: set extension class in wireMockServer like below,
final WireMockServer wireMockServer = new WireMockServer(options()
.extensions(ConnectionCloseExtension.class)
.port(httpPort));
We have a socket application which the snippet of the while loop is as below. What we would like to check is that say if it pass 30 seconds and no more data then shut the socket connection. At event if the some data is in then we reset the timer. Must I use the timer or system milliseconds
while ((readChar=readSocket.read()) != -1)
{
//processing.
}
You can configure the socket so that a read operation times out if no data is received within the specified interval.
From the Socket Javadoc:
public void setSoTimeout(int timeout) throws SocketException
Enable/disable SO_TIMEOUT with the specified timeout, in milliseconds. With this option set to a non-zero timeout, a read() call on the InputStream associated with this Socket will block for only this amount of time. If the timeout expires, a java.net.SocketTimeoutException is raised, though the Socket is still valid. The option must be enabled prior to entering the blocking operation to have effect. The timeout must be > 0. A timeout of zero is interpreted as an infinite timeout.
Parameters:
timeout - the specified timeout, in milliseconds.
Throws:
SocketException - if there is an error in the underlying protocol, such as a TCP error.
Since:
JDK 1.1
See Also:
getSoTimeout()
Using this approach, you can read data, consume it (however your need to), and then read from the socket again. If you get the timeout exception, then close the socket.
socket.setSoTimeout(30 * 1000); // timeout after 30 seconds
try
{
while ((readChar=readSocket.read()) != -1) // block reading data ...
{
// processing ...
}
}
catch (SocketTimeoutException e) // we didn't get any data within 30 seconds ...
{
socket.close(); // ... close the socket
}
Use asynchronous NIO operations.
If you use java6, async operations are tricky, but there are many network libraries (Mina, Netty) though they are rather heavy.
If you use java7, true async network operations are implemented and are easy to use (nio2). Even more easier is to use a lightweight nio2 library from https://github.com/rfqu/df4j.