Here's what I know so far (please correct me):
In the RabbitMQ Java client, operations on a channel throw IOException when there is a general network failure (malformed data from broker, authentication failures, missed heartbeats).
Operations on a channel can also throw the ShutdownSignalException unchecked exception, typically an AlreadyClosedException when we tried to perform an action on the channel/connection after it has been shut down.
The shutting down process happens in the event of "network failure, internal failure or explicit local shutdown" (e.g. via channel.close() or connection.close()). The shutdown event propagates down the "topology", from Connection -> Channel -> Consumer, and when the Channel it calls the Consumer's handleShutdown() method gets called.
A user can also add a shutdown listener which is called after the shutdown process completes.
Here is what I'm missing:
Since an IOException indicates a network failure, does it also initiate a shutdown request?
How does using auto-recovery mode affect shutdown requests? Does it cause channel operations to block while it tries to reconnect to the channel, or will the ShutdownSignalException still be thrown?
Here is how I'm handling exceptions at the moment, is this a sensible approach?
My setup is that I'm polling a QueueingConsumer and dispatching tasks to a worker pool. The rabbitmq client is encapsulated in MyRabbitMQWrapper here. When an exception occurs polling the queue I just gracefully shutdown everything and restart the client. When an exception occurs in the worker I also just log it and finish the worker.
My biggest worry (related to Question 1): Suppose an IOException occurs in the worker, then the task doesn't get acked. If the shutdown does not then occur, I now have an un-acked task that will be in limbo forever.
Pseudo-code:
class Main {
public static void main(String[] args) {
while(true) {
run();
//Easy way to restart the client, the connection has been
//closed so RabbitMQ will re-queue any un-acked tasks.
log.info("Shutdown occurred, restarting in 5 seconds");
Thread.sleep(5000);
}
}
public void run() {
MyRabbitMQWrapper rw = new MyRabbitMQWrapper("localhost");
try {
rw.connect();
while(!Thread.currentThread().isInterrupted()) {
try {
//Wait for a message on the QueueingConsumer
MyMessage t = rw.getNextMessage();
workerPool.submit(new MyTaskRunnable(rw, t));
} catch (InterruptedException | IOException | ShutdownSignalException e) {
//Handle all AMQP library exceptions by cleaning up and returning
log.warn("Shutting down", e);
workerPool.shutdown();
break;
}
}
} catch (IOException e) {
log.error("Could not connect to broker", e);
} finally {
try {
rw.close();
} catch(IOException e) {
log.info("Could not close connection");
}
}
}
}
class MyTaskRunnable implements Runnable {
....
public void run() {
doStuff();
try {
rw.ack(...);
} catch (IOException | ShutdownSignalException e) {
log.warn("Could not ack task");
}
}
}
Related
The below program acts as TCP client and uses NIO to open socket to a remote server, as below
private Selector itsSelector;
private SocketChannel itsChannel;
public boolean getConnection(Selector selector, String host, int port)
{
try
{
itsSelector = selector;
itsChannel = SocketChannel.open();
itsChannel.configureBlocking(false);
itsChannel.register(itsSelector, SelectionKey.OP_CONNECT);
itsChannel.connect(new InetSocketAddress(host, port));
if (itsChannel.isConnectionPending())
{
while (!itsChannel.finishConnect())
{
// waiting until connection is finished
}
}
itsChannel.register(itsSelector, SelectionKey.OP_WRITE);
return (itsChannel != null);
}
catch (IOException ex)
{
close();
if(ex instanceof ConnectException)
{
LOGGER.log(Level.WARNING, "The remoteserver cannot be reached");
}
}
}
public void close()
{
try
{
if (itsChannel != null)
{
itsChannel.close();
itsChannel.socket().close();
itsSelector.selectNow();
}
}
catch (IOException e)
{
LOGGER.log(Level.WARNING, "Connection cannot be closed");
}
}
This program runs on Red Hat Enterprise Linux Server release 6.2 (Santiago)
When number of concurrent sockets are in establishment phase, file descriptor limit reaches a max value and I see below exception while trying to establish more socket connections.
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
This happens only when the remote Node is down, and while it is up, all is fine.
When the remote TCP server is down, below exception is thrown as is handled as IOException in the above code
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
Is there any way to forcefully close the underlying file descriptor in this case.
Thanks in advance for all the help.
private Selector itsSelector;
I cannot see the point of this declaration. You can always get the selector the channel is registered with, if you need it, which you never do. Possibly you are leaking Selectors?
itsChannel.configureBlocking(false);
itsChannel.register(itsSelector, SelectionKey.OP_CONNECT);
Here you are registering for OP_CONNECT but never making the slightest use of the facility.
itsChannel.connect(new InetSocketAddress(host, port));
Here you are starting a pending connection.
if (itsChannel.isConnectionPending())
It is. You just started it. The test is pointless.
{
while (!itsChannel.finishConnect())
{
// waiting until connection is finished
}
}
This is just a complete waste of time and space. If you don't want to use the selector to detect when OP_CONNECT fires, you should call connect() before setting the channel to non-blocking, and get rid of this pointless test and loop.
itsChannel.register(itsSelector, SelectionKey.OP_WRITE);
return (itsChannel != null);
itsChannel cannot possibly be null at this point. The test is pointless. You would be better off allowing the IOExceptions that can arise to propagate out of this method, so that the caller can get some idea of the failure mode. That also places the onus on the caller to close on any exception, not just the ones you're catching here.
catch (IOException ex)
{
close();
if(ex instanceof ConnectException)
{
LOGGER.log(Level.WARNING, "The remoteserver cannot be reached");
}
}
See above. Remove all this. If you want to distinguish ConnectException from the other IOExceptions, catch it, separately. And you are forgetting to log anything that isn't a ConnectException.
public void close()
{
try
{
if (itsChannel != null)
{
itsChannel.close();
itsChannel.socket().close();
itsSelector.selectNow();
The second close() call is pointless, as the channel is already closed.
catch (IOException e)
{
LOGGER.log(Level.WARNING, "Connection cannot be closed");
}
I'm glad to see you finally logged an IOException, but you're not likely to get any here.
Don't write code like this.
Is there any way to stop SedaConsumer without waiting for BlockingQueue.take(pollTimeout, ...) to return? I have a lot of sedas in my application and a graceful shutdown takes a lot of time. When DefaultShutdownStrategy shutdown sedaConsumers there are no more messages in the queue and no more messages will be produced (because of implementation of routes shutdown before). So each sedaConsumer has to wait about 1 second.
Is it possible to force doStop instead of prepareShutdown for seda? Or interrupt workers threads?
I know I can decrease pollTimeout, but I afraid it will affect runtime performance.
In SedaConsumer.java:
try {
// use the end user configured poll timeout
exchange = queue.poll(pollTimeout, TimeUnit.MILLISECONDS);
// Omitted
} catch (InterruptedException e) {
LOG.debug("Sleep interrupted, are we stopping? {}", isStopping() || isStopped());
continue;
} catch (Throwable e) {
if (exchange != null) {
getExceptionHandler().handleException("Error processing exchange", exchange, e);
} else {
getExceptionHandler().handleException(e);
}
}
This construct is at most places in the thread where an InterruptedException can be thrown so if the consumer is stopping and is interrupted it will stop gracefully.
I'm developing a Google Glass app which needs to listen for UDP packets in a worker thread (integrating with an existing system which sends UDP packets). I previously posted a question (see here) and received an answer which provided some guidance on how to do this. Using the approach in the other discussion I'll have a worker thread which is blocked on DatagramSocket.receive().
Further reading suggests to me that I'll need to be able to start/stop this on demand. So this brings me to the question I'm posting here. How can I do the above in such a way as to be able to interrupt (gracefully) the UDP listening? Is there some way I can "nicely" ask the socket to break out of the receive() call from another thread?
Or is there another way to listen for UDP packets in an interruptable fashion so I can start/stop the listener thread as needed in response to device events?
My recommendation:
private DatagramSocket mSocket;
#Override
public void run() {
Exception ex = null;
try {
// read while not interrupted
while (!interrupted()) {
....
mSocket.receive(...); // excepts when interrupted
}
} catch (Exception e) {
if (interrupted())
// the user did it
else
ex = e;
} finally {
// always release
release();
// rethrow the exception if we need to
if (ex != null)
throw ex;
}
}
public void release() {
// causes exception if in middle of rcv
if (mSocket != null) {
mSocket.close();
mSocket = null;
}
}
#Override
public void interrupt() {
super.interrupt();
release();
}
clean cut, simple, always releases and interrupting stops you cleanly in 2 cases.
I'm digging a bug in my netty program:I used a heartbeat handler between the server and client,when client system rebooting,the heartbeat handler in server side will be aware of timeout and then close the Channel,but sometimes the listener registered in Channel's CloseFuture never be notified,that's weird.
After digging netty 3.5.7 source code,I figure out that the only way a Channel's CloseFuture be notified is through AbstractChannel.setClosed();May be this method not be executed when Channel is closed,see below:
NioServerSocketPipelineSink:
private static void close(NioServerSocketChannel channel, ChannelFuture future) {
boolean bound = channel.isBound();
try {
if (channel.socket.isOpen()) {
channel.socket.close();
Selector selector = channel.selector;
if (selector != null) {
selector.wakeup();
}
}
// Make sure the boss thread is not running so that that the future
// is notified after a new connection cannot be accepted anymore.
// See NETTY-256 for more information.
channel.shutdownLock.lock();
try {
if (channel.setClosed()) {
future.setSuccess();
if (bound) {
fireChannelUnbound(channel);
}
fireChannelClosed(channel);
} else {
future.setSuccess();
}
} finally {
channel.shutdownLock.unlock();
}
} catch (Throwable t) {
future.setFailure(t);
fireExceptionCaught(channel, t);
}
}
in some platform channel.socket.close() may throw IOException,that means channel.setClosed() may never executed,so the listener registered in CloseFuture may not be notified.
Here is my question:Do you ever encounter this problem? Is the analysis right?
I figure out it's my heartbeat handler cause the problem:never timeout,so never close the channel,below is running in a timer :
if ((now - lastReadTime > heartbeatTimeout)
&& (now - lastWriteTime > heartbeatTimeout)) {
getChannel().close();
stopHeartbeatTimer();
}
where lastReadTime and lastWriteTime are updated like below:
public void writeComplete(ChannelHandlerContext ctx, WriteCompletionEvent e)
throws Exception {
lastWriteTime = System.currentTimeMillis();
super.writeComplete(ctx, e);
}
public void messageReceived(ChannelHandlerContext ctx, MessageEvent e)
throws Exception {
lastReadTime = System.currentTimeMillis();
super.messageReceived(ctx, e);
}
Remote client is Windows xp,current server is Linux,both jdk1.6.
I think the writeComplete still invoked internally after remote client's system is rebooting,although messageReceived not invoked,no IOExceptoin is thrown during this period.
I will redesign the heartbeat handler,attaching a timestamp and a HEART_BEAT flag in heartbeat packet,when the peer side received the packet,send back the packet with the same timestamp and a ACK_HEART_BEAT flag,when the current side received this ack packet,use this timestamp to update lastWriteTime.
I'm using a variation of the example at http://svn.apache.org/repos/asf/activemq/trunk/assembly/src/release/example/src/StompExample.java to receive message from a queue. What I'm trying to do is to keep listening to a queue and perform some action upon reception of a new message. The problem is that I couldn't find a way to register a listener to any of the related objects. I've tried something like:
public static void main(String args[]) throws Exception {
StompConnection connection = null;
try {
connection = new StompConnection();
connection.open("localhost", 61613);
connection.connect("admin", "activemq");
connection.subscribe("/queue/worker", Subscribe.AckModeValues.AUTO);
while (true) {
StompFrame message = connection.receive();
System.out.println(message.getBody());
}
} catch (UnknownHostException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
} finally {
if (connection != null) {
connection.disconnect();
}
}
}
but this doesn't work as a time out occurs after a few seconds (java.net.SocketTimeoutException: Read timed out). Is there anything I can do to indefinitely listen to this queue?
ActiveMQ's StompConnection class is a relatively primitive STOMP client. Its not capable of async callbacks on Message or for indefinite waits. You can pass a timeout to receive but depending on whether you are using STOMP v1.1 it could still timeout early if a heart-beat isn't received in time. You can of course always catch the timeout exception and try again.
For STOMP via Java you're better off using StompJMS or the like which behaves like a real JMS client and allows for async Message receipt.
#Tim Bish: I tried StompJMS, but couldn't find any example that I could use (maybe you can provide a link). I 'fixed' the problem by setting the timeout to 0 which seems to be blocking.
even i was facing the same issue.. you can fix this by adding time out to your receive() method.
Declare a long type variable.
long waitTimeOut = 5000; //this is 5 seconds
now modify your receive function like below.
StompFrame message = connection.receive(waitTimeOut);
This will definitely work.