We have abrupt connection close issue to rabbitmq on localhost dev/testing scenario. In our development environment we have rabbitmq installed on each developers Windows 7 machine, and we connect to it using java client via Spring AMQP library. Everything works fine for a while but at some point of time the connection is dropped with the following message in rabbitmq log:
=WARNING REPORT==== 4-Jan-2016::14:39:37 ===
closing AMQP connection <0.3731.0> (127.0.0.1:50792 -> 127.0.0.1:5672):
connection_closed_abruptly
In the client log we have this exception:
04/01/2016 14:39:37.181 (AMQP Connection 127.0.0.1:5672) ERROR [CachingConnectionFactory] Channel shutdown: connection error
04/01/2016 14:39:38.188 (SimpleAsyncTaskExecutor-2) WARN [SimpleMessageListenerContainer] Consumer raised exception, processing can restart if the connection factory supports it
com.rabbitmq.client.ShutdownSignalException: connection error
at com.rabbitmq.client.impl.AMQConnection.startShutdown(AMQConnection.java:723)
at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:713)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:571)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Unrecognized Windows Sockets error: 0: recv failed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:288)
at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95)
at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:139)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:536)
... 1 more
In the client start up we create a dynamic queue and attach several consumers to it. This connection drop causes the queue to be removed and any subsequent attempt to recreate it fails with the following exception:
04/01/2016 14:39:38.239 (SimpleAsyncTaskExecutor-8) WARN [BlockingQueueConsumer] Queue declaration failed; retries left=3
org.springframework.amqp.rabbit.listener.BlockingQueueConsumer$DeclarationException: Failed to declare queue(s):[MY_TEST_QU]
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.attemptPassiveDeclarations(BlockingQueueConsumer.java:571)
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:470)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1165)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106)
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:124)
at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:885)
at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:704)
at com.sun.proxy.$Proxy77.queueDeclarePassive(Unknown Source)
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.attemptPassiveDeclarations(BlockingQueueConsumer.java:550)
... 3 more
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'MY_TEST_QU' in vhost '/', class-id=50, method-id=10)
at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67)
at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33)
at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:361)
at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:226)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118)
... 12 more
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'MY_TEST_QU' in vhost '/', class-id=50, method-id=10)
at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:484)
at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:321)
at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:144)
at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:91)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:554)
... 1 more
We have had this problem with rabbitmq 3.4.x and 3.5.x and Spring AMQP 1.3.x and 1.5.x. The interesting thing is this never happens in our QA and production environment where rabbitmq is installed on a separate server.
Any help is much appreciated.
The org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.attemptPassiveDeclarations() fails when there is no the queue to listen to anymore.
It isn't queue recreation it is just PassiveDeclarations. So, if you drop that "dynamic queue" manually, you should recreate it manually as well.
I mean if you don't follow with the RabbitAdmin bean contract, you don't have choice unless re-create it manually. You don't need to worry about the listener in that case: it will reconnect correctly, when the queue comes back.
UPDATE
Starting with version 1.5 Spring AMQP provides ListenerContainerConsumerFailedEvent, which is emitted exactly after all attemptPassiveDeclarations() attempts. So, you can catch that ApplicationEvent, stop() your listener container, declare queue back and start() the container again.
If your queue isn't a bean, it won't be redeclared automatically.
Related
We have a microservice developed using Sprig Boot. This microservice will called an external Rest API. This is deployed on multiple servers. On one of the servers we are seeing the following error
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:743)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:669)
at org.springframework.web.client.RestTemplate.postForObject(RestTemplate.java:413)
Caused by: javax.net.ssl.SSLException: Connection reset
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:127)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:320)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:258)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:137)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
....
Suppressed: java.net.SocketException: Broken pipe (Write failed)
at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
at java.base/sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:351)
....
Caused by: java.net.SocketException: Connection reset
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:186)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:448)
at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:165)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:108)
on the other servers the same service is running fine.
We tried traceroute on this server and other servers but the output on both are same. We restarted the service but same error.
Any pointers on what we can debug and find where the issue is?
This thread can be closed as after the restart of that service, its working fine.
When a command is executed via SSH connection, often the command execution fails with Exception - "net.schmizz.sshj.connection.ConnectionException: Connection reset Exception is thrown".
Problem description:
I created a SSH connection instance to a host using net.schmizz.sshj.SSHClient. I execute bunch of commands using this SSH connection and finally close the connection.
On checking my java code execution logs, Often I could see the error
2018-04-12T06:15:23.288Z ERROR n.s.s.t.TransportImpl [die:589] [reader] - Dying because - {}
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at net.schmizz.sshj.transport.Reader.run(Reader.java:50)
2018-04-12T06:15:23.288Z INFO n.s.s.t.TransportImpl [notifyDisconnect:182] [reader] - Disconnected - UNKNOWN
The java code execution will continue to run.
But when executing a command using the SSH connection, in that time if this java.net.SocketException: Connection reset Exception comes, my java code execution is terminated.
2018-04-11T12:02:51.755Z INFO c.v.v.u.SSHUtil [executeRemoteSSHCommand:861] [main] - Successfully executed 'tar -C /usr/local/ -zxvf /usr/local/VMware-GuestSDK-10.2.0-8225092.tar.gz --no-same-owner' command on remote ssh host
2018-04-11T12:03:18.121Z ERROR n.s.s.t.TransportImpl [die:589] [reader] - Dying because - {}
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at net.schmizz.sshj.transport.Reader.run(Reader.java:50)
2018-04-11T12:03:18.123Z INFO n.s.s.t.TransportImpl [notifyDisconnect:182] [reader] - Disconnected - UNKNOWN
2018-04-11T12:03:18.127Z ERROR n.s.c.Promise [tryRetrieve:175] [main] - <<chan#6 / open>> woke to: net.schmizz.sshj.connection.ConnectionException: Connection reset
2018-04-11T12:03:18.130Z ERROR c.v.v.e.OutcomePrinter [afterInvocation:40] [main] - exception thrown
net.schmizz.sshj.connection.ConnectionException: Connection reset
at net.schmizz.sshj.connection.ConnectionException$1.chain(ConnectionException.java:32)
at net.schmizz.sshj.connection.ConnectionException$1.chain(ConnectionException.java:26)
at net.schmizz.concurrent.Promise.deliverError(Promise.java:96)
at net.schmizz.concurrent.Event.deliverError(Event.java:74)
at net.schmizz.concurrent.ErrorDeliveryUtil.alertEvents(ErrorDeliveryUtil.java:34)
at net.schmizz.sshj.connection.channel.AbstractChannel.notifyError(AbstractChannel.java:226)
at net.schmizz.sshj.connection.channel.direct.SessionChannel.notifyError(SessionChannel.java:224)
at net.schmizz.sshj.common.ErrorNotifiable$Util.alertAll(ErrorNotifiable.java:35)
at net.schmizz.sshj.connection.ConnectionImpl.notifyError(ConnectionImpl.java:258)
at net.schmizz.sshj.transport.TransportImpl.die(TransportImpl.java:597)
at net.schmizz.sshj.transport.Reader.run(Reader.java:68)
Caused by: net.schmizz.sshj.common.SSHException: Connection reset
at net.schmizz.sshj.common.SSHException$1.chain(SSHException.java:36)
at net.schmizz.sshj.common.SSHException$1.chain(SSHException.java:29)
at net.schmizz.sshj.transport.TransportImpl.die(TransportImpl.java:591)
... 1 common frames omitted
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at net.schmizz.sshj.transport.Reader.run(Reader.java:50)
2018-04-11T12:03:18.131Z INFO c.v.v.e.OutcomePrinter [afterInvocation:47] [main] - OUTCOME:FAIL
Issue:
often the SSH connection disconnects and reconnects but when I execute a command via this SSH connection and in that time if the disconnect happens, it never reconnects instead net.schmizz.sshj.connection.ConnectionException: Connection reset Exception is thrown.
My Query:
How to overcome this ssh disconnection issue from my Java code especially while executing commands via the SSH connection ?
Thanks.
From javadoc:
SocketException: Thrown to indicate that there is an error creating or accessing a Socket.
So basically you don't have way to prevent that as long as it's not your system causing eg. connection drops. Usually this error is because the server from whatever reasons is disconnecting your client.
What you might be able to do is that you simply make the client reconnect if error is thrown. Or if this is an user operated client you could monitor the connectivity in parallel thread and make the client reconnect before user even knows the connection was lost.
Could be because of ssh host verification. Try to ssh from this server first so that it's added to known hosts.
Encountered Internal Server Exception
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "localhost:8034/v1/users/81ebc4ca-2496-4cc3-b533-311f237b6fe9/crop_details/10/summary": Read timed out; nested exception is java.net.SocketTimeoutException: Read timed out
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:666)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:621)
at org.springframework.web.client.RestTemplate.getForEntity(RestTemplate.java:321)
at ...
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1535)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:84)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:652)
... 91 common frames omitted
Scenario - When I am trying to load testing using JMeter for Rest API from one service(running on 8033 port). It internally calls using RestTemplate to other service rest API(above one on 8034). For few request it works fine but as I go beyond just 10 concurrent requests it start giving above error.
What I tried till now
I already tried using Different ClientManagers to set both connection and read time out values.
Also tried using PoolConnectionMangers.
Custom Keep Alive Strategy.
Setting JDK's values for above time out properties.
But Still getting same problem.
Links referred
http://www.baeldung.com/httpclient-connection-management
http://www.baeldung.com/rest-template
http:/blog.technogemsinc.com/2010/08/java-url-connection-timeout-default.html
Same thing works on my local machine for 10000 request without any problem.
Box/Server details - 3.7 GB/50GB single core CPU
We are using RabbitMQ 3.3.1, spring-amqp-1.3.1, spring-rabbit-1.3.1 in our project and we are getting following exception. Our environment is in Azure cloud and we are seeing that the RabbitMQ server is losing the connection with the consumers more often. Once we restart our OSGI consumer bundle, it works for some time and then same issue occurs. Any help/tips are appreciated.
[SimpleAsyncTaskExecutor2]|WARN|org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer|370-spring-rabbit-1.3.1.RELEASE|Consumer
raised exception, processing can restart if the connection factory
supports it com.rabbitmq.client.ShutdownSignalException: connection
error; reason: java.net.SocketException: Connection reset at
com.rabbitmq.client.impl.AMQConnection.startShutdown(AMQConnection.java:715)[367:com.rabbitmq.client:3.2.4]
at
com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:705)[367:com.rabbitmq.client:3.2.4]
at
com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:568)[367:com.rabbitmq.client:3.2.4]
Caused by: java.net.SocketException: Connection reset at
java.net.SocketInputStream.read(SocketInputStream.java:196)[:1.7.0_51]
at
java.net.SocketInputStream.read(SocketInputStream.java:122)[:1.7.0_51]
at
sun.security.ssl.InputRecord.readFully(InputRecord.java:442)[:1.7.0_51]
at sun.security.ssl.InputRecord.read(InputRecord.java:480)[:1.7.0_51]
at
sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)[:1.7.0_51]
at
sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)[:1.7.0_51]
at
sun.security.ssl.AppInputStream.read(AppInputStream.java:102)[:1.7.0_51]
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)[:1.7.0_51]
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)[:1.7.0_51]
at
java.io.DataInputStream.readUnsignedByte(DataInputStream.java:288)[:1.7.0_51]
at
com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95)[367:com.rabbitmq.client:3.2.4]
at
com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:131[367:com.rabbitmq.client:3.2.4]
at
com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:533[367:com.rabbitmq.client:3.2.4]
We have a Spring MVC web app (WAR) deploying to Tomcat (6.0.35) that launches a thread inside a separate JVM at deploy time (don't ask why - not my design) and then communicates with that thread via RMI over port 8888.
Despite being totally convoluded, this was working perfectly fine up until yesterday, and now the thread is failing at startup and despite our best efforts to add logging into the mix, we are hitting a wall. This is the only exception we are able to find in the logs:
Jun 12, 2012 3:11:36 AM com.ourapp.ImageController destroy
SEVERE: Shutdown Error: Lookup of RMI stub failed; nested exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused
Jun 12, 2012 3:11:37 AM org.apache.catalina.core.StandardContext listenerStop
SEVERE: Exception sending context destroyed event to listener instance of class org.springframework.web.context.ContextLoaderListener
java.lang.NoClassDefFoundError: org/springframework/web/context/ContextCleanupListener
at org.springframework.web.context.ContextLoaderListener.contextDestroyed(ContextLoaderListener.java:80)
at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:3973)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4577)
at org.apache.catalina.startup.HostConfig.checkResources(HostConfig.java:1165)
at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1271)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:296)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1337)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1601)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1610)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1590)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException: org.springframework.web.context.ContextCleanupListener
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1387)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1233)
... 12 more
The ImageController is the Spring MVC Controller that is responsible for kicking off this daemon/spawned RMI thread. Based on the verbage of this error, does anybody have any idea what might be causing this "connection refused" error?
Running a netstat -an | grep 8888 (this is a Linux machine) produces no output which means nothing is listening on that port. Thanks in advance for any ideas/suggestions that lead to a fix.
Edit: Here's another ConnectionException we're seeing:
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at java.net.Socket.<init>(Socket.java:375)
at java.net.Socket.<init>(Socket.java:189)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
... 74 more
I'll move my comments to this answer.
Everything in your logs suggests that the problem is on your other machine where you connect to on port 8888. The netstat results prove that there's no process listening on that port - hence connection errors in your logs.
Next step I would take is check whether the application on the other machine (or in other JVM) is running at all (e.g. ps ax|grep java). And if it does - check its logs for clues why it failed to start the RMI service, or if it doesn't - start it.
Typically if an application can not open a socket, it's due to the fact that some other application is already using the port. It's not the case in your situation. It could be a case that two instances of the same application have been started: first application took 8888 port, second application couldn't take 8888 and failed, and then first application was shut down. After all you end up with nobody listening on the port.
does anybody have any idea what might be causing this "connection refused" error?
It's right there in the stack trace:
java.lang.NoClassDefFoundError: org/springframework/web/context/ContextCleanupListener
Your deployment is missing that class, i.e. the JAR file that it comes in.