Short description of the issue:
I'm using ActiveMQ 5.10.2 (AMQ) in a project where I have Application A sending data over to Application B via amq and saw this exception :
javax.jms.JMSException: Could not post command: KeepAliveInfo
Here's the full stack trace:
ERROR JMSConsumer:148 - Consumer Exception
javax.jms.JMSException: Could not post command: KeepAliveInfo {} due to: java.net.SocketTimeoutException: Read timed out
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:54)
...
Caused by: java.io.IOException: Could not post command: KeepAliveInfo {} due to: java.net.SocketTimeoutException: Read timed out
at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:33)
at org.apache.activemq.transport.http.HttpClientTransport.oneway(HttpClientTransport.java:138)
...
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
That exception was seen in App B and due to this exception it could not connect to the AMQ. Restarting the app allowed it to talk to the AMQ again.
Does anyone know how I can reproduce that exception so I can do proper error handling/recovery?
More details about the issue:
As mentioned above, I'm trying to recreate the exception to do proper error handling/recovery. But so far I'm unable to reproduce it.
I've looked online and don't see too much on this specific exception nor on the keepaliveinfo message other than from activemq's own site.
Per its link https://activemq.apache.org/activemq-inactivitymonitor if the useInactivityMonitor parameter is set to true, which is the default value, then the apps connecting to that AMQ on the same topic/queue will have that keepaliveinfo message sent if it doesn't detect any normal traffic. In my environment that parameter is set to true
My thought is, with this value set to true, something happened in my environment (perhaps the AMQ got in a funky state or was low in memory) that it failed to handle a keepaliveinfo message sent from my apps and then that jms exception occurred.
In trying to reproduce that exception I have done the following tests to no avail:
Test 1:
Start App A, App B and the AMQ. After everything is up and running and data is flowing I would then stop the AMQ broker.
I got this exception :
javax.jms.JMSException: Failed to perform GET on: <my amq ip>:443 Reason: Connection refused: connect
Which isn't the exception I noticed before.
Test 2:
I would start App A, App B, and the AMQ, and then after everything is up and running I would then stop App A (which sends data to App B via AMQ).
I did not witness any keepaliveinfo messages getting sent and more importantly I don't see the specific jms exception above.
Test 3:
I started App B and the AMQ. I modified App A to connect to the AMQ but commented out the code that sends data to App B on the queue/topic.
Again I did not witness this message JMSException: could not post command: keepaliveinfo
Test 4
I modified the AMQ broker activemq.xml file to have its <memoryUsage> <storeUsage> and <tempUsage> values to be really low. In a way to simulate that the ActiveMQ is super busy and can't handle more data being sent. More info on those parameters are mentioned here https://activemq.apache.org/producer-flow-control.html
After restarting AMQ and App A and App B, I still didn't witness that exception message.
End of tests
So again, does anyone know what is the setup/conditions on reproducing this issue?
Extra note:
It seems like I can avoid this exception altogether if I set the useInactivityMonitor parameter to false. With that set then no keepaliveinfo messages are sent if no normal traffic occurs over a connection. But I don't want to change it to false.
And here is my pom file defining the amq artifact id's:
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-all</artifactId>
<version>5.10.2</version>
</dependency>
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-http</artifactId>
<version>5.10.2</version>
</dependency>
Related
I am testing out torchserve with its default configuration... its inference api listen to port 8080. and I got this error which means the port is in use. my bad, I still have apache process listening to that port. I killed the apache process. But i still get the same error. Tried again by changing port configuration of torchserve to use port that I know for sure is not being used by any process. still got the same error. with TCPView app, I could see that for a second, the configured port has been opened in LISTENING state for the torcheserve process. But soon after it got removed again.
so anyone has anymore idea why is this happening? I am pretty sure this is no longer an issue of port conflict
the stack trace
java.io.IOException: Failed to bind
at io.grpc.netty.shaded.io.grpc.netty.NettyServer.start(NettyServer.java:264)
at io.grpc.internal.ServerImpl.start(ServerImpl.java:183)
at io.grpc.internal.ServerImpl.start(ServerImpl.java:90)
at org.pytorch.serve.ModelServer.startGRPCServer(ModelServer.java:396)
at org.pytorch.serve.ModelServer.startGRPCServers(ModelServer.java:377)
at org.pytorch.serve.ModelServer.startAndWait(ModelServer.java:116)
at org.pytorch.serve.ModelServer.main(ModelServer.java:95)
Caused by: java.net.BindException: Address already in use: bind
at java.base/sun.nio.ch.Net.bind0(Native Method)
at java.base/sun.nio.ch.Net.bind(Net.java:461)
at java.base/sun.nio.ch.Net.bind(Net.java:453)
at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227)
at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:134)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:550)
at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491)
at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:248)
at io.grpc.netty.shaded.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356)
at io.grpc.netty.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
Exception in thread "Thread-0" java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926)
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353)
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346)
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828)
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818)
at io.netty.channel.AbstractChannelHandlerContext.safeExecute(AbstractChannelHandlerContext.java:989)
at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608)
at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:472)
at io.netty.channel.DefaultChannelPipeline.close(DefaultChannelPipeline.java:957)
at io.netty.channel.AbstractChannel.close(AbstractChannel.java:232)
at org.pytorch.serve.ModelServer.stop(ModelServer.java:473)
at org.pytorch.serve.ModelServer$1.run(ModelServer.java:91)
I had the same error and resolved it by adding the following lines to the config.properties file.
grpc_inference_port=7000
grpc_management_port=7001
I'm working on a apache pulsar cluster and it was working fine, but for some reason I started getting the following message when I try to send a message:
ERROR [id: 0xc5de4911, L:/127.0.0.1:55672 - R:localhost/127.0.0.1:6650] Close connection becaues received internal-server error java.lang.IllegalArgumentException: bound must be positive
Also i'm getting the following error:
persistent://global/mycluster-cluster1/ns1/topic-partition-0] [null] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException$LookupException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /loadbalance/brokers/ip-11-55-123-80.ec2.internal:8080
I know the topic is created because i doble checked it using the pulsar-admin cli. I'm lost on where to start looking anyone could point me out.
from the error message, it seems to be a race condition when Pulsar splits a namespace bundle. What version of Pulsar are you using?
After general server configuration i'm able to log in to carbon, store or publisher without any issue with default admin login.
Unfortunately when i sign in EMM: https://myserver:port/emm -> sso/login
i end up in /emm/acs - "500: Something has gone wrong" page.
Debug: 2 errors in live terminal:
*[2015-06-12 20:35:21,254] ERROR {org.jaggeryjs.hostobjects.xhr.XMLHttpRequestHostObject} - Connection refused java.net.ConnectException: Connection refused*
*[2015-06-12 20:35:21,261] ERROR {org.jaggeryjs.jaggery.core.manager.WebAppManager} - org.mozilla.javascript.WrappedException: Wrapped org.jaggeryjs.scriptengine.exceptions.ScriptException: java.net.ConnectException: Connection refused (http#220)
org.jaggeryjs.scriptengine.exceptions.ScriptException: org.mozilla.javascript.WrappedException: Wrapped org.jaggeryjs.scriptengine.exceptions.ScriptException: java.net.ConnectException: Connection refused (http#220)*
And i got another 2 errors after wso2server startup:
*[2015-06-12 21:44:34,231] ERROR {org.wso2.carbon.apimgt.keymgt.internal.APIKeyMgtServiceComponent} - Error in initializing thrift transport
org.apache.thrift.transport.TTransportException: Could not bind to port 10397*
*[2015-06-12 21:44:34,247] ERROR {org.wso2.carbon.apimgt.keymgt.internal.APIKeyMgtServiceComponent} - Failed to initialize key management service.
java.lang.Exception: Error in initializing thrift transport*
Any hints?
Regards,
Lukas
I encountered with the /emm/acs redirection issue, actually I don't know why it has been happening, but when it happens, I do a refresh on the page, everything works fine.
The second issue you have mentioned, is nothing to do with the first one, it says that Apache Thrift Transport layer could not bind the port. Since EMM does not require Thrift, you can simply disable by editing the following XML configuration files.
Change true to false for the following config files, under the target folder.
./repository/conf/api-manager.xml:280: <EnableThriftServer>true</EnableThriftServer>
./repository/conf/identity.xml:218: <EnableThriftService>true</EnableThriftService>
I am using nexus 2.10 having recently upgraded from nexus 2.8. The reason I upgraded was because I was getting the following exception in my logs and thus the jar was not downloaded and Nexus 2.10 gives a few more details in the log regarding the broken pipe error.
So in my logs I get the following error (replaced i[p address with string host name).
2014-10-21 15:18:22,556+0100 INFO [qtp-855494215-66] anonymous org.sonatype.nexus.proxy.storage.remote.httpclient.HttpClientRemoteStorage - Initializing remote transport for proxy repository "Central" [id=central]...
2014-10-21 15:23:59,440+0100 WARN [qtp-855494215-69] anonymous org.sonatype.nexus.content.internal.ContentServlet - org.eclipse.jetty.io.EofException, caused by: java.io.IOException: Broken pipe [client=host.name,ua=m2e/3.9.1.v20130814-1242/1.4.0.20130601-0317/1.6.0_45,req=GET http://xx.xx:8081/nexus/content/groups/public/org/springframework/spring-jdbc/4.1.1.RELEASE/spring-jdbc-4.1.1.RELEASE.jar]
2014-10-21 15:24:43,329+0100 WARN [qtp-855494215-67] anonymous org.sonatype.nexus.content.internal.ContentServlet - org.eclipse.jetty.io.EofException, caused by: java.io.IOException: Broken pipe [client=host.name,ua=m2e/3.9.1.v20130814-1242/1.4.0.20130601-0317/1.6.0_45,req=GET http://xx.xx5:8081/nexus/content/groups/public/org/springframework/spring-tx/4.1.1.RELEASE/spring-tx-4.1.1.RELEASE.jar]
2014-10-21 15:24:58,333+0100 WARN [qtp-855494215-66] anonymous org.sonatype.nexus.content.internal.ContentServlet - org.eclipse.jetty.io.EofException, caused by: java.io.IOException: Broken pipe [client=host.name,ua=m2e/3.9.1.v20130814-1242/1.4.0.20130601-0317/1.6.0_45,req=GET http://xx.xx:8081/nexus/content/groups/public/org/springframework/spring-context/4.1.1.RELEASE/spring-context-4.1.1.RELEASE-sources.jar]
Can anyone advise how to get past this error because currently some jars are not being downloaded. This does not apply to all jars for eg i just downloaded the nekohtml-0.9.5.jar no issues. Could this be possibly something to do with size? In my maven repository I just get a spring-test-4.1.1.RELEASE.jar.lastUpdated jar which is only 1kb in size.
Also I am running behind a firewall so have a proxy server configured in the nexus settings.
UPDATE:
this seems to be an issue only when done through eclipse. I was able to download sources and jars etc fine when I did it from the command line. I am using Eclipse keplar and the m2e plugin. Can anyone comment on this?
All advice appreciated
Thanks
The broken pipe exception occurs when nexus has received an http request from a client, prepares a response, but when it goes to write the response it finds the socket has already been closed.
Likely causes of this are:
An end user initiating a request and then canceling it (or
navigating away from a web page before it completes)
A client which does not properly close socket connections
A client's read timeout setting being reached
A reverse proxy in between nexus and the client (such as apache or
nginx) terminating the connection
Therefore you'll need to begin your investigation on the client side. If it turns out the client or reverse proxy's read timeout setting is triggering this then you'll need to investigate why nexus is responding slowly.
Your Nexus server is probably not able to reach some external repositories if it is running inside a network behind a proxy. You have to make sure you have the http proxy configured in Nexus and that any URL's of remote proxy repositories you want to access are allowed in the proxy.
You can use the "Browse Remote" feature to check if the access works. More details are in the Nexus book http://books.sonatype.com/nexus-book/reference/configxn-sect-customizing-server.html
Getting below error with MQ(Message Queue), how to resolve this?
Message : com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ1107: A problem with
this connection has occurred. An error has occurred with the WebSphere MQ JMS
connection. Use the linked exception to determine the cause of this error.
Caused by [1] -->
Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with
compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').
Caused by [3] -->
Message : com.ibm.mq.jmqi.JmqiException: CC=2;RC=2009;AMQ9208: Error on receive
from host '/xx.xx.xx.xx:xxxx
(emb701t7.dci.bt.com)'. [1=-1,2=ffffffff,3=/xx.xx.xx.xx:xxxx
(emb701t7.dci.bt.com),4=TCP]
Please take me forward.
I faced the same error and its resolved now!
As Magic Wand mentioned in the comment, below error comes up if you have a lower version of MQ jar i.e if your MQ jar version is 7.0.1.5 or below
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2009;AMQ9208: Error on receive from host
This bug has been fixed in IBM MQ version 7.0.1.6 or higher as mentioned here: http://www-01.ibm.com/support/docview.wss?uid=swg1IV00348
Add the newer version of below ibm.mq jars and make sure you have the additional jars too mentioned below:
com.ibm.mq.commonservices.jar
com.ibm.mq.jar
com.ibm.mq.jmqi.jar
com.ibm.mqbind.jar
com.ibm.mqjms.jar
fscontext.jar
jms-1.1-20020430.jar
providerutil.jar
The TCP connection between the MQ client running in weblogic and the MQ Queue Manager channel process is being broken unexpectedly (that is what the return code 2009 means). This could be down to a number of issues such as the queue manager being killed, the channel process exiting for some reason, a firewall killing the connection etc. Have a look on the MQ queue manager error logs for issues with the channel process at the same time. Take a note of how often the error message occurs i.e. is the first error 2 hours after the weblogic server starts.
You could take a network trace to see how the TCP connection is being closed and by who.