Getting below error with MQ(Message Queue), how to resolve this?
Message : com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ1107: A problem with
this connection has occurred. An error has occurred with the WebSphere MQ JMS
connection. Use the linked exception to determine the cause of this error.
Caused by [1] -->
Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with
compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').
Caused by [3] -->
Message : com.ibm.mq.jmqi.JmqiException: CC=2;RC=2009;AMQ9208: Error on receive
from host '/xx.xx.xx.xx:xxxx
(emb701t7.dci.bt.com)'. [1=-1,2=ffffffff,3=/xx.xx.xx.xx:xxxx
(emb701t7.dci.bt.com),4=TCP]
Please take me forward.
I faced the same error and its resolved now!
As Magic Wand mentioned in the comment, below error comes up if you have a lower version of MQ jar i.e if your MQ jar version is 7.0.1.5 or below
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2009;AMQ9208: Error on receive from host
This bug has been fixed in IBM MQ version 7.0.1.6 or higher as mentioned here: http://www-01.ibm.com/support/docview.wss?uid=swg1IV00348
Add the newer version of below ibm.mq jars and make sure you have the additional jars too mentioned below:
com.ibm.mq.commonservices.jar
com.ibm.mq.jar
com.ibm.mq.jmqi.jar
com.ibm.mqbind.jar
com.ibm.mqjms.jar
fscontext.jar
jms-1.1-20020430.jar
providerutil.jar
The TCP connection between the MQ client running in weblogic and the MQ Queue Manager channel process is being broken unexpectedly (that is what the return code 2009 means). This could be down to a number of issues such as the queue manager being killed, the channel process exiting for some reason, a firewall killing the connection etc. Have a look on the MQ queue manager error logs for issues with the channel process at the same time. Take a note of how often the error message occurs i.e. is the first error 2 hours after the weblogic server starts.
You could take a network trace to see how the TCP connection is being closed and by who.
Related
We use IBM MQ for some JMS integration between some of our micro services. Because IBM MQ docker image does not run on Mac book laptops with Apple M1 processor we configured our applications requiring JMS integration to be able to run against ActiveMQ broker as well.
We use the latest version of ActiveMQ classic which is 5.17.3 and we run it as a docker container.
For most of us who tried to run against ActiveMQ everything works OK but we have a colleague that is unable to start the application because it fails to connect to the ActiveMQ broker. All we can see in the application logs is this:
Caused by: javax.jms.JMSException: Cannot send, channel has already failed: tcp://127.0.0.1:61616
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:80)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1423)
at org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1488)
at org.apache.activemq.ActiveMQXAConnection.createSession(ActiveMQXAConnection.java:74)
at org.apache.activemq.ActiveMQXAConnection.createXASession(ActiveMQXAConnection.java:61)
at com.atomikos.datasource.xa.jms.JmsTransactionalResource.refreshXAConnection(JmsTransactionalResource.java:74)
... 10 common frames omitted
Caused by: org.apache.activemq.transport.InactivityIOException: Cannot send, channel has already failed: tcp://127.0.0.1:61616
at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:329)
at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:318)
at org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:94)
at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:116)
at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1394)
... 14 common frames omitted
Googling for org.apache.activemq.transport.InactivityIOException: Cannot send, channel has already failed brought us some old internet posts but none of them brought any light.
Part of the ActiveMQ docker logs we can see below log entry after broker is started which makes us think the broker is started OK. Also he can access the ActiveMQ admin console when hitting http://localhost:8161/admin URL
INFO | Listening for connections at : tcp://somelaptopid:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
Short description of the issue:
I'm using ActiveMQ 5.10.2 (AMQ) in a project where I have Application A sending data over to Application B via amq and saw this exception :
javax.jms.JMSException: Could not post command: KeepAliveInfo
Here's the full stack trace:
ERROR JMSConsumer:148 - Consumer Exception
javax.jms.JMSException: Could not post command: KeepAliveInfo {} due to: java.net.SocketTimeoutException: Read timed out
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:54)
...
Caused by: java.io.IOException: Could not post command: KeepAliveInfo {} due to: java.net.SocketTimeoutException: Read timed out
at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:33)
at org.apache.activemq.transport.http.HttpClientTransport.oneway(HttpClientTransport.java:138)
...
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
That exception was seen in App B and due to this exception it could not connect to the AMQ. Restarting the app allowed it to talk to the AMQ again.
Does anyone know how I can reproduce that exception so I can do proper error handling/recovery?
More details about the issue:
As mentioned above, I'm trying to recreate the exception to do proper error handling/recovery. But so far I'm unable to reproduce it.
I've looked online and don't see too much on this specific exception nor on the keepaliveinfo message other than from activemq's own site.
Per its link https://activemq.apache.org/activemq-inactivitymonitor if the useInactivityMonitor parameter is set to true, which is the default value, then the apps connecting to that AMQ on the same topic/queue will have that keepaliveinfo message sent if it doesn't detect any normal traffic. In my environment that parameter is set to true
My thought is, with this value set to true, something happened in my environment (perhaps the AMQ got in a funky state or was low in memory) that it failed to handle a keepaliveinfo message sent from my apps and then that jms exception occurred.
In trying to reproduce that exception I have done the following tests to no avail:
Test 1:
Start App A, App B and the AMQ. After everything is up and running and data is flowing I would then stop the AMQ broker.
I got this exception :
javax.jms.JMSException: Failed to perform GET on: <my amq ip>:443 Reason: Connection refused: connect
Which isn't the exception I noticed before.
Test 2:
I would start App A, App B, and the AMQ, and then after everything is up and running I would then stop App A (which sends data to App B via AMQ).
I did not witness any keepaliveinfo messages getting sent and more importantly I don't see the specific jms exception above.
Test 3:
I started App B and the AMQ. I modified App A to connect to the AMQ but commented out the code that sends data to App B on the queue/topic.
Again I did not witness this message JMSException: could not post command: keepaliveinfo
Test 4
I modified the AMQ broker activemq.xml file to have its <memoryUsage> <storeUsage> and <tempUsage> values to be really low. In a way to simulate that the ActiveMQ is super busy and can't handle more data being sent. More info on those parameters are mentioned here https://activemq.apache.org/producer-flow-control.html
After restarting AMQ and App A and App B, I still didn't witness that exception message.
End of tests
So again, does anyone know what is the setup/conditions on reproducing this issue?
Extra note:
It seems like I can avoid this exception altogether if I set the useInactivityMonitor parameter to false. With that set then no keepaliveinfo messages are sent if no normal traffic occurs over a connection. But I don't want to change it to false.
And here is my pom file defining the amq artifact id's:
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-all</artifactId>
<version>5.10.2</version>
</dependency>
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-http</artifactId>
<version>5.10.2</version>
</dependency>
I am using IBM MQ v7.5 with JBOSS EAP 6.4 with JCA resource adapter and MDB.
MQ server is running in HP NonStop Integrity Server v5.3.1.12
The application is working correctly. But I am seeing the following exception in my log
[com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016027: Local XARecoveryModule.xaRecovery got XA exception XAException.XAER_RMFAIL: javax.transaction.xa.XAException
at com.ibm.mq.connector.RecoveryXAResource.checkExceptions(RecoveryXAResource.java:147)
at com.ibm.mq.connector.RecoveryXAResource.recover(RecoveryXAResource.java:514)
at org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.recover(XAResourceWrapperImpl.java:185)
at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoveryFirstPass(XARecoveryModule.java:541) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkFirstPass(XARecoveryModule.java:181) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:747) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:375) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1] Caused by: com.ibm.msg.client.wmq.v6.jms.internal.ConfigEnvironment$1: MQJMS1068: failed to obtain XAResource.
at com.ibm.msg.client.wmq.v6.jms.internal.ConfigEnvironment.newException(ConfigEnvironment.java:379)
at com.ibm.msg.client.wmq.v6.jms.internal.MQXAConnection.createXASession(MQXAConnection.java:155)
at com.ibm.msg.client.jms.internal.JmsXAConnectionImpl.createXASession(JmsXAConnectionImpl.java:125)
at com.ibm.mq.jms.MQXAConnection.createXASession(MQXAConnection.java:88)
at com.ibm.mq.connector.RecoveryXAResource.recover(RecoveryXAResource.java:490)
... 5 more Caused by: javax.transaction.xa.XAException: The method 'xa_open' has failed with errorCode '-3'.
at com.ibm.mq.jmqi.JmqiXAResource.<init>(JmqiXAResource.java:274)
at com.ibm.mq.jmqi.JmqiXAResource.getInstance(JmqiXAResource.java:122)
at com.ibm.mq.jmqi.JmqiEnvironment.newJmqiXAResource(JmqiEnvironment.java:1598)
at com.ibm.msg.client.wmq.v6.base.internal.MQXAQueueManager.getXAResource(MQXAQueueManager.java:175)
at com.ibm.msg.client.wmq.v6.jms.internal.MQXAConnection.createXASession(MQXAConnection.java:134)
... 8 more
Can anyone help my understanding why this is happening?
Also, even if I stop my server, the connection to the MQ server was not killed. It remains as an orphan. Is it somehow related to this exception?
I am using NoTransaction in resource adapter configuration in standalone.xml
The XA error code -3 is XAER_RMERR - which means that a problem occurred with the resource manager, in this case MQ.
Have you looked at the queue manager's error logs?
I missed an important point in this question (due to lack of knowledge in IBM MQ).That is, my MQ server is running in HP NonStop Integrity Server. These servers doesn't support XA connection. Hence this error.
Now, the question is how to stop it.
I have stopped the periodic recovery in JBOSS with the following properties:
name="RecoveryEnvironmentBean.periodicRecoveryPeriod" value="0"
name="RecoveryEnvironmentBean.recoveryBackoffPeriod" value="0"
This is a known issue. IBM has issued fixpacks to address this issue. Please see:
http://www-01.ibm.com/support/docview.wss?uid=swg1IC97579
IBM MQ versions 7.5.0.4 and later have this fix.
And, yes, your orphaned connection is a result of this issue.
I try to send TextMessage via JMS from JBoss 7 client on remote queue which is located on JBoss 4 server.
When server receive the message, exception is thrown:
2012-08-24 12:06:26,988 ERROR [ServerThread] Worker thread initialization failure
java.io.IOException: Can not read data for version 0. Supported versions: 1,2,22
at org.jboss.remoting.transport.socket.ServerThread.versionedRead(ServerThread.java:712)
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:409)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:176)
So I added -Djboss.remoting.pre_2_0_compatible=true to jvm arguments but now I got another exception:
java.lang.IllegalArgumentException: Invalid packet type: 17
at org.jboss.jms.wireformat.PacketSupport.createPacket(PacketSupport.java:379)
at org.jboss.jms.wireformat.JMSWireFormat.read(JMSWireFormat.java:290)
at org.jboss.remoting.transport.socket.ServerThread.versionedRead(ServerThread.java:704)
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:409)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:176)
Could anyone help me how to send message via JMS from JBoss 7 to JBoss 4 server?
I configured a queue and a QueueConnectionFactory using the V5 Messaging provider in WebSphere Application Server 6.1.
I also have a ListenerPort configured for this destination.
It throws an exception when I try to connect to this queue using the queue connection factory or when the Message Listener listening to this ListenerPort starts up.
javax.jms.JMSException: MQJMS2005: failed to create MQQueueManager
for 'localhost:WAS_Node01_server1'
and the nested exception is
com.ibm.mq.MQException: MQJE001: An MQException occurred: Completion Code 2, Reason 2059
MQJE011: Socket connection attempt refused
Can someone please point me to what possibily could have gone wrong in configuring the queue and queue connection factory?
The error indicates that the WebSphere MQ transport is being used. Per the clarification in your comments, no queue manager is present so a 2059 is expected. It will be necessary to configure a WAS messaging Engine to provide the native transport or to point to a queue manager.