We Keep Coding

Redistribution fails for large messages in ActiveMQ Artemis

I'm using ActiveMQ Artemis version 2.19.1, and I'm facing an issue in a 6-node (3 masters) cluster where redistribution is failing for large messages with the below warning logs:
23:35:05,551 WARN [org.apache.activemq.artemis.core.server] AMQ222303: Redistribution by Redistributor[TEST_QUEUE/2244] of messageID = 196,950,715 failed: java.lang.UnsupportedOperationException: Method not supported with Large Messages
at org.apache.activemq.artemis.protocol.amqp.broker.AMQPLargeMessage.getData(AMQPLargeMessage.java:311) [artemis-amqp-protocol-2.19.1.jar:2.19.1]
at org.apache.activemq.artemis.protocol.amqp.broker.AMQPMessage.anyMessageAnnotations(AMQPMessage.java:1374) [artemis-amqp-protocol-2.19.1.jar:2.19.1]
at org.apache.activemq.artemis.protocol.amqp.broker.AMQPMessage.hasScheduledDeliveryTime(AMQPMessage.java:1352) [artemis-amqp-protocol-2.19.1.jar:2.19.1]
at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.processRoute(PostOfficeImpl.java:1499) [artemis-server-2.19.1.jar:2.19.1]
at org.apache.activemq.artemis.core.server.cluster.impl.Redistributor$1.run(Redistributor.java:169) [artemis-server-2.19.1.jar:2.19.1]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.19.1.jar:2.19.1]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.19.1.jar:2.19.1]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) [artemis-commons-2.19.1.jar:2.19.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_322]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_322]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.19.1.jar:2.19.1]
Later I see broker is removing consumers with below warning since properties=null:
00:10:28,280 WARN [org.apache.activemq.artemis.core.server] AMQ222151: removing consumer which did not handle a message, consumer=ServerConsumerImpl [id=57, filter=null, binding=LocalQueueBinding [address=TEST_QUEUE, queue=QueueImpl[name=TEST_QUEUE, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=localhost], temp=false]#9c000b7, filter=null, name=TEST_QUEUE, clusterName=TEST_QUEUE1e359f55-c92b-11ec-b908-005056a3af3f]], message=Reference[157995028]:RELIABLE:AMQPLargeMessage( [durable=true, messageID=157995028, address=TEST_QUEUE, size=0, scanningStatus=SCANNED, applicationProperties={VER=05, trackingId=62701757c80c3004d037ded6}, messageAnnotations={}, properties=null, extraProperties = TypedProperties[_AMQ_AD=TEST_QUEUE]]: java.lang.IllegalArgumentException: Array must not be empty or null
at org.apache.qpid.proton.codec.CompositeReadableBuffer.append(CompositeReadableBuffer.java:688) [proton-j-0.33.10.jar:]
at org.apache.qpid.proton.engine.impl.DeliveryImpl.send(DeliveryImpl.java:345) [proton-j-0.33.10.jar:]
at org.apache.qpid.proton.engine.impl.SenderImpl.send(SenderImpl.java:74) [proton-j-0.33.10.jar:]
at org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext$LargeMessageDeliveryContext.deliverInitialPacket(ProtonServerSenderContext.java:686) [artemis-amqp-protocol-2.19.1.jar:2.19.1]
at org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext$LargeMessageDeliveryContext.deliver(ProtonServerSenderContext.java:587) [artemis-amqp-protocol-2.19.1.jar:2.19.1]
I have 6 consumers for this queue if one message out of many (let's say 1,000) is large - it should process other messages but, the processing is stopped completely with 0 consumers on the queue.

When you send large messages, within the first block sent the large message is parsed for properties. Most likely you are using large properties in a way the server is not being able to parse the first few bytes of the message. You should avoid using large properties and let the large portion only for the client.
We tried following up with many tests on this JIRA, and the only plausible scenarios would be through large properties, or some incomplete message your client is generating in a way the server can't parse it.
https://issues.apache.org/jira/browse/ARTEMIS-3837
if you provide a reproducer showing how the property is failing to be parsed we will follow up with a possible fix.
Please move any discussion regarding this to the JIRA. It could be a bug caused by your anti-pattern.

Why errorChannel is not getting called in case of MQException from int-jms:outbound-channel-adapter?

Jms outbound-channel-adapter works perfectly fine but I see intermittently this error in the log, however, MQ message still gets delivered.
2019-06-07 10:16:22 [JMSCCThreadPoolWorker-5] INFO o.s.j.c.CachingConnectionFactory - Encountered a JMSException - resetting the underlying JMS Connection
com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ1107: A problem with this connection has occurred.
at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:578)
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:214)
at com.ibm.msg.client.wmq.internal.WMQConnection.consumer(WMQConnection.java:794)
at com.ibm.mq.jmqi.remote.api.RemoteHconn.callEventHandler(RemoteHconn.java:2903)
at com.ibm.mq.jmqi.remote.api.RemoteHconn.driveEventsEH(RemoteHconn.java:628)
at com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.processHconn(RemoteDispatchThread.java:691)
at com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.run(RemoteDispatchThread.java:233)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:263)
at com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:99)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:284)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:312)
at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1214)
Caused by: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').
...This is errorChannel configuration:
<int:header-enricher id="errorMsg.HeaderEnricher"
input-channel="errorChannel"
output-channel="omniAlertsJmsErrorChannel">
...and jms outbound-channel-adapter configured:
<int-jms:outbound-channel-adapter
id="jmsOutToNE" channel="umpAlertNotificationJMSChannel"
destination="senderTopic"
jms-template="jmsQueueTemplate"
>
I am expecting omniAlertsJmsErrorChannel to receive the MessageHandlingException which is not happening from jmsOutToNE adapter. All other channel/flow errors are being routed to omniAlertsJmsErrorChannel.
Also, wondering if there jms outbound-channel-adapter is retrying internally when com.ibm.mq.MQException happens and it gets successful in the subsequent try?

The errorChannel is out of use in any MessageHandler. Their exception are just thrown to the caller. To handle those exception you need to consider adding a request-handler-advice-chain with the ExpressionEvaluatingRequestHandlerAdvice. Or if we talk about a retry, you also can add to that chain a RequestHandlerRetryAdvice.
See more info in the Reference Manual.
Not sure though why your message is still delivered to the MQ. There is no any retry out-of-the-box in the <int-jms:outbound-channel-adapter. It might a behavior of the MQ Connection Factory adapter in the IBM library.

Why does Camel noErrorHandler log warning message with stack trace?

We are using Camel 2.15.4 to connect our Swing clients to our OSGi server.
In the server, under certain circumstances, we throw a sub-class of RuntimeException which we want to propagate back to the client.
In the building of our routes we are using:
errorHandler(noErrorHandler());
But every time we throw one of these server errors, we get a WARN log message with a full stack trace. The testers are finding these log messages rather unnerving as they look like problems.
Is there any way to switch off the logging of these warnings?
Here's an example:
16:37:12,565 | WARN | che.camel.camel-core | Execution of JMS message listener failed. Caused by: [org.apache.camel.RuntimeCamelException]
org.apache.camel.RuntimeCamelException: xxxxxx.yyyyyyy.exceptions.CustomException: blah
at org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1619)
at org.apache.camel.component.bean.BeanInvocation.invoke(BeanInvocation.java:87)
at org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:134)
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
at org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:68)
at org.apache.camel.component.bean.BeanProducer.process(BeanProducer.java:38)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:129)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:91)
at uniworks.camel.interceptor.CamelInterceptor$1.process(CamelInterceptor.java:79)
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:91)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:448)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:87)
at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:103)[58:org.apache.camel.camel-jms:2.15.4]
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:569)[176:org.apache.servicemix.bundles.spring-jms:3.2.14.RELEASE_1]
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:507)[176:org.apache.servicemix.bundles.spring-jms:3.2.14.RELEASE_1]
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:474)[176:org.apache.servicemix.bundles.spring-jms:3.2.14.RELEASE_1]
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)[176:org.apache.servicemix.bundles.spring-jms:3.2.14.RELEASE_1]
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)[176:org.apache.servicemix.bundles.spring-jms:3.2.14.RELEASE_1]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1103)[176:org.apache.servicemix.bundles.spring-jms:3.2.14.RELEASE_1]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1095)[176:org.apache.servicemix.bundles.spring-jms:3.2.14.RELEASE_1]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:992)[176:org.apache.servicemix.bundles.spring-jms:3.2.14.RELEASE_1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_92]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_92]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_92]
In the previous version of our server, which used Camel 2.11.0 and Spring DSL, we defined our route contexts with:
<camel:errorHandler id="noErrorHandler" type="NoErrorHandler"/>
<camel:camelContext id="blah_camel" errorHandlerRef="noErrorHandler">
.....
and we did not get these warning log messages.

That is actually not Camel's no error handler doing this logging but the JMS component. The JMS component has some fallback logging of exceptions to indicate the JMS message was not processeed successfully. You can see the two options: errorHandlerLoggingLevel and errorHandlerLogStackTrace on the JMS component/endpoint which you can turn off/configure: http://camel.apache.org/jms

Parsing Java Server Exception logs

I have a requirement wherein I have to read all "history" exception logs from a WebSphere server and load them in Hive.
Below is what a typical log looks like but message rows are sometimes extended for 4-5 lines as well. I do not really care about the stack trace but definitely need the Timestamp, ThereadId, Short name, Event Type and full error message in their individual columns.
[5/20/16 22:35:39:841 CDT] 00233723 SystemOut O 22:35:39,840 ERROR [com.xxx.app.yyy.hms.jms.receivers.impl.B2bTonnn278InReceiverImpl]
xxxRuntimeException{errorVO=com.xxx.app.yyy.nnn.mmm.data.mmmCompleteIntakeErrorVO(diagnosesMessagesExist:false, mmmMessagesExist:false, incrementedKey:null, numPagesWithMessages:1, primaryKeyFields:[], providersMessagesExist:false, requiredFields:[], servicesMessagesExist:true, changeDateTime:05-20-2016 10:35:39:840 PM CDT, changeUserID:SYSTEM, createDateTime:null, createUserID:null, dataSecured:false, dataSecurityTypeList:null, globalMessages:[], historyID:0, messages:{procedureUnitCount=[Field For Label: procedureUnitCount Message ID: 'ERR0010', Message Arguments: '[]']}, trackChanges:false, updateVersion:-1, messages={procedureUnitCount=[Field For Label: procedureUnitCount Message ID: 'ERR0010', Message Arguments: '[]']})}
at com.xxx.app.yyy.nnn.mmm.businesslogic.impl.mmmImpl.completemmm(mmmImpl.groovy:612)
at sun.reflect.GeneratedMethodAccessor4988.invoke(Unknown Source)
I tried doing this by reading one line at a time and parsing using Regex - which failed miserably (only 20% of data met the Regex) and that quality is also poor. I really do not know to proceed here and what delimiter to choose to break that exception string to columns (\t already tried - not working too.)
Any help or pointer to right direction here ?

Use Logstash to read and parse the WebSphere logs and post them into Elasticsearch for further processing (i.e use ELK Stack).
Read related discussion here.
With Logstash, you can use Grok to parse any crappy unstructured log data into something structured and queryable.

grep -A 1 SystemOut LogFile | awk 'NR%3{printf $0" ";next;}2' | awk '{print $2" "$4" "$8" "$10}'

Jetty + OSGi results in strange cache errors

I'm trying to run an embedded jetty instance from an OSGi server.
When the server starts I can see the following in the log:
Started o.e.j.w.WebAppContext#1f437060{/browser,bundle://201.0:24/browser,AVAILABLE}
The first request is successful but later requests will result in a stack trace, e.g.
WARN | ResourceCache | Could not load bundle://201.0:24/browser/index.html true 1415275444922 java.nio.HeapByteBuffer[pos=0 lim=9 cap=9] java.nio.HeapByteBuffer[pos=0 lim=29 cap=29]
WARN | ResourceCache |
java.io.FileNotFoundException: \browser\index.html (Det går inte att hitta sökvägen)
at java.io.RandomAccessFile.open(Native Method)[:1.8.0_20]
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)[:1.8.0_20]
at org.eclipse.jetty.util.BufferUtil.readFrom(BufferUtil.java:408)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.ResourceCache.getIndirectBuffer(ResourceCache.java:296)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.ResourceCache$Content.getIndirectBuffer(ResourceCache.java:478)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.ResourceCache$Content.getInputStream(ResourceCache.java:525)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.HttpOutput.sendContent(HttpOutput.java:427)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.HttpOutput.sendContent(HttpOutput.java:345)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.servlet.DefaultServlet.sendData(DefaultServlet.java:887)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:493)[201:com.test.mybundle:1.0.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:735)[201:com.test.mybundle:1.0.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:698)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:505)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:582)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1096)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:432)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1030)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:261)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:101)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:546)[201:com.test.mybundle:1.0.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:735)[201:com.test.mybundle:1.0.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:698)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:505)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:564)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1096)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:432)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1030)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.Server.handle(Server.java:445)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:268)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:229)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)[201:com.test.mybundle:1.0.0]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_20]
My guess is that the path prefix bundle:// is not supported. Could this be the problem, and if so, how could this be resolved?

ServiceMix already has a build in Web Feature. You just need install the war or http feature which will also install a Jetty Server. That's all you need to do, no extras to install in a ServiceMix/Karaf Container.