We Keep Coding

activemq oom after enabling stomp

After enabling STOMP protocol (before it was only default protocol enabled) on the Activemq server it started to fail with oom. I have only 1 client using STOMP. It can work for 1 week without fails or fail a day after a restart. Here is the config file:
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
<bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
lazy-init="false" scope="singleton"
init-method="start" destroy-method="stop">
</bean>
<!--
The <broker> element is used to configure the ActiveMQ broker.
-->
<broker useJmx="true" xmlns="http://activemq.apache.org/schema/core" brokerName="cms-mq" dataDirectory="${activemq.data}">
<destinationInterceptors>
<virtualDestinationInterceptor>
<virtualDestinations>
<virtualTopic name="VirtualTopic.>" selectorAware="true"/>
</virtualDestinations>
</virtualDestinationInterceptor>
</destinationInterceptors>
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="false">
</policyEntry>
<policyEntry queue=">" producerFlowControl="false">
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="70" />
</memoryUsage>
<storeUsage>
<storeUsage limit="4 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="4 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="auto" uri="auto+nio://0.0.0.0:61616?maximumConnections=1000&auto.protocols=default,stomp"/>
</transportConnectors>
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
</shutdownHooks>
<plugins>
... security plugins config...
</plugins>
</broker>
<import resource="jetty.xml"/>
</beans>
start args:
/usr/java/default/bin/java -Xms256M -Xmx1G -Dorg.apache.activemq.UseDedicatedTaskRunner=false -XX:HeapDumpPath=/var/logs/heapDumps -XX:+HeapDumpOnOutOfMemoryError -Dcom.sun.management.jmxremote.port=8162 -Dcom.sun.management.jmxremote.rmi.port=8162 -Dcom.sun.management.jmxremote.password.file=/opt/apache-activemq-5.13.0//conf/jmx.password -Dcom.sun.management.jmxremote.access.file=/opt/apache-activemq-5.13.0//conf/jmx.access -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote -Djava.awt.headless=true -Djava.io.tmpdir=/opt/apache-activemq-5.13.0//tmp -Dactivemq.classpath=/opt/apache-activemq-5.13.0//conf:/opt/apache-activemq-5.13.0//../lib/ -Dactivemq.home=/opt/activemq -Dactivemq.base=/opt/activemq -Dactivemq.conf=/opt/apache-activemq-5.13.0//conf -Dactivemq.data=/opt/apache-activemq-5.13.0//data -jar /opt/activemq/bin/activemq.jar start
UPD:
From Eclipse MemoryAnalizer:
Leak Suspects
247,036 instances of "org.apache.activemq.command.ActiveMQBytesMessage", loaded by "java.net.URLClassLoader # 0xc02e9470" occupy 811,943,360 (76.92%) bytes.
81 instances of "org.apache.activemq.broker.region.cursors.FilePendingMessageCursor", loaded by "java.net.URLClassLoader # 0xc02e9470" occupy 146,604,368 (13.89%) bytes.
UPD:
Before having OOM error there are several error in the log like the following:
| ERROR | Could not accept connection from null: java.lang.IllegalStateException: Timer already cancelled. | org.apache.activemq.broker.TransportConnector | ActiveMQ BrokerService[cms-mq] Task-13707
| INFO | The connection to 'null' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ BrokerService[cms-mq] Task-13738
Would appriciate any help in debugging it.
Can provide additional info if needed.

One guess is that you are flooding the broker with messages from the producer over STOMP and eventually blowing the broker memory. You have turned producer flow control off which can lead to this with even the default JMS client, and STOMP is even easier to get into this situation since there isn't by default an ack going back to the producer to allow for a flow control mechanism, you have to request a receipt on each send to get that.
To debug this you need to start examining your broker logs and the destination and usage stats via the console or other tool of your choosing to see what the state of the broker is.

I examined the client code (ruby stomp client) and it turned out that there was 'activemq.subscriptionName' sub header without 'client-id' connect header. When you set 'activemq.subscriptionName' subscription header that means that you want your subscriber to be durable. But you should also set 'client-id' connect header because otherwise it is autogenerated. When 'client-id' header is not set we have a situation in which the broker can't identify the stomp client by it's client id when it reconnects. As a result there were a lot of Offline Durable Topic Subscribers and messages were piling up for every client-id => OOM error.

Spring JMS - Unable to connect to broker URL with embedded Broker

Here's my JmsMessageSender
#Service
public class JmsMessageSender {
#Autowired
private JmsTemplate jmsTemplate;
/**
* send text to default destination
* #param text
*/
public void send(final String text) {
this.jmsTemplate.send(new MessageCreator() {
#Override
public Message createMessage(Session session) throws JMSException {
Message message = session.createTextMessage(text);
return message;
}
});
}
/**
* Simplify the send by using convertAndSend
* #param text
*/
public void sendText(final String text) {
this.jmsTemplate.convertAndSend(text);
}
/**
* Send text message to a specified destination
* #param text
*/
public void send(final Destination dest,final String text) {
this.jmsTemplate.send(dest,new MessageCreator() {
#Override
public Message createMessage(Session session) throws JMSException {
Message message = session.createTextMessage(text);
return message;
}
});
}
}
Here's my DemoClass
public class DemoMain {
public static void main(String[] args) {
// init spring context
ApplicationContext ctx = new ClassPathXmlApplicationContext("context.xml");
// get bean from context
JmsMessageSender jmsMessageSender = (JmsMessageSender)ctx.getBean("jmsMessageSender");
// send to default destination
jmsMessageSender.send("hello JMS");
// send to a code specified destination
Queue queue = new ActiveMQQueue("AnotherDest");
jmsMessageSender.send(queue, "hello Another Message");
// close spring application context
((ClassPathXmlApplicationContext)ctx).close();
}
}
My context.xml
and my activemq.xml configuration.
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:amq="http://activemq.apache.org/schema/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
<!-- Allows us to use system properties as variables in this configuration file -->
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.conf}/credentials.properties</value>
</property>
</bean>
<!--
The <broker> element is used to configure the ActiveMQ broker.
-->
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="activemq-data">
<!--
For better performances use VM cursor and small memory limit.
For more information, see:
http://activemq.apache.org/message-cursors.html
Also, if your producer is "hanging", it's probably due to producer flow control.
For more information, see:
http://activemq.apache.org/producer-flow-control.html
-->
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true">
<!-- The constantPendingMessageLimitStrategy is used to prevent
slow topic consumers to block producers and affect other consumers
by limiting the number of messages that are retained
For more information, see:
http://activemq.apache.org/slow-consumer-handling.html
-->
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1000"/>
</pendingMessageLimitStrategy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">
<!-- Use VM cursor for better latency
For more information, see:
http://activemq.apache.org/message-cursors.html
<pendingQueuePolicy>
<vmQueueCursor/>
</pendingQueuePolicy>
-->
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<!--
The managementContext is used to configure how ActiveMQ is exposed in
JMX. By default, ActiveMQ uses the MBean server that is started by
the JVM. For more information, see:
http://activemq.apache.org/jmx.html
-->
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<!--
Configure message persistence for the broker. The default persistence
mechanism is the KahaDB store (identified by the kahaDB tag).
For more information, see:
http://activemq.apache.org/persistence.html
-->
<persistenceAdapter>
<kahaDB directory="activemq-data/kahadb"/>
</persistenceAdapter>
<!--
The systemUsage controls the maximum amount of space the broker will
use before slowing down producers. For more information, see:
http://activemq.apache.org/producer-flow-control.html
If using ActiveMQ embedded - the following limits could safely be used:
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="20 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="100 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
-->
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="64 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<!--
The transport connectors expose ActiveMQ over a given protocol to
clients and other brokers. For more information, see:
http://activemq.apache.org/configuring-transports.html
-->
<transportConnectors>
<!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="amqp" uri="amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors>
<!-- destroy the spring context on shutdown to stop jetty -->
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
</shutdownHooks>
</broker>
</beans>
However when I run my main class this error shows. Did I miss anything in the config?
Caused by: javax.jms.JMSException: Could not connect to broker URL: tcp://192.168.203.143:61616. Reason: java.net.ConnectException: Connection timed out: connect

Have you started the AMQ-broker on tcp://192.168.203.143:61616 ? If not then starting it should solve the problem.

Had simillar Errors.
Heres solution.
try to connect to borker at your ip address
Could not connect to broker URL: tcp://192.168.203.143:61616.
above mentioned ip address is relevent to Example's System.
try to run same code for your ip address.(Know your ipAddress in network settings or command "ipconfig")
Could not connect to broker URL: tcp://192.168.1.184:61616.

Is it possible to embed ActiveMQ like instance in Mulesoft

I would like to embed Active-MQ broker into Mule flow. I have read that the messages are saved into the file (KahaDB). I read the book "Active-MQ in action" and so I have used "Embedding ActiveMQ using Spring" and I have configured by Using the BrokerFactoryBean:
<spring:beans>
<spring:bean class="org.apache.activemq.xbean.BrokerFactoryBean"
id="broker">
<spring:property value="classpath:activemq.xml" name="config" />
<spring:property value="true" name="start" />
</spring:bean>
</spring:beans>
(Add the library Activemq-all-5.8.0.jar)
And the configuration file is activemq.xml:
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:amq="http://activemq.apache.org/schema/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd
http://activemq.apache.org/camel/schema/spring http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer" />
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost">
<!-- Destination specific policies using destination names or wildcards -->
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" memoryLimit="50mb"/>
<policyEntry topic=">" memoryLimit="50mb"/>
</policyEntries>
</policyMap>
</destinationPolicy>
<!-- Use the following to configure how ActiveMQ is exposed in JMX -->
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<!-- The store and forward broker networks ActiveMQ will listen to -->
<networkConnectors>
<!-- by default just auto discover the other brokers -->
<networkConnector name="default-nc" uri="multicast://default"/>
<!-- Example of a static configuration:
<networkConnector name="host1 and host2" uri="static://(tcp://host1:61616,tcp://host2:61616)"/>
-->
</networkConnectors>
<!-- KahaDB definition -->
<persistenceAdapter>
<kahaDB directory="{user.dir}/activemq-data/KahaDB" />
</persistenceAdapter>
<!-- The maximum about of space the broker will use before slowing down producers -->
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="200 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="10 gb" name="foo"/>
</storeUsage>
<tempUsage>
<tempUsage limit="1000 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<!-- The transport connectors ActiveMQ will listen to -->
<transportConnectors>
<transportConnector name="openwire" uri="tcp://localhost:61616"/>
</transportConnectors>
</broker>
The questions are:
1) I have the flow with JMS, how to visualize with web interface the messages in the tail? Is it possible to visualize the queue from web interface, like when the broken url is tcp://localhost:61616 (URL used to connect to the JMS Server) Active-MQ ?
2) I would like to store the messages into DB from the MULE console, is it possible?

Connect active mq to my standalone JVM

I have downloaded Apache active mq and I am able to run it via activemq.xml config file. How can I monitor my JVMs ESB data via this xml file.
I need to expose these attributes: Name, Enqueue Count, Dequeue Count, Consumer count etc
The XML file is as under:
<!--
<!-- START SNIPPET: example -->
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
<!-- Allows us to use system properties as variables in this configuration file -->
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.conf}/credentials.properties</value>
</property>
</bean>
<!--
The <broker> element is used to configure the ActiveMQ broker.
-->
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="BROKER1" useJmx="true" dataDirectory="${activemq.data}">
<destinationPolicy>
<policyMap>
<policyEntries>
<!--
Limit ALL queues and topics to using 5mb of memory and turn on producer flow control
-->
<policyEntry queue=">" producerFlowControl="true" memoryLimit="5mb"/>
<policyEntry topic=">" producerFlowControl="true" memoryLimit="5mb">
<dispatchPolicy>
<!--
Use total ordering, see:
http://activemq.apache.org/total-ordering.html
-->
<strictOrderDispatchPolicy/>
</dispatchPolicy>
<subscriptionRecoveryPolicy>
<!--
Upon subscription, receive the last image sent
on the destination.
-->
<lastImageSubscriptionRecoveryPolicy/>
</subscriptionRecoveryPolicy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<!--
The managementContext is used to configure how ActiveMQ is exposed in
JMX. By default, ActiveMQ uses the MBean server that is started by
the JVM. For more information, see:
http://activemq.apache.org/jmx.html
-->
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<!--
Configure message persistence for the broker. The default persistence
mechanism is the KahaDB store (identified by the kahaDB tag).
For more information, see:
http://activemq.apache.org/persistence.html
-->
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
<!--
The systemUsage controls the maximum amount of space the broker will
use before disabling caching and/or slowing down producers. For more information, see:
http://activemq.apache.org/producer-flow-control.html
-->
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="70" />
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<!--
The transport connectors expose ActiveMQ over a given protocol to
clients and other brokers. For more information, see:
http://activemq.apache.org/configuring-transports.html
-->
<transportConnectors>
<!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="amqp" uri="amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61613? maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="mqtt" uri="mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="ws" uri="ws://0.0.0.0:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors>
<!-- destroy the spring context on shutdown to stop jetty -->
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
</shutdownHooks>
</broker>
<!--
Enable web consoles, REST and Ajax APIs and demos
The web consoles requires by default login, you can disable this in the jetty.xml file
Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details
-->
<import resource="jetty.xml"/>
</beans>
<!-- END SNIPPET: example -->

This article looks like it should be helpful for you. I imagine that you would want to use jconsole (run from command line) to connect to ActiveMQ via JMX, which gives you access to the data you need.
http://activemq.apache.org/jmx.html

KahaDB db.data and ActiveMQ 5.6

I am looking for answers for the following questions related to KahaDB.
I have an app that uses ActiveMQ and the app receives about 500,000 to 1 million messages which are written to an ActiveMQ queue with consumers picking them up.
I observed that the db.data file size varies anywhere between a few KB to almost 600 or 700 MB.
I noticed that when the db.data file size grows, messages get stuck, forcing me to shutdown my app, restart the app to have new consumers connecting to the broker to drain all messages.
Ideally what should be the size of db.data? If an app produces and consumes at the same time, irrespective of the number of messages, I think the size of db.data should be small.
Why sometimes the file size grows and what triggers the file size of db.data to grow?
I have ActiveMQ 5.6 and I see this issue at least 2 or 3 times a month. In the brokers that are running smooth, I observed that the file size of db.data is in KB and not 500 MB or 700 MB.
Here is the sample activemq.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:amq="http://activemq.apache.org/schema/core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
<!-- Allows us to use system properties as variables in this configuration file -->
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.conf}/credentials.properties</value>
</property>
</bean>
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="/foo/activemq_5.6/data">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
<pendingSubscriberPolicy>
<vmCursor />
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb" useCache="false" />
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext createConnector="true" />
</managementContext>
<persistenceAdapter>
<kahaDB directory="/foo/activemq_5.6/data/kahadb" />
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="64 mb" />
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb" />
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb" />
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616" />
</transportConnectors>
</broker>
<import resource="jetty.xml" />
</beans>

Generally the kahadb directory has segments for messages that haven't been consumed yet. If there is any message in the segment that hasn't been consumed yet, it will keep the whole segment. So if you nothing in any queue, you generally won't have much of anything in the kaha directory.
If you have the jetty console enabled, you can use it to see which queues have messages in them.
If you sometimes have some queues that are consumed slowly, you probably want to adjust the segment size from its default (something like 33 meg) to something smaller to avoid filling up your disk/hitting your storage limit from the config file)