We Keep Coding

Messages DO NOT appear in the Spring Integration (Kafka) ErrorChannel when Broker is unavailable

I am working with a simple Kafka based project using Spring Integration and we require that when the Broker is down, messages will pass into the ErrorChannel and we can deal with them /save as 'dead-letters' etc.
What we are getting is a countless run of Exceptions:
2017-09-19 17:14:19.651 DEBUG 12171 --- [ad | producer-1] o.apache.kafka.common.network.Selector : Connection with localhost/127.0.0.1 disconnected
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_131]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_131]
But the error channel is not referenced :-/
I have tried to hook it up, but to no avail - here is part of my app-context:
<bean id="channelExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="corePoolSize" value="1"/>
<property name="maxPoolSize" value="10"/>
<property name="queueCapacity" value="1000"/>
</bean>
<int:channel id="producingChannel" >
<int:dispatcher task-executor="channelExecutor" />
</int:channel>
<int-kafka:outbound-channel-adapter id="kafkaOutboundChannelAdapter"
kafka-template="kafkaTemplate"
auto-startup="true"
channel="producingChannel"
topic="${kafka.topic}">
</int-kafka:outbound-channel-adapter>
<int:service-activator input-channel="errorChannel" ref="errorLogger" method="logError" />
<bean id="errorLogger" class="uk.co.sainsburys.integration.service.ErrorLogger" />
<bean id="kafkaTemplate" class="org.springframework.kafka.core.KafkaTemplate">
<constructor-arg ref="producerConfigs"/> <!-- producerConfigs piece is NOT included! -->
</bean>
Sadly, I am not an expert at Spring Integration - any ideas what I am doing wrong?
Thanks for your help.

Everything is correct so far. The problem that you are missing the fact of async behavior by default in the KafkaProducerMessageHandler:
/**
* A {#code boolean} indicating if the {#link KafkaProducerMessageHandler}
* should wait for the send operation results or not. Defaults to {#code false}.
* In {#code sync} mode a downstream send operation exception will be re-thrown.
* #param sync the send mode; async by default.
* #since 2.0.1
*/
public void setSync(boolean sync) {
So, consider to use sync="true" attribute on the <int-kafka:outbound-channel-adapter>.
In addition, with the latest upcoming versions we have introduced:
<xsd:attribute name="send-failure-channel" type="xsd:string">
<xsd:annotation>
<xsd:documentation><![CDATA[
Specifies the channel to which an ErrorMessage for a failed send will be sent.
]]></xsd:documentation>
<xsd:appinfo>
<tool:annotation kind="ref">
<tool:expected-type type="org.springframework.messaging.MessageChannel" />
</tool:annotation>
</xsd:appinfo>
</xsd:annotation>
</xsd:attribute>
which is useful for the async behavior to catch those errors.

reply-timeout meaning in tcp-inbound-gateway in spring integration

Spring integration tcp gateway can be setup as follows:
<!-- Server side -->
<int-ip:tcp-connection-factory id="crLfServer"
type="server"
port="${availableServerSocket}"/>
<int-ip:tcp-inbound-gateway id="gatewayCrLf"
connection-factory="crLfServer"
request-channel="serverBytes2StringChannel"
error-channel="errorChannel"
reply-timeout="10000" />
<int:channel id="toSA" />
<int:service-activator input-channel="toSA"
ref="echoService"
method="test"/>
<bean id="echoService"
class="org.springframework.integration.samples.tcpclientserver.EchoService" />
<int:object-to-string-transformer id="serverBytes2String"
input-channel="serverBytes2StringChannel"
output-channel="toSA"/>
<int:transformer id="errorHandler"
input-channel="errorChannel"
expression="Error processing payload"/>
Notice the reply-timeout which is set as 10 seconds.
Does it mean that the TCP server will call the service and can wait for a maximum of 10 seconds? If the service does not reply within 10 seconds, Does the TCP server will send the message to errorChannel which in turn sends the client error message "Error processing payload"?
When I tested the TCP Server with a service that takes 20 seconds, client is taking 20 seconds to get the response. I am not seeing error message.
Can you please help in understanding the reply-timeout in TCP inbound-gateway?
Thanks
UPDATE:
Thanks for Artem to help out with this issue.
Best way to solve this problem is with the following config:
<beans>
<int-ip:tcp-connection-factory id="crLfServer" type="server" port="${availableServerSocket}"/>
<int-ip:tcp-inbound-gateway id="gatewayCrLf" connection-factory="crLfServer" request-channel="requestChannel" error-channel="errorChannel" reply-timeout="5000" />
<int:service-activator input-channel="requestChannel" ref="gateway" requires-reply="true"/>
<int:gateway id="gateway" default-request-channel="timeoutChannel" default-reply-timeout="5000" />
<int:object-to-string-transformer id="serverBytes2String" input-channel="timeoutChannel" output-channel="serviceChannel"/>
<int:channel id="timeoutChannel">
<int:dispatcher task-executor="executor"/>
</int:channel>
<bean id="executor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="corePoolSize" value="5" />
<property name="maxPoolSize" value="10" />
<property name="queueCapacity" value="25" />
</bean>
<int:service-activator input-channel="serviceChannel" ref="echoService" method="test"/>
<bean id="echoService" class="org.springframework.integration.samples.tcpclientserver.EchoService" />
<int:transformer id="errorHandler" input-channel="errorChannel" expression="payload.failedMessage.payload + ' errorHandleMsg: may be timeout error'"/>
</beans>
Thanks

Well, actually we should on that attribute a description like we have in other similar places, e.g. HTTP Inbound Gateway:
<xsd:attribute name="reply-timeout" type="xsd:string">
<xsd:annotation>
<xsd:documentation><![CDATA[
Used to set the receiveTimeout on the underlying MessagingTemplate instance
(org.springframework.integration.core.MessagingTemplate) for receiving messages
from the reply channel. If not specified this property will default to "1000"
(1 second).
]]></xsd:documentation>
</xsd:annotation>
</xsd:attribute>
That timeout means how much to wait for reply from downstream flow. But! That is possible if you flow is shifted to another thread somewhere. Otherwise everything is performed in the caller's Thread and therefore the wait time isn't deterministic.
Anyway we return null there after timeout without reply. And it is reflected in the TcpInboundGateway:
Message<?> reply = this.sendAndReceiveMessage(message);
if (reply == null) {
if (logger.isDebugEnabled()) {
logger.debug("null reply received for " + message + " nothing to send");
}
return false;
}
We can reconsider a logic in the TcpInboundGateway for :
if (reply == null && this.errorOnTimeout) {
if (object instanceof Message) {
error = new MessageTimeoutException((Message<?>) object, "No reply received within timeout");
}
else {
error = new MessageTimeoutException("No reply received within timeout");
}
}
But seems for me it really would be better on to rely on the timeout from the client.
UPDATE
I think we can overcome the limitation and meet you requirements with the midflow <gateway>:
<gateway id="gateway" default-request-channel="timeoutChannel" default-reply-timeout="10000"/>
<channel id="timeoutChannel">
<dispatcher task-executor="executor"/>
</channel>
<service-activator input-channel="requestChannel"
ref="gateway"
requires-reply="true"/>
So, the <service-activator> calls <gateway> and waits for reply from there. Requiring the last one, of course, to end up with the ReplyRequiredException, which you can convert into desired MessageTimeoutException in your error flow on the error-channel="errorChannel".
The timeoutChannel is an executor one, making our default-reply-timeout="10000" very useful because we shift a message on the gateway into separate thread immediately and move right from there into reply waiting process wrapped with that timeout on the CountDonwLatch.
Hope that is clear.

activemq oom after enabling stomp

After enabling STOMP protocol (before it was only default protocol enabled) on the Activemq server it started to fail with oom. I have only 1 client using STOMP. It can work for 1 week without fails or fail a day after a restart. Here is the config file:
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
<bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
lazy-init="false" scope="singleton"
init-method="start" destroy-method="stop">
</bean>
<!--
The <broker> element is used to configure the ActiveMQ broker.
-->
<broker useJmx="true" xmlns="http://activemq.apache.org/schema/core" brokerName="cms-mq" dataDirectory="${activemq.data}">
<destinationInterceptors>
<virtualDestinationInterceptor>
<virtualDestinations>
<virtualTopic name="VirtualTopic.>" selectorAware="true"/>
</virtualDestinations>
</virtualDestinationInterceptor>
</destinationInterceptors>
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="false">
</policyEntry>
<policyEntry queue=">" producerFlowControl="false">
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="70" />
</memoryUsage>
<storeUsage>
<storeUsage limit="4 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="4 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="auto" uri="auto+nio://0.0.0.0:61616?maximumConnections=1000&auto.protocols=default,stomp"/>
</transportConnectors>
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
</shutdownHooks>
<plugins>
... security plugins config...
</plugins>
</broker>
<import resource="jetty.xml"/>
</beans>
start args:
/usr/java/default/bin/java -Xms256M -Xmx1G -Dorg.apache.activemq.UseDedicatedTaskRunner=false -XX:HeapDumpPath=/var/logs/heapDumps -XX:+HeapDumpOnOutOfMemoryError -Dcom.sun.management.jmxremote.port=8162 -Dcom.sun.management.jmxremote.rmi.port=8162 -Dcom.sun.management.jmxremote.password.file=/opt/apache-activemq-5.13.0//conf/jmx.password -Dcom.sun.management.jmxremote.access.file=/opt/apache-activemq-5.13.0//conf/jmx.access -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote -Djava.awt.headless=true -Djava.io.tmpdir=/opt/apache-activemq-5.13.0//tmp -Dactivemq.classpath=/opt/apache-activemq-5.13.0//conf:/opt/apache-activemq-5.13.0//../lib/ -Dactivemq.home=/opt/activemq -Dactivemq.base=/opt/activemq -Dactivemq.conf=/opt/apache-activemq-5.13.0//conf -Dactivemq.data=/opt/apache-activemq-5.13.0//data -jar /opt/activemq/bin/activemq.jar start
UPD:
From Eclipse MemoryAnalizer:
Leak Suspects
247,036 instances of "org.apache.activemq.command.ActiveMQBytesMessage", loaded by "java.net.URLClassLoader # 0xc02e9470" occupy 811,943,360 (76.92%) bytes.
81 instances of "org.apache.activemq.broker.region.cursors.FilePendingMessageCursor", loaded by "java.net.URLClassLoader # 0xc02e9470" occupy 146,604,368 (13.89%) bytes.
UPD:
Before having OOM error there are several error in the log like the following:
| ERROR | Could not accept connection from null: java.lang.IllegalStateException: Timer already cancelled. | org.apache.activemq.broker.TransportConnector | ActiveMQ BrokerService[cms-mq] Task-13707
| INFO | The connection to 'null' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ BrokerService[cms-mq] Task-13738
Would appriciate any help in debugging it.
Can provide additional info if needed.

One guess is that you are flooding the broker with messages from the producer over STOMP and eventually blowing the broker memory. You have turned producer flow control off which can lead to this with even the default JMS client, and STOMP is even easier to get into this situation since there isn't by default an ack going back to the producer to allow for a flow control mechanism, you have to request a receipt on each send to get that.
To debug this you need to start examining your broker logs and the destination and usage stats via the console or other tool of your choosing to see what the state of the broker is.

I examined the client code (ruby stomp client) and it turned out that there was 'activemq.subscriptionName' sub header without 'client-id' connect header. When you set 'activemq.subscriptionName' subscription header that means that you want your subscriber to be durable. But you should also set 'client-id' connect header because otherwise it is autogenerated. When 'client-id' header is not set we have a situation in which the broker can't identify the stomp client by it's client id when it reconnects. As a result there were a lot of Offline Durable Topic Subscribers and messages were piling up for every client-id => OOM error.

KahaDB db.data and ActiveMQ 5.6

I am looking for answers for the following questions related to KahaDB.
I have an app that uses ActiveMQ and the app receives about 500,000 to 1 million messages which are written to an ActiveMQ queue with consumers picking them up.
I observed that the db.data file size varies anywhere between a few KB to almost 600 or 700 MB.
I noticed that when the db.data file size grows, messages get stuck, forcing me to shutdown my app, restart the app to have new consumers connecting to the broker to drain all messages.
Ideally what should be the size of db.data? If an app produces and consumes at the same time, irrespective of the number of messages, I think the size of db.data should be small.
Why sometimes the file size grows and what triggers the file size of db.data to grow?
I have ActiveMQ 5.6 and I see this issue at least 2 or 3 times a month. In the brokers that are running smooth, I observed that the file size of db.data is in KB and not 500 MB or 700 MB.
Here is the sample activemq.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:amq="http://activemq.apache.org/schema/core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
<!-- Allows us to use system properties as variables in this configuration file -->
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.conf}/credentials.properties</value>
</property>
</bean>
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="/foo/activemq_5.6/data">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
<pendingSubscriberPolicy>
<vmCursor />
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb" useCache="false" />
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext createConnector="true" />
</managementContext>
<persistenceAdapter>
<kahaDB directory="/foo/activemq_5.6/data/kahadb" />
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="64 mb" />
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb" />
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb" />
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616" />
</transportConnectors>
</broker>
<import resource="jetty.xml" />
</beans>

Generally the kahadb directory has segments for messages that haven't been consumed yet. If there is any message in the segment that hasn't been consumed yet, it will keep the whole segment. So if you nothing in any queue, you generally won't have much of anything in the kaha directory.
If you have the jetty console enabled, you can use it to see which queues have messages in them.
If you sometimes have some queues that are consumed slowly, you probably want to adjust the segment size from its default (something like 33 meg) to something smaller to avoid filling up your disk/hitting your storage limit from the config file)

ActiveMQ producer generating heapdump

We are using Jboss FuseESB Active MQ on IBM AIX machine. We are sending upto 10000 requests per day to consumer. Each request has about 10Kb size. We are getting heap dump in every 3 days. Here is image of IBM Heap Analyzer.
Right before heap is generated, there are upto I think JMS messages are getting processed slowly by consumer. What are strategies to increase the processing speed of JMS consumer.
Edit:
Here is image of Heap Analyzer:
Edit:
We are using embedded broker. Here is activemq.xml that we are using:
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:amq="http://activemq.apache.org/schema/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org /schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
<!-- Allows us to use system properties and fabric as variables in this configuration file -->
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="properties">
<bean class="org.fusesource.mq.fabric.ConfigurationProperties"/>
</property>
</bean>
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="${broker-name}"
dataDirectory="${data}"
start="false">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true">
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1000"/>
</pendingMessageLimitStrategy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true" queuePrefetch="1">
<deadLetterStrategy>
<individualDeadLetterStrategy queuePrefix="DLQ."/>
</deadLetterStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<persistenceAdapter>
<kahaDB directory="${data}/kahadb"/>
</persistenceAdapter>
<!--
<plugins>
<jaasAuthenticationPlugin configuration="karaf" />
</plugins>
-->
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="128 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="openwire" uri="tcp://0.0.0.0:0?maximumConnections=1000"/>
</transportConnectors>
</broker>
</beans>

Is it possible that there are any Queue and/or durable topic subscribers that are not started and that their messages are persisted in the JMS broker?
You should browse the JMS destinations, for example using JMX, and identify the messages and their consumers. If you checked that every consumer is correctly configured and running, debug why the messages are consumed that slowly.