We Keep Coding

SPRING BATCH : How to configure remote chunking for multiple jobs running in a task executor

I am new to spring batch processing. I am using remote chunking where there is a master , multiple slaves and ActiveMQ for messaging.
Master has a job and a job launcher and the job launcher has a task-executor which is having following configuration
<task:executor id="batchJobExecutor" pool-size="2"queue-capacity="100" />.
Chunk configuration is
<bean id="chunkWriter"
class="org.springframework.batch.integration.chunk.ChunkMessageChannelItemWriter" scope="step">
<property name="messagingOperations" ref="messagingGateway" />
<property name="replyChannel" ref="replies" />
<property name="throttleLimit" value="50" />
<property name="maxWaitTimeouts" value="60000" />
</bean>
<bean id="chunkHandler"
class="org.springframework.batch.integration.chunk.RemoteChunkHandlerFactoryBean">
<property name="chunkWriter" ref="chunkWriter" />
<property name="step" ref="someJobId" />
</bean>
<integration:service-activator
input-channel="requests" output-channel="replies" ref="chunkHandler" />
So we are allowed to run two jobs at a time and the remaining jobs will be in the queue.
When two jobs are submitted Master is creating the chunk and submitting to the queue and slave is processing.
But the acknowledgment from the slave to master is giving error
java.lang.IllegalStateException: Message contained wrong job instance id [9331] should have been [9332].
at org.springframework.util.Assert.state(Assert.java:385) ~[Assert.class:4.1.6.RELEASE]
at org.springframework.batch.integration.chunk.ChunkMessageChannelItemWriter.getNextResult
Please help me with this.

The ChunkMessageChannelItemWriter is only designed for one concurrent step - you need to put it in step scope so each job gets its own instance - see this test case
EDIT
Actually, no; that won't work - since the bean instances are using the same reply channel, they could get each other's replies. I opened a JIRA Issue.

This is a very old post, but I think the issue you see here might be related to the throttle limit being larger than the maxWaitTimouts value 4.
What we have seen is that the implementation will not read more than maxWaitTimeouts entries from the reply queue after the job finished. I think this is a bug.
See also the question I asked on stackoverflow here : Remote batch job does not read all responses in afterStep method
I made a bug report for this as well: https://jira.spring.io/browse/BATCH-2651 and am creating a PR to fix the issue.

Recreate all connections of spring rabbit CachingConnectionFactory

I have a CachingConnectionFactory with multiple addresses. When one broker goes down, it connects with the second. Now, when the broker comes up again, I need to destroy existing connections and reconnect to it. But CachingConnectionFactory doesn't have any start, stop methods, instead has only destroy, which might render the factory unusable(?).
Config:
<bean id="testConnFactory" class="org.springframework.amqp.rabbit.connection.CachingConnectionFactory">
<property name="addresses" value="rabbit1,rabbit2" />
<property name="cacheMode" value="CONNECTION" />
<property name="connectionCacheSize" value="${connection.cache.size}" />
</bean>
Is there any way to do this, programatically?

Calling destroy() is safe; the connection(s) will be reset and re-established the next time a component wants one.
Bear in mind, though, that this will impact any in-process operations.
We should probably add a less scary method, such as resetConnection() like we have with the Spring JMS connection factory.

Ehcache Distributed Multiple channel

I am using ehcache in ditributed mode .
The caches are synchronized by channel .
<cacheManagerPeerProviderFactory
class="net.sf.ehcache.distribution.jgroups.JGroupsCacheManagerPeerProviderFactory"
properties="channelName=CHANNEL1:connect=UDP(mcast_port=45568)"
propertySeparator=":" />
For a new requirement , i need to synch with two channels : CHANNEL1 and CHANNEL2 .
Is this possible ? if yes , how i can do that ?
Thanks in advance

Usually One instance of cacheManagerPeerProviderFactory is used to replicate ( or Synchronize) number of caches across cluster. In that case, "channelName=CHANNEL1:" is more like simply giving a name. I don't think ehCache supports multiple channels.
If you're requirement is to have some replication specific one channel, you can try one of the following
Run two EhCache Managers in the same application each with it's ehcache.xml, One for the specific replication logic, and one for the common replication logic of caches.
You will have only one cache manager, but your ehcache.xml will vary in a way that, you won't include the cache ( which requires specific replication logic) with in the third application.
First one is more cleaner approach.
You can have multiple EhcacheManagers with Spring in following way,
<ehcache:annotation-driven cache-manager="ehCacheManager1" />
<bean id="ehCacheManager1" class="org.springframework.cache.ehcache.EhCacheManagerFactoryBean">
<property name="configLocation" value="classpath:ehcache-1.xml" />
</bean>
<bean id="ehCacheManager2" class="org.springframework.cache.ehcache.EhCacheManagerFactoryBean">
<property name="configLocation" value="classpath:ehcache-2.xml" />
</bean>

Spring OpenSessionInViewFilter with #Transactional annotation

This is regarding Spring OpenSessionInViewFilter using with #Transactional annotation at service layer.
i went through so many stack overflow post on this but still confused about whether i should use OpenSessionInViewFilter or not to avoid LazyInitializationException
It would be great help if somebody help me find out answer to below queries.
Is it bad practice to use OpenSessionInViewFilter in application
having complex schema.
using this filter can cause N+1 problem
if we are using OpenSessionInViewFilter does it mean #Transactional not required?
Below is my Spring config file
<context:component-scan base-package="com.test"/>
<context:annotation-config/>
<bean id="messageSource"
class="org.springframework.context.support.ReloadableResourceBundleMessageSource">
<property name="basename" value="resources/messages" />
<property name="defaultEncoding" value="UTF-8" />
</bean>
<bean id="propertyConfigurer"
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"
p:location="/WEB-INF/jdbc.properties" />
<bean id="dataSource"
class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close"
p:driverClassName="${jdbc.driverClassName}"
p:url="${jdbc.databaseurl}" p:username="${jdbc.username}"
p:password="${jdbc.password}" />
<bean id="sessionFactory"
class="org.springframework.orm.hibernate3.LocalSessionFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="configLocation">
<value>classpath:hibernate.cfg.xml</value>
</property>
<property name="configurationClass">
<value>org.hibernate.cfg.AnnotationConfiguration</value>
</property>
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">${jdbc.dialect}</prop>
<prop key="hibernate.show_sql">true</prop>
<!--
<prop key="hibernate.hbm2ddl.auto">create</prop>
-->
</props>
</property>
</bean>
<tx:annotation-driven />
<bean id="transactionManager"
class="org.springframework.orm.hibernate3.HibernateTransactionManager">
<property name="sessionFactory" ref="sessionFactory" />
</bean>

OpenSessionInView is a servlet filter than just Open a hibernate session and store it in the SessionHolder for the thread that is serving the request. With this session opened, hibernate can read the Lazy initialized collections and objects when you use this in the rendering stage of the request. This session can be accessed when you invoke SessionFactory.getCurrentSession().
But, OpenSessionInView just opens the session and it doesn't begin any transactions. With a session opened you can read objects from database but, if you want to do something in a transaction you need #Transactional annotations or other mechanism to demarcate the begin and the end of the transaction when you want.
Then the answer of the questions:
Is it bad practice to use OpenSessionInViewFilter in application having complex schema.
This is a good practice if you need avoid the LazyInitializationException and the overload is just open new Hibernate Session and close it at the end of the request for each request.
Using this filter can cause N+1 problem
I use this filter in many projects and not cause any problem.
if we are using OpenSessionInViewFilter does it mean #Transactional not required?
No. You only have a Hibernate Session opened in the SessionHolder of the thread, but if you need Transactions you need put #Transactional.

Throwing in my 0.02c here (and expanding on Fernando Rincon's excellent answer):
You shouldn't be using a OpenSessionInView filter just because you need to get around a LazyInitializationException. Its just going to add another layer of confusion and complexity to your system. You should know from your system design exactly where you are going to need to access collections on the front end. From there, it's easy and (in my experience) more logical to build a controller method to call a service method to retrieve your collection.
However if you have another problem that using the OpenSessionInView filter solves, and as a happy side effect you then have a session open, then I don't see the harm in using it to access your collections. However, I'd say that if you use the OpenSessionInView to fetch a collection object in one place, you should refactor your code in other places to do the same thing so as the strategy used to fetch collections is standardised across your application.
Weigh up the costs of this refactor against the cost of writing the controller & service methods to determine if you should be using a OpenSessionInView filter.

OpenSessionInViewFilter is a servlet filter that binds a hibernate session to http request and for all db operations, transactional and non transactional, same hibernate session is used for a given http request. This exposes db layer to web layer that makes it anti-pattern.
My experience is that this makes the code difficult to debug when we want to make changes to java objects and do not want those to get reflected in database. Since the hibernate session is always open, it expects to flush the data in database.
This should be used only when JS base rest services are there with no service layer in between.

The typical usage pattern for OpenSessionInViewFilter is that some Entity is lazily loaded but during the view rendering phase the view needs some attribute of this Entity that was not loaded initially thus necessitating the need to fetch this data from the database. Now typically the transaction demarcation is made to happen in the service layer of your web application so by the time the view rendering takes place the view is working with a detached entity which results in a LazyInitializationException when accessing the unloaded attribute.
From this url https://developer.jboss.org/wiki/OpenSessionInView :
The problem
A common issue in a typical web-application is the rendering of the view, after the main logic of the action has been completed, and therefore, the Hibernate Session has already been closed and the database transaction has ended. If you access detached objects that have been loaded in the Session inside your JSP (or any other view rendering mechanism), you might hit an unloaded collection or a proxy that isn't initialized. The exception you get is: LazyInitializationException: Session has been closed (or a very similar message). Of course, this is to be expected, after all you already ended your unit of work.
A first solution would be to open another unit of work for rendering the view. This can easily be done but is usually not the right approach. Rendering the view for a completed action is supposed to be inside the first unit of work, not a separate one. The solution, in two-tiered systems, with the action execution, data access through the Session, and the rendering of the view all in the same virtual machine, is to keep the Session open until the view has been rendered.
As an alternative, consider loading the Entity with just the right amount of data required by your view. This can be accomplished by using DTO projections. This article lists some of the downsides of using the Open Session In View pattern : https://vladmihalcea.com/the-open-session-in-view-anti-pattern/

How to run DefaultMessageListenerContainer on the 'main' thread

I have a case where i want to run DefaultMessageListenerContainer in the same 'main' thread. Right now it uses SimpleAsyncTaskExecutor which spawns new thread every time it receives a message.
We have a test case which connects to different distributed system and do the processing and in the end it asserts few things. As DefaultMessageListenerContainer runs in seperate thread, main thread returns and start executing assertions before DefaultMessageListenerContainer can complete. This leads to failure of the test case. As work around we have made the main thread to sleep for few seconds.
Sample config
<int-jms:message-driven-channel-adapter
id="mq.txbus.publisher.channel.adapter"
container="defaultMessageListenerContainer"
channel="inbound.endpoint.publisher"
acknowledge="transacted"
extract-payload="true" />
<beans:bean id="defaultMessageListenerContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<beans:property name="connectionFactory" ref="mockConnectionFactory"/>
<beans:property name="destination" ref="publisherToTxmQueue"/>
<beans:property name="taskExecutor" ref="taskExecutor"/>
<beans:property name="maxMessagesPerTask" value="10"/>
<beans:property name="sessionTransacted" value="true"/>
</beans:bean>
<beans:bean id="taskExecutor" class="org.springframework.scheduling.timer.TimerTaskExecutor" />
I am trying to use TimerTaskExecutor here because it creates single thread but that thread is seperate than main thread so problem is unresolved. I tried using SyncTaskExecutor but that does not work either (or may be i dint provide the correct property value?).
Answer:
We solved this problem by using SimpleMessageListenerContainer.
This is the new config
<int-jms:message-driven-channel-adapter
id="mq.txbus.publisher.channel.adapter"
container="messageListenerContainer"
channel="inbound.endpoint.publisher"
acknowledge="transacted"
extract-payload="true" />
<beans:bean id="messageListenerContainer" class="org.springframework.jms.listener.SimpleMessageListenerContainer">
<beans:property name="connectionFactory" ref="mockConnectionFactory"/>
<beans:property name="destination" ref="publisherToTxmQueue"/>
<beans:property name="sessionTransacted" value="true"/>
<beans:property name="exposeListenerSession" value="false"/>
</beans:bean>

First you must understand that jms is inherently asynchronous and does not block. This means once you send a message to a queue it will be processed by another thread, maybe on a different machine, possibly minutes or hours later if the consumer is down.
Reading the description of your test case it looks like you are doing some system/integration testing. Unfortunately there is not much you can do except waiting, however you should not wait blindly as this makes your tests slow but also not very stable - no matter how long you wait, on a busy system or some lengthy GC process your test might still time out even though there is no error.
So instead of sleeping for a fixed number of seconds - sleep for e.g. ~100 milliseconds and check some condition that is only met when the processing of message was done. For example if processing the message insert some record into the database, check the database periodically.
Much more elegant way (without busy waiting) is to implement request/repply pattern, see How should I implement request response with JMS? for implementation details. Basically when sending a message you define a reply queue and block waiting for a message in that queue. When processing the original message is done, the consumer should send a reply message to a defined queue. When you receive that message - perform all assertions.

if its for testing purpose then why not use a CyclicBarrier that runs the assertion once the jms activity is completed.