I have a configuration to read the data from DB using jdbc:inbound-channel-adapter. The configuration:
<int-jdbc:inbound-channel-adapter query="SELECT * FROM requests WHERE processed_status = '' OR processed_status IS NULL LIMIT 5" channel="requestsJdbcChannel"
data-source="dataSource" update="UPDATE requests SET processed_status = 'INPROGRESS', date_processed = NOW() WHERE id IN (:id)" >
<int:poller fixed-rate="30000" />
</int-jdbc:inbound-channel-adapter>
<int:splitter input-channel="requestsJdbcChannel" output-channel="requestsQueueChannel"/>
<int:channel id="requestsQueueChannel">
<int:queue capacity="1000"/>
</int:channel>
<int:chain id="requestsChain" input-channel="requestsQueueChannel" output-channel="requestsApiChannel">
<int:poller max-messages-per-poll="1" fixed-rate="1000" />
.
.
</int:chain>
In the above configuration, I have defined the jdbc poller with fixed-rate of 30 seconds. When there is direct channel instead of requestsQueueChannel the select query gets only 5 rows (since I am using limiting the rows in select query) and waits for another 30 seconds for next poll.
But after I introduce requestsQueueChannel with queue and added poller inside requestsChain, the jdbc-inbound doesn't work as expected. It doesn't wait for another 30 second for next poll. Sometimes it polls the DB twice in a row(within a second) as if there are 2 threads running and gets two sets of rows from DB. However, there is no async handoff except these mentioned above.
My understanding is that even if there is requestsQueueChannel, once it executes the select query it should wait for another 30 seconds to poll the DB. Is there anything I am missing? I just want to understand the behavior of this configuration.
When using a DirectChannel the next poll isn't considered until the current one ends.
When using a QueueChannel (or task executor), the poller is free to run again.
Inbound adapters have max-messages-per-poll set to 1 by default so your config should work as expected. Can you post a DEBUG log somewhere?
The issue of Spring integration pollers activating twice, as though they are 2 threads, is the basically the same problem I came across here, with file system pollers:
How to prevent duplicate Spring Integration service activations when polling directory
Apparently this is a relatively common misconfiguration, where Spring root and servlet contexts both load the Spring Integration configuration. As a result of this, there are indeed two threads, and pollers can be seen to activate twice within their polling period. Usually within a few seconds of each other, as each will start when its context loads.
My approach to ensuring that the Spring Integration configuration was only loaded in a single context was to structure the project packages to ensure separation.
First define a web config which only picks up classes under the "web" package.
#Configuration
#ComponentScan(basePackages = { "com.myapp.web" })
#EnableWebMvc
public class WebConfig extends WebMvcConfigurerAdapter {
#Override
public void configureDefaultServletHandling(
DefaultServletHandlerConfigurer configurer) {
configurer.enable();
}
}
Create separate root configuration classes to load beans such as services and repositories, which do not belong in the servlet context. One of these should load the Spring Integration configuration. i.e.:
#Configuration
#ComponentScan(basePackages = { "com.myapp.eip" })
#ImportResource(value = { "classpath:META-INF/spring/integration-context.xml" })
public class EipConfig {
}
An additional factor in the configuration that took a little while to work out, was that my servlet filters and web security config needed to be in the root context rather than the servlet context.
Related
I have following configuration to run task--
<bean id="trendDataJob" class="com.ge.og.realtrack.scheduler.TrendDataJob"> </bean>
<task:scheduled-tasks>
<task:scheduled ref="trendDataJob" method="trendJob" cron="#{trendDataJob.configMap['corn_exp']}"></task:scheduled>
<task:scheduled ref="trendDataJob" method="metaDataTrendJob" cron="#{trendDataJob.configMap['metadata_corn_exp']}"></task:scheduled>
</task:scheduled-tasks>
cron expression for this is corn_exp=0 0/1 * * * ? to run every minute.
Here is problem as both method of trendDataJob schedule to run every minute but they are executing one after another first trendJob once its completed then its executing metaDataTrendJob i am not able to understand this behavior .
Also another problem is in case of method takes more than one minute to finish finish..its not triggering next call till current call finish and return.
By default the scheduler uses a ConcurrentTaskScheduler with a single thread. If you want another one configure it and pass it to the scheduled-tasks scheduler attribute.
The easiest way, in XML, is to use the scheduler element. (See this section in the reference guide).
<task:scheduler id="scheduler" pool-size="10"/>
Then simply register it on the other element.
<task:scheduled-tasks scheduler="scheduler"> ...
Have you used #EnableScheduling in your java code?
#EnableScheduling ensures that a background task executor is created. Without it, nothing gets scheduled.
For more, you can go through
Spring 3 #Scheduled – 4 Ways to Schedule Tasks
Spring Batch + Spring TaskScheduler example
Scheduling Tasks
Enable scheduling annotations
To enable support for #Scheduled and #Async annotations add #EnableScheduling and #EnableAsync to one of your #Configuration classes:
#Configuration
#EnableAsync
#EnableScheduling
public class AppConfig {
}
You are free to pick and choose the relevant annotations for your application. For example, if you only need support for #Scheduled, simply omit #EnableAsync. For more fine-grained control you can additionally implement the SchedulingConfigurer and/or AsyncConfigurer interfaces. See the javadocs for full details.
If you prefer XML configuration use the <task:annotation-driven> element.
<task:annotation-driven executor="myExecutor" scheduler="myScheduler"/>
<task:executor id="myExecutor" pool-size="5"/>
<task:scheduler id="myScheduler" pool-size="10"/>
Notice with the above XML that an executor reference is provided for
handling those tasks that correspond to methods with the #Async
annotation, and the scheduler reference is provided for managing those
methods annotated with #Scheduled.
If you're using a default task scheduler in spring, i'm pretty sure it only runs on a single thread, hence why you cannot make them run in parallel.
You need to configure some kind of BatchScheduler with a pool size, to make it run in parallel.
I have used ActiveMQ as JMS implementation (activemq-spring 5.12.1) and Spring JMS integration (spring-jms 4.2.3.RELEASE), all wrapped in Spring Boot web application, being deployed on Tomcat.
I have following Spring configuration (code reduced for the verbosity of code sample):
#Configuration
#EnableJms
public class AppConfiguration {
#Bean
public XAConnectionFactory jmsXaConnection(String activeMqUsername, String activeMqPassword) {
ActiveMQXAConnectionFactory activeMQXAConnectionFactory = new ActiveMQXAConnectionFactory(activeMqUsername, activeMqPassword, activeMqUrl);
ActiveMQPrefetchPolicy prefetchPolicy = new ActiveMQPrefetchPolicy();
prefetchPolicy.setAll(0);
activeMQXAConnectionFactory.setPrefetchPolicy(prefetchPolicy);
return activeMQXAConnectionFactory;
}
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory(ConnectionFactory connectionFactory, JtaTransactionManager jtaTransactionManager) {
DefaultJmsListenerContainerFactory containerFactory = new DefaultJmsListenerContainerFactory();
containerFactory.setConnectionFactory(connectionFactory);
containerFactory.setTransactionManager(jtaTransactionManager);
containerFactory.setSessionTransacted(true);
containerFactory.setTaskExecutor(Executors.newFixedThreadPool(2));
containerFactory.setConcurrency("2-2");
containerFactory.setCacheLevel(DefaultMessageListenerContainer.CACHE_CONSUMER);
return containerFactory;
}
}
My target was to configure two consumers (hence concurrecny set to 2-2) and to prevent any messages caching (hence prefetch policy set to 0).
It works, but causes very unpleasent side effect:
When I try to undeploy the application via Tomcat Manager, it hangs for a while and then indefinitely, every second produces following DEBUG message:
"DefaultMessageListenerContainer:563 - Still waiting for shutdown of 2 Message listener invokers".
Therefore, I am forced to kill Tomcat process every time. What have I done wrong?
One of my lucky shots (documentation both ActiveMQ and Spring JMS was not that helpful), was to set prefetch policy to 1 instead of 0. Then it undeploys gracefully, but I cannot see how it can relate.
Also I am curious, why having cache level set to CACHE_CONSUMER is required for the ActiveMQ to create two consumers. When default setting was left (CACHE_NONE while using external transaction manager), only one consumer was created (while concurrency was still set two 2-2, and so was TaskExecutor).
If it matters, for connection factory and transaction manager, Atomikos is used. I can paste its configuration also, but it seems irrelevant.
Most likely this means the consumer threads are "stuck" in user code; take a thread dump with jstack to see what the container threads are doing.
I have a grails 2.2 application that uses the JMS plugin (using version 1.3).
The situation I have is that when my server starts up, the JMS plugin initialises and the Listener service grabs any waiting messages on the queue before the server has completed setting up.
Specifically, it hits the first hibernate query in the code and fails with the following error:
| Error 2014-10-14 11:06:56,535 [ruleInputDataListenerJmsListenerContainer-1] ERROR drms.RuleInputDataListenerService - Message Exception: Failed to process JMS Message.
groovy.lang.MissingMethodException: No signature of method: au.edu.csu.drms.Field.executeQuery() is applicable for argument types: () values: []
Possible solutions: executeQuery(java.lang.String), executeQuery(java.lang.String, java.util.Collection), executeQuery(java.lang.String, java.util.Map), executeQuery(java.lang.String, java.util.Collection, java.util.Map), executeQuery(java.lang.String, java.util.Map, java.util.Map)
The code in question is correct:
String query = "SELECT f FROM field f WHERE (attributeName = :attributeName AND entityName = :entityName)"
def fieldList = Field.executeQuery(query, [attributeName: _attributeName, entityName: _entityName])
From what I can tell, it's a matter of hibernate not being initialised when the JMS listener executes the onMessage method. It also happens with a withCriteria or any other hibernate query method.
It only happens when there are messages on the queue on server start-up and fails for each message waiting. Once the queue is completed and it processes new messages, it works fine.
Is there a way to either get hibernate to initialise in time or to delay the Listener service from execute (much like the Quartz plugin that has a start up delay timer)?
Update:
I don't use a bean configuration because it's a daemon type application - we have no beans to define.
Is there a way to use #DependsOn and have my listener depend on Hibernate itself?
Let's say you have the following EntityManagerFactory configuration:
<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="jpaDialect" ref="jpaDialect"/>
</bean>
You need to make your JMS connection factory depend on entityManagerFactory:
<bean id="jmsConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory"
destroy-method="stop" depends-on="jmsBroker, entityManagerFactory">
<property name="connectionFactory" ref="activeMQConnectionFactory"/>
</bean>
Unfortunately, the #DependsOn notation didn't work due to the nature of my application (no bean configuration).
Given that there are a few bugs/problems with the Grails JMS plugin, the solution to my problem was to use the following code prior to processing the JMS Message:
def onMessage(msg) {
try {
Rule.withNewTransaction {
log.info("Hibernate is up and running!")
}
} catch (Exception e) {
resendMessage(msg)
}
// rest of code...
}
Where I use a transaction to test if Hibernate is fully initialised (already tested that it's not when the JMS listener fires at start up) and if it catches an exception, it will resend the message back to the queue for re-processing.
I want to collect some messages(lets say 10) and pass them as a list to the service activator instead of passing them one by one.
The context:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns=...>
<int:channel id="ch.http.in"/>
<int:channel id="ch.http.trans"/>
<int:channel id="ch.http.aggr"/>
<int-http:inbound-channel-adapter path="test" channel="ch.http.in"/>
<int:map-to-object-transformer input-channel="ch.http.in" output-channel="ch.http.trans" type="demo.Req"/>
<int:aggregator
input-channel="ch.http.trans"
output-channel="ch.http.aggr"
release-strategy-expression="size() == 10"
correlation-strategy-expression="headers['id']"
ref="aggr" method="add"/>
<int:service-activator ref="srv" method="httpTest" input-channel="ch.http.aggr"/>
<bean id="srv" class="demo.IntService"/>
<bean id="aggr" class="demo.HttpAggregator"/>
</beans>
The aggreagator:
public class HttpAggregator{
public List<Req> add(List<Req> reqs) {
System.out.println(reqs);
return reqs;
}
}
The service:
public class IntService {
public void httpTest(Req msg){
System.out.println(msg);
}
}
Req is just a POJO.
The problem is that the aggregator method is never called. Without the aggregtor the messages are passed to the service activator with no problem.
Using Spring Integration 3.0.2.RELEASE (Spring Boot 1.0.2.RELEASE)
Edit:
When I changed correlation-strategy-expression="headers['id']" to correlation-strategy-expression="payload.id"(the Req object has property id) it works when I pass different ids for every chunk(e.g. id=1 for the first 10; 2 for the next 10...) Looks that that's how the correlation strategy works. How can I baypass it? I just want to limit the size of the aggregated list.
Right; you have to correlate on something; using the headers['id'] will end up with lots of group of 1 item which will never meet the release strategy.
For a simple use case like yours, correlate on a literal - e.g. correlation-expression="'foo'" and set expire-groups-on-completion="true". This resets the group after the release, so a new one (with the same correlation id) can start on the next message.
If you want to release a partial group after some timeout, you will need a MessageGroupStoreReaper. Or, if you can upgrade to 4.0.x, the aggregator now has a group-timeout (or group-timeout-expression).
I have a problem where in I need to receive a series of messages from an MQ queue and write this to a file and initiate a spring batch job with the file as input. Right now I'm thinking of launching the job with wired #Autowired JobLauncher jobLauncher and #Autowired Job job; from the MDB itself.But I feel this is not a good approach as spring batch may create a series of threads and EJB as such doesnt support multi threading.
Is there any other effective way to do this ? I dont want to use quartz scheduler or anything else since it adds complexity. Is there any interface in spring batch itself which launches a job soon after a file comes in a directory ? Any leads in doing this better would be appreciated.
Thanks.
I have a problem where in I need to receive a series of messages from an MQ queue and write this to a file and initiate a spring batch job with the file as input
One way to do that would be engage a bit of Spring Integration, where you would have a file poller, that would poll for a new file:
<file:inbound-channel-adapter id="filePoller"
channel="filesAreComing"
directory="file:${input.directory}"
filename-pattern="test*" />
Adapt a file message ( java.io.File ) to a file name ( String ), since that is what Spring Batch needs. This can be done with a JobLauncher adapter, that is already available from Spring Batch Admin here:
#ServiceActivator
public JobLaunchRequest adapt(File file) throws NoSuchJobException {
JobParameters jobParameters = new JobParametersBuilder().addString(
"input.file", file.getAbsolutePath()).toJobParameters();
return new JobLaunchRequest(job, jobParameters);
}
wrap it to a JobLaunchRequest ( which is just a holder for a Job and JobParameters ) and send this request [as a message] to JobLaunchingMessageHandler:
<service-activator input-channel="jobLauncher">
<beans:bean class="org.springframework.batch.integration.launch.JobLaunchingMessageHandler">
<beans:constructor-arg ref="jobLauncher" />
</beans:bean>
</service-activator>
that would launch the job.
"input.file" is a parameter that is bound at runtime ( hence #{...} ):
<bean id="reader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<property name="resource" value="#{jobParameters[input.file]}" />
... line mapper and other props
</bean>
I'm not sure I understand why you have a message queue, a message-driven POJO/EJB, AND a batch job.
One way to do it is to have the message driven POJO/EJB do the work. It's already an asynch process. You can pool the message driven beans so there are sufficient workers to handle the load. Why add complexity?
If you'd rather not do that, forget the queue and use Spring Batch on its own. I wouldn't do both.