I have a spring rabbit consumer like:
#Override public void onMessage(Message amqpMessage, Channel channel)
throws Exception {
//..some code goes here - I want it to be in spring transaction
}
The issue is the code which is in onMessage method is not under transaction. I checked it, I save data to 3 tables, then throw exception, then save to 4th table. And data from 3 previos tables is not being rolled back. How to do that properly in spring? I want all code in onMessage method to be within a transaction. Thanks
UPDATE
My rabbit conf:
#Configuration #ComponentScan(basePackages = {"com.mycompany"})
public class TicketModeRabbit {
#Bean TicketModeConsumer ticketModeConsumer() {
return new TicketModeConsumer();
}
#Bean(name = TicketModeRabbitData.QUEUE_BEAN_NAME) Queue queue() {
return new Queue(TicketModeRabbitData.QUEUE_BEAN_NAME);
}
#Bean(name = TicketModeRabbitData.QUEUE_BINDING_NAME) Binding binding(
#Qualifier(TicketModeRabbitData.QUEUE_BEAN_NAME) Queue q, TopicExchange e) {
return BindingBuilder.bind(q).to(e).with(TicketModeRabbitData.QUEUE_TOKEN_NAME);
}
#Bean(name = TicketModeRabbitData.CONTAINER_NAME)
SimpleMessageListenerContainer container(ConnectionFactory connectionFactory,
#Qualifier(TicketModeRabbitData.LISTENER_ADAPTED_NAME)
MessageListenerAdapter listenerAdapter) {
return WorkerConfigHelper
.rabbitConfigListenerContainer(connectionFactory, listenerAdapter,
TicketModeRabbitData.QUEUE_BEAN_NAME,
WorkerConfigHelper.GLOBAL_CONCURRENT_CONSUMERS);
}
#Bean(name = TicketModeRabbitData.LISTENER_ADAPTED_NAME)
MessageListenerAdapter listenerAdapter() {
return new MessageListenerAdapter(ticketModeConsumer());
}
}
If your transaction manager is properly set up for your database, the only thing you need to do is add the #Transactional annotation on the onMessage method. Note that the consumer (MessageListener) needs to be a bean managed by the Spring container.
#Override
#Transactional
public void onMessage(Message amqpMessage, Channel channel)
throws Exception {
//..some code goes here - I want it to be in spring transaction
}
Related
I'm currently reading through Spring AMQP's official sample project along with it's corresponding explanations from Spring AMQP docs. The project involves an sync and async version, and the two only differs slightly. Here's the async version:
Producer config:
#Configuration
public class ProducerConfiguration {
protected final String helloWorldQueueName = "hello.world.queue";
#Bean
public RabbitTemplate rabbitTemplate() {
RabbitTemplate template = new RabbitTemplate(connectionFactory());
template.setRoutingKey(this.helloWorldQueueName);
return template;
}
#Bean
public ConnectionFactory connectionFactory() {
return new CachingConnectionFactory();
}
#Bean
public ScheduledProducer scheduledProducer() {
return new ScheduledProducer();
}
#Bean
public BeanPostProcessor postProcessor() {
return new ScheduledAnnotationBeanPostProcessor();
}
static class ScheduledProducer {
#Autowired
private volatile RabbitTemplate rabbitTemplate;
private final AtomicInteger counter = new AtomicInteger();
#Scheduled(fixedRate = 3000)
public void sendMessage() {
rabbitTemplate.convertAndSend("Hello World " + counter.incrementAndGet());
}
}
}
Consumer config:
#Configuration
public class ConsumerConfiguration {
protected final String helloWorldQueueName = "hello.world.queue";
#Bean
public ConnectionFactory connectionFactory() {
return new CachingConnectionFactory();
}
#Bean
public SimpleMessageListenerContainer listenerContainer() {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory());
container.setQueueNames(this.helloWorldQueueName);
container.setMessageListener(new MessageListenerAdapter(new HelloWorldHandler()));
return container;
}
#Bean
public RabbitTemplate rabbitTemplate() {
RabbitTemplate template = new RabbitTemplate(connectionFactory());
template.setRoutingKey(this.helloWorldQueueName);
template.setDefaultReceiveQueue(this.helloWorldQueueName);
return template;
}
#Bean
public Queue helloWorldQueue() {
return new Queue(this.helloWorldQueueName);
}
}
HelloWorldHandler:
public class HelloWorldHandler {
public void handleMessage(String text) {
System.out.println("Received: " + text);
}
}
As the docs explains:
Since this sample demonstrates asynchronous message reception, the producing side is designed to continuously send messages (if it were a message-per-execution model like the synchronous version, it would not be quite so obvious that it is, in fact, a message-driven consumer). The component responsible for continuously sending messages is defined as an inner class within the ProducerConfiguration. It is configured to run every three seconds.
I failed to understand what's "async" about this code, since, from my understanding, in a basic "synchronous fashion", operations like amqpTemplate.converAndSend() and amqpTemplate.receiveAndConvert() already peforms Rabbitmq's async actions, neither producer nor consumer are blocking when sending/receiving messages.
So, how's async in this example manifested? And how to understand async vs sync in the Spring AMQP context?
With async, the MessageListener is invoked by the framework; messages arrive whenever they are available.
With sync, the application calls a receive method which either returns immediately if no message is available, or blocks until a message arrives or a timeout expires.
In the sync case, the application controls when messages are received, with async, the framework is in control.
I am currently working on an event-based async AMQP message listener, like this:
#Configuration
public class ExampleAmqpConfiguration {
#Bean(name = "container")
public SimpleMessageListenerContainer messageListenerContainer() {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(rabbitConnectionFactory());
container.setQueueName("some.queue");
container.setMessageListener(exampleListener());
return container;
}
#Bean
public ConnectionFactory rabbitConnectionFactory() {
CachingConnectionFactory connectionFactory =
new CachingConnectionFactory("localhost");
connectionFactory.setUsername("guest");
connectionFactory.setPassword("guest");
return connectionFactory;
}
#Bean
public MessageListener exampleListener() {
return new MessageListener() {
public void onMessage(Message message) {
System.out.println("received: " + message);
}
};
}
}
I have changed the container bean's autostart property to false. And I have autowired this bean to an event lister that starts the container when StartEvent happens:
#EventListener
public void startContainer(StartEvent startEvent) {
this.container.start();
}
At the same time, I have also autowired the bean to another event that stops the container and shutdowns the container, hoping that the container will be stopped and that there will be no lingering connection:
#EventListener
public void endContainer(EndEvent endEvent) {
this.container.stop();
this.container.shutdown();
}
However, after the EndEvent, I find in my RabbitMQ admin console that all the channels are closed but there is still a lingering connection.
Does that mean that shutdown() is not the right method to use to remove the lingering connection? If that's the case, what is the right method to use?
Thanks.
You need to call resetConnection() on the CachingConnectionFactory to close the connection; as implied by the class name; the connection is cached.
I have seen a lot of examples of Spring Batch projects where either (a) a dataSource is defined, or (b) no dataSource is defined.
However, in my project, I would like my business logic to have access to a dataSource, but I want Spring Batch to NOT use the dataSource. Is this possible?
This guy has a similar problem: Spring boot + spring batch without DataSource
Generally, using spring-batch without a database is not a good idea, since there could be concurrency issues depending on the kind of job you define. So at least an using an inmemory db is strongly advised, especially if you plan to use the job in production.
Using SpringBatch with SpringBoot will initialize an inmemory datasource, if you do not configure your own datasource(s).
Taking this into account, let me redefine your question as follows: Can my businesslogic use another datasource than springbatch is using to update its BATCH-tables?
Yes, it can. As a matter of fact, you can use as many datasources as you want inside your SpringBatch Jobs. Just use by-name autowiring.
Here is how I do it:
I always use Configuration class, which defines all the datasources I have to use in my Jobs
Configuration
public class DatasourceConfiguration {
#Bean
#ConditionalOnMissingBean(name = "dataSource")
public DataSource dataSource() {
// create datasource, that is used by springbatch
// for instance, create an inmemory datasource using the
// EmbeddedDatabaseFactory
return ...;
}
#Bean
#ConditionalOnMissingBean(name = "bl1datasource")
public DataSource bl1datasource() {
return ...; // your first datasource that is used in your businesslogic
}
#Bean
#ConditionalOnMissingBean(name = "bl2datasource")
public DataSource bl2datasource() {
return ...; // your second datasource that is used in your businesslogic
}
}
Three points to note:
SpringBatch is looking for a datasource with the name "dataSource", if you do not provide this EXACT (uppercase 'S') name as the name, spring batch will try to autowire by type and if it finds more than one instance of DataSource, it will throw an exception.
Put your datasource configuration in its own class. Do not put them in the same class as your jobdefinitions are. Spring needs to be able to instantiate the datasource-SpringBean with the name "dataSource" very early when it loads the context. Before it starts to instantiate your Job- and Step-Beans. Spring will not be able to do it correctly, if you put your datasource definitions in the same class as you have your job/step definitions.
Using #ConditionalOnMissingBean is not mandatory, but I found it a good practics. It makes it easy to change the datasources for unit/integration tests. Just provide an additional test configuration in the ContextConfiguration of your unit/IT test which, for instance, overwrites the "bl1Datasource" with an inMemoryDataSource:
Configuration
public class TestBL1DatasourceConfiguration {
// overwritting bl1datasource with an inMemoryDatasource.
#Bean
public DataSource bl1datasource() {
return new EmbeddedDatabaseFactory.getDatabase();
}
}
In order to use the businesslogic datasources, use injection by name:
#Component
public class PrepareRe1Re2BezStepCreatorComponent {
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
private DataSource bl1datasource;
#Autowired
private DataSource bl2datasource;
public Step createStep() throws Exception {
SimpleStepBuilder<..., ...> builder =
stepBuilderFactory.get("astep") //
.<..., ...> chunk(100) //
.reader(createReader(bl1datasource)) //
.writer(createWriter(bl2datasource)); //
return builder.build();
}
}
Furthermore, you probably want to consider using XA-Datasources if you'd like to work with several datasources.
Edited:
Since it seems that you really don't want to use a datasource, you have to implement your own BatchConfigurer (http://docs.spring.io/spring-batch/trunk/apidocs/org/springframework/batch/core/configuration/annotation/BatchConfigurer.html) (as Michael Minella - the SpringBatch project lead - pointed out above).
You can use the code of org.springframework.batch.core.configuration.annotation.DefaultBatchConfigurer as a starting point for your own implementation. Simply remove all the datasource/transactionmanager code and keep the content of the if (datasource === null) part in the initialize method. This will initialize a MapBasedJobRepository and MapBasedJobExplorer. But again, this is NOT a useable solution in a productive environment, since it is not threadsafe.
Edited:
How to implement it:
Configuration class that defines the "businessDataSource":
#Configuration
public class DataSourceConfigurationSimple {
DataSource embeddedDataSource;
#Bean
public DataSource myBusinessDataSource() {
if (embeddedDataSource == null) {
EmbeddedDatabaseFactory factory = new EmbeddedDatabaseFactory();
embeddedDataSource = factory.getDatabase();
}
return embeddedDataSource;
}
}
The implementation of a specific BatchConfigurer:
(of course, the methods have to be implemented...)
public class MyBatchConfigurer implements BatchConfigurer {
#Override
public JobRepository getJobRepository() throws Exception {
return null;
}
#Override
public PlatformTransactionManager getTransactionManager() throws Exception {
return null;
}
#Override
public JobLauncher getJobLauncher() throws Exception {
return null;
}
#Override
public JobExplorer getJobExplorer() throws Exception {
return null;
}
}
And finally the main configuration and launch class:
#SpringBootApplication
#Configuration
#EnableBatchProcessing
// Importing MyBatchConfigurer will install your BatchConfigurer instead of
// SpringBatch default configurer.
#Import({DataSourceConfigurationSimple.class, MyBatchConfigurer.class})
public class SimpleTestJob {
#Autowired
private JobBuilderFactory jobs;
#Autowired
private StepBuilderFactory steps;
#Bean
public Job job() throws Exception {
SimpleJobBuilder standardJob = this.jobs.get(JOB_NAME)
.start(step1());
return standardJob.build();
}
protected Step step1() throws Exception {
TaskletStepBuilder standardStep1 = this.steps.get("SimpleTest_step1_Step")
.tasklet(tasklet());
return standardStep1.build();
}
protected Tasklet tasklet() {
return (contribution, context) -> {
System.out.println("tasklet called");
return RepeatStatus.FINISHED;
};
}
public static void main(String[] args) throws Exception {
SpringApplication.run(SimpleTestJob.class, args);
}
}
This is a separate yet related question to my earlier post here: Safely Terminating a Spring JMS application
My JMS application that I have using spring boot processes everything correctly and shuts down with no errors. To get this to work I changed a bean from:
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(mqConnectionFactory());
factory.setDestinationResolver(destinationResolver());
factory.setConcurrency("1");
factory.setErrorHandler(errorHandler());
factory.setSessionTransacted(true);
factory.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE);
return factory;
}
To:
#Bean
public DefaultMessageListenerContainer defaultMessageListenerContainer() {
DefaultMessageListenerContainer jmsListenerContainer = new DefaultMessageListenerContainer();
jmsListenerContainer.setConnectionFactory(mqConnectionFactory());
jmsListenerContainer.setDestinationResolver(destinationResolver());
jmsListenerContainer.setDestinationName(queueName);
jmsListenerContainer.setConcurrency("1");
jmsListenerContainer.setErrorHandler(errorHandler());
jmsListenerContainer.setSessionTransacted(true);
jmsListenerContainer.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE);
jmsListenerContainer.setAutoStartup(false);
return jmsListenerContainer;
}
The problem with this is, I could have created just a "hotfix", as my knowledge about spring is little. the line in the changed bean jmsListenerContainer.setAutoStartup(false); was added when I stumbled upon this post: http://forum.spring.io/forum/spring-projects/integration/79176-illegalstateexception-no-message-listener-specified as without the autoStartup set to false I get this after every processed message in my logs:
java.lang.IllegalStateException: No message listener specified - see property 'messageListener'
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:691) [spring-jms-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:651) [spring-jms-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:315) [spring-jms-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:253) [spring-jms-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1150) [spring-jms-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1142) [spring-jms-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1039) [spring-jms-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_74]
Everything still is processed and shut down correctly, I just see this in my logs. Not sure if there is a conflict in my config file or not that may be the root of this. I just want to make sure the changes won't cause problems, even though everything currently works as intended with no errors.
Lastly here is my entire configuration class:
#Configuration
#EnableJms
public class MQConfig {
private static final Logger logger = LogManager.getLogger(MQConfig.class.getName());
#Value("${mq.hostName}")
String host;
#Value("${mq.port}")
Integer port;
#Value("${mq.queueManager}")
String queueManager;
#Value("${mq.queueName}")
String queueName;
#Value("${mq.channel}")
String channel;
#Autowired
Environment environment;
#Bean
public DefaultMessageListenerContainer defaultMessageListenerContainer() {
DefaultMessageListenerContainer jmsListenerContainer = new DefaultMessageListenerContainer();
jmsListenerContainer.setConnectionFactory(mqConnectionFactory());
jmsListenerContainer.setDestinationResolver(destinationResolver());
jmsListenerContainer.setDestinationName(queueName);
jmsListenerContainer.setConcurrency("1");
jmsListenerContainer.setErrorHandler(errorHandler());
jmsListenerContainer.setSessionTransacted(true);
jmsListenerContainer.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE);
jmsListenerContainer.setAutoStartup(false);
return jmsListenerContainer;
}
#Bean
public MQConnectionFactory mqConnectionFactory() {
MQConnectionFactory mqConnectionFactory = new MQConnectionFactory();
try {
mqConnectionFactory.setHostName(host);
mqConnectionFactory.setPort(port);
mqConnectionFactory.setQueueManager(queueManager);
mqConnectionFactory.setTransportType(1);
} catch (JMSException ex) {
logger.error(ex.getStackTrace());
}
return mqConnectionFactory;
}
#Bean
public DynamicDestinationResolver destinationResolver() {
DynamicDestinationResolver destinationResolver = new DynamicDestinationResolver();
try {
Connection connection = mqConnectionFactory().createConnection();
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
destinationResolver.resolveDestinationName(session, queueName, false);
} catch (JMSException ex) {
logger.error(ex.getStackTrace());
}
return destinationResolver;
}
#Bean
public MQQueue mqQueue() {
MQQueue mqQueue = new MQQueue();
try {
mqQueue.setBaseQueueName(queueName);
mqQueue.setBaseQueueManagerName(queueManager);
} catch (JMSException ex) {
logger.error(ex.getStackTrace());
}
return mqQueue;
}
#Bean
public JmsErrorHandler errorHandler(){
return new JmsErrorHandler();
}
#Bean
public MQManager mqHoldManager(){
return new MQManager(host, port, queueName,
queueManager, channel);
}
#Bean
public MQInitializer mqInitializer(){
return new MQInitializer(environment);
}
}
Thoughts? Is this messy?
EDIT: JMS Listener
#Component
public class MQConsumer {
#Resource(name = "mqHoldManager")
private MQManager mqHoldManager;
#Resource(name = "defaultMessageListenerContainer")
private DefaultMessageListenerContainer listenerContainer;
#Autowired
MQInitializer mqInitializer; /* To ensure this bean executes before Listener Setup not necessarily for any particular usage*/
final Logger logger = LogManager.getLogger(MQConsumer.class.getName());
private static ReportManager reportManager = new ReportManager();
private static boolean isFirstQueue = true;
#JmsListener(destination = "${mq.queueName}")
public void processOrder(String message) throws Exception {...}
I am confused.
DefaultJmsListenerContainerFactory is for use with annotated POJO methods:
#JmsListener(...)
public void foo(String bar) {...}
The factory is used to create the listener container for the method.
With your replacement configuration, I don't see you ever configuring a message listener into the DefaultMessageListenerContainer.
Usually you would have container.setMessageListner(myListener) where myListener is a MessageListener, a SessionAwareMessageListener or a MessageListenerAdapter that wraps a POJO listener.
Setting autoStartup to false and never starting the container does nothing except add the container bean to the context.
I don't see how you can ever get any messages with that configuration.
EDIT
Are you using Spring Boot? If so, it will create a default jmsListenerContainerFactory for you - that is my best guess.
In which case, your stop code is not really stopping the actual container - just the "dummy" one that was never started.
I suggest you give your #JmsListener an id, #Autowire the JmsListenerEndpointRegistry and call registry.getListenerContainer("myListener").stop().
#JmsListener(id = "myListener", destination = "${mq.queueName}")
I'm using Spring AMQP to set up remoting between different services, as described here. However, as I set a reply-timeout on my configuration, the first ever request always fails because the time taken to declare the queues, exchanges and binding exceeds the timeout:
The RabbitAdmin component can declare exchanges, queues and bindings
on startup. It does this lazily, through a ConnectionListener, so if
the broker is not present on startup it doesn't matter. The first time
a Connection is used (e.g. by sending a message) the listener will
fire and the admin features will be applied.
Is there any way the declaration can be made eagerly on startup instead of being made on the very first publish event to prevent the first request from always failing?
If you declare your queues with annotations:
#Configuration
public class QueuesConfiguration {
#Bean
public FanoutExchange exchange() {
return new FanoutExchange("exchange", true, false);
}
#Bean
public Binding binding() {
return BindingBuilder.bind(queue()).to(exchange());
}
#Bean
public Queue queue() {
return new Queue("queue");
}
#Bean
public RabbitAdmin rabbitAdmin(ConnectionFactory connectionFactory) {
return new RabbitAdmin(connectionFactory);
}
}
then call the RabbitAdmin.initialize() manually on application startup with this:
#Component
public class ApplicationStartup implements ApplicationListener<ApplicationReadyEvent> {
#Autowired
private RabbitAdmin rabbitAdmin;
#Override
public void onApplicationEvent(final ApplicationReadyEvent event) {
rabbitAdmin.initialize();
}
}
As we see by that description and the code from the RabbitAdmin, the last one just populates the ConnectionListener to the provided ConnectionFactory.
That ConnectionListener.onCreate is invoked from the ConnectionFactory.createConnection().
So, you can handle, for example, ContextRefreshedEvent and just do void connectionFactory.createConnection() eagerly.
From other side RabbitAdmin has initialize() public method for the same purpose.
UPDATE
Actually ListenerContainer does that on its start() too. You must declare your queues, exchanges and binding in the app where you a have a listener and make it autoStartup = true. To be honest the listener app is responsible for the real Broker entities.
The sending app should get deal just only with exchangeName and routingKey.