Here is a small Spring program that is expected to insert a message into a rabbitmq queue:
public class Main {
public static void main(String [] args) throws IOException {
AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext(QueueConfiguration.class);
AmqpTemplate template = context.getBean(AmqpTemplate.class);
template.convertAndSend("asdflk ...");
context.destroy();
}
}
The ApplicationContext is as follows:
#Configuration
public class QueueConfiguration {
#Bean
public ConnectionFactory connectionFactory() {
return new CachingConnectionFactory("192.168.1.39");
}
#Bean
public RabbitTemplate rabbitTemplate() {
return new RabbitTemplate(connectionFactory());
}
}
When I check the contents of the queues on the server, nothing gets inserted. I also tried to set the name of the exchange or the name of the queue on the RabbitTemplate, but still nothing shows up on the server.
The log of the application does not show any errors, but logs this:
17:28:02.441 [main] DEBUG o.s.amqp.rabbit.core.RabbitTemplate - Executing callback on RabbitMQ Channel: Cached Rabbit Channel: AMQChannel(amqp://guest#192.168.1.39:5672/,1)
17:28:02.441 [main] DEBUG o.s.amqp.rabbit.core.RabbitTemplate - Publishing message on exchange [], routingKey = []
Any ideas what's wrong?
I had to give the queue as a parameter in the call to convertAndSend():
template.convertAndSend("hello2", "asdflk ...");
Still wondering why spring-amqp would not throw an exception. Anybody knows where the messages are delivered when no queue is given?
I think I will keep the Routing Key and Queue name in the bean rabbitTemplate() as per spring-amqp example. Since I am working with multiple queues currently I have different class for each queue in which I have the rabbitTemplate like this:
#Bean
public RabbitTemplate rabbitTemplate() {
RabbitTemplate template = new RabbitTemplate(connectionFactory());
//The routing key = name of the queue in default exchange.
template.setRoutingKey("MyQueue");
// Queue name
template.setQueue("MyQueue");
return template;
}
Are you using tomcat to deploy this? If yes then these can be loaded at startup which will initialize all the connection/channel/queues etc as well.
Related
In my testing and review of the Artemis LastValueQueue code, it looks like the scheduling delay for a message takes precedence over its evaluation of the "last-value-key". In other words, if you schedule a message, it is only evaluated for replacing the last-value in the queue at the time it is prepared for delivery.
My question is whether I have correctly understood the code, and if so, if there's a workaround or a feature of ActiveMQ / Artemis that might help meet our requirements.
Our requirements are as follows:
Generate a message, and delay processing of that message to a point in the future (usually 30 seconds out).
If an updated version of the message is generated due to a new external event, replace any existing scheduled message with the new version of the message - the scheduled delivery time should also be updated, in addition to the message payload.
Some other notes:
My current prototype is using Artemis embedded server
Spring-jms JmsTemplate is being used to produce messages
Spring-jms JmsListenerContainerFactory is being used to consume messages
We don't currently use SpringBoot, so you'll see some bean setup below.
ArtemisConfig.java:
#Configuration
#EnableJms
public class ArtemisConfig {
#Bean
public org.apache.activemq.artemis.core.config.Configuration configuration() throws Exception {
org.apache.activemq.artemis.core.config.Configuration config = new ConfigurationImpl();
config.addAcceptorConfiguration("in-vm", "vm://0");
config.setPersistenceEnabled(true);
config.setSecurityEnabled(false);
config.setJournalType(JournalType.ASYNCIO);
config.setCreateJournalDir(true);
config.setJournalDirectory("/var/mq/journal");
config.setBindingsDirectory("/var/mq/bindings");
config.setLargeMessagesDirectory("/var/mq/large-messages");
config.setJMXManagementEnabled(true);
QueueConfiguration queueConfiguration = new QueueConfiguration("MYLASTVALUEQUEUE");
queueConfiguration.setAddress("MYLASTVALUEQUEUE");
queueConfiguration.setLastValueKey("uniqueJobId");
queueConfiguration.setDurable(true);
queueConfiguration.setEnabled(true);
queueConfiguration.setRoutingType(RoutingType.ANYCAST);
CoreAddressConfiguration coreAddressConfiguration = new CoreAddressConfiguration();
coreAddressConfiguration.addQueueConfiguration(queueConfiguration);
config.addAddressConfiguration(coreAddressConfiguration);
return config;
}
#Bean
public EmbeddedActiveMQ artemisServer() throws Exception {
EmbeddedActiveMQ server = new EmbeddedActiveMQ();
server.setConfiguration(configuration());
server.start();
return server;
}
#PreDestroy
public void preDestroy() throws Exception {
artemisServer().stop();
}
#Bean
public ConnectionFactory activeMqConnectionFactory() throws Exception {
return ActiveMQJMSClient.createConnectionFactory("vm://0", "artemis-client");
}
#Bean
public JmsListenerContainerFactory<?> jmsListenerContainerFactory() throws Exception {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(activeMqConnectionFactory());
factory.setSessionTransacted(true);
factory.setConcurrency("8");
factory.setMessageConverter(jacksonJmsMessageConverter());
return factory;
}
#Bean
public MessageConverter jacksonJmsMessageConverter() {
MappingJackson2MessageConverter converter = new MappingJackson2MessageConverter();
converter.setTargetType(MessageType.TEXT);
converter.setTypeIdPropertyName("_type");
return converter;
}
#Bean
public JmsTemplate jmsTemplate() throws Exception {
JmsTemplate jmsTemplate = new JmsTemplate(activeMqConnectionFactory());
jmsTemplate.setMessageConverter(jacksonJmsMessageConverter());
jmsTemplate.setDeliveryPersistent(true);
return jmsTemplate;
}
#Bean
QueueMessageService queueMessageService() {
return new QueueMessageService();
}
}
QueueMessageService.java
public class QueueMessageService {
#Resource
private JmsTemplate jmsTemplate;
public void queueJobRequest(
final String queue,
final int priority,
final long deliveryDelayInSeconds,
final MyMessage message) {
jmsTemplate.convertAndSend(queue, jobRequest, message -> {
message.setJMSPriority(priority);
if (deliveryDelayInSeconds > 0 && deliveryDelayInSeconds <= 86400) {
message.setLongProperty(
Message.HDR_SCHEDULED_DELIVERY_TIME.toString(),
Instant.now().plus(deliveryDelayInSeconds, ChronoUnit.SECONDS).toEpochMilli()
);
}
message.setStringProperty(Message.HDR_LAST_VALUE_NAME.toString(), "uniqueJobId");
message.setStringProperty("uniqueJobId", jobRequest.getUniqueJobId().toString());
return message;
});
}
}
Your understanding about the semantics of scheduled messages with a last-value queue is correct. When a message is scheduled it is not technically on the queue yet. It is not put onto the queue until the scheduled time arrives at which point last-value queue semantics are enforced.
Short of implementing a new feature I don't see how you can implement your desired behavior in any kind of automatic way. My recommendation at this point would be to use the management API (i.e. QueueControl) to manually remove the "old" scheduled message before you send the "new" scheduled message. You can use one of the removeMessage methods for this as they will work on scheduled messages and non-scheduled messages alike.
We're using RabbitMq for communication between some of our services. Sometimes there are a lot of messages beeing queued at once. We want to be able to see that there are still unhandled messages, i.e. if the Service handling the messages is busy.
I've been looking around for a programmatical way to check if a queue has messages and found this
channel.queueDeclarePassive(queueName).getMessageCount()
The problem is: I dont have a channel object. Our RabbitMq setup has been created a couple of years ago and usually looks like this:
#Configuration
#EnableRabbit
public class RabbitMqConfig {
public static final String RENDER_HTML_QUEUE = "render.html";
private String rabbitUri;
private int connectionTimeout;
private String exchangeName;
private int concurrentConsumers;
public RabbitMqConfig(
#Value("${rabbitmq.uri}") String rabbitUri,
#Value("${rabbitmq.exchange.name}") String exchangeName,
#Value("${rabbitmq.connection.timeout}") int timeout,
#Value("${rabbitmq.concurrent-consumers:1}") int concurrentConsumers) {
this.exchangeName = exchangeName;
this.rabbitUri = rabbitUri;
this.connectionTimeout = timeout;
this.concurrentConsumers = concurrentConsumers;
}
#Bean
DirectExchange directExchangeBean() {
return new DirectExchange(this.exchangeName, true, false);
}
#Bean
SimpleMessageListenerContainer container(ConnectionFactory connectionFactory, MessageListenerAdapter listenerAdapter) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames(RENDER_HTML_QUEUE);
container.setConcurrentConsumers(concurrentConsumers);
container.setMessageListener(listenerAdapter);
return container;
}
#Bean
MessageListenerAdapter listenerAdapter(RenderItemMessageConsumer receiver) {
return new MessageListenerAdapter(receiver, "reciveMessageFromRenderQueue");
}
#Bean
public ConnectionFactory connectionFactory() {
CachingConnectionFactory connectionFactory;
try {
connectionFactory = new CachingConnectionFactory(new URI(this.rabbitUri));
connectionFactory.setConnectionTimeout(this.connectionTimeout);
} catch (URISyntaxException e) {
throw new ApiException(e, BaseErrorCode.UNKOWN_ERROR, e.getMessage());
}
return connectionFactory;
}
#Bean
public AmqpAdmin amqpAdmin() {
return new RabbitAdmin(connectionFactory());
}
#Bean
public RabbitTemplate rabbitTemplate() {
return new RabbitTemplate(connectionFactory());
}
#Bean
public Queue renderRenderQueue() {
return new Queue(RENDER_HTML_QUEUE, true);
}
#Bean
Binding rendererRenderBinding() {
return BindingBuilder.bind(renderRenderQueue()).to(directExchangeBean()).with(
RENDER_HTML_QUEUE);
}
}
Messages are then consumed like this:
#Component
public class RenderItemMessageConsumer {
#RabbitListener(queues = RENDER_HTML_QUEUE)
public void reciveMessageFromRenderQueue(String message) {
//...
}
The exchangeName is shared across services. So generally I need a way to get the channel that probably is created for the queue and connection to see how many messages are inside. Ideally I want to access that information at the other service that produces the messages consumed in the rendering service.
Or am I doing something wrong? Do I have to explicitly create a channel and connect the queue to it? I'm not even sure what channels are created under the hood, as I mentioned I've set this up some years ago and didnt dig deeper after everything was running fine.
Can I maybe somehow use the amqpAdmin to get all channels?
Turns out I can answer my own question after just a little more trying around:
#Autowired
private ConnectionFactory connectionFactory;
public Boolean isBusy(String queue) throws IOException {
Connection connection = connectionFactory.createConnection();
Channel channel = connection.createChannel(false);
return channel.queueDeclarePassive(queue).getMessageCount() > 0;
}
Since all my services have a similar setup exposing the connectionFactory as a bean and they all connect to the shared rabbitMq server using the same exchange name, I can just use any service to do the above. I can put that snippet behind a rest resource and thus from my management UI can request all the information about the queues of which I know the names to decide if I want to post another batch of messages to it.
So I'm diving deeper into the world of JMS.
I am writing some dummy projects right now and understanding how to consume messages. I am using Active MQ artemis as the message broker.
Whilst following a tutorial, I stumbled upon something in terms on consuming messages. What exactly is the difference between a message listener to listen for messages and using the #JmsListener annotion?
This is what I have so far:
public class Receiver {
#JmsListener(containerFactory = "jmsListenerContainerFactory", destination = "helloworld .q")
public void receive(String message) {
System.out.println("received message='" + message + "'.");
}
}
#Configuration
#EnableJms
public class ReceiverConfig {
#Value("${artemis.broker-url}")
private String brokerUrl;
#Bean
public ActiveMQConnectionFactory activeMQConnectionFactory(){
return new ActiveMQConnectionFactory(brokerUrl);
}
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory(){
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(activeMQConnectionFactory());
factory.setConcurrency("3-10");
return factory;
}
#Bean
public DefaultMessageListenerContainer orderMessageListenerContainer() {
SimpleJmsListenerEndpoint endpoint =
new SimpleJmsListenerEndpoint();
endpoint.setMessageListener(new StatusMessageListener("DMLC"));
endpoint.setDestination("helloworld.q"); //Try renaming this and see what happens.
return jmsListenerContainerFactory()
.createListenerContainer(endpoint);
}
#Bean
public Receiver receiver() {
return new Receiver();
}
}
public class StatusMessageListener implements MessageListener {
public StatusMessageListener(String dmlc) {
}
#Override
public void onMessage(Message message) {
System.out.println("In the onMessage().");
System.out.println(message);
}
}
From what I've read is that we register a message listener to the container listener which in turn is created by the listener factory. So essentially the flow is this:
DefaultJmsListenerContainerFactory -> creates -> DefaultMessageListenerContainer -> registers a message listener which is used to listen to messages from the endpoint configured.
From my research, i've gathered that messageListeners are used to asynchornously consume messages from the queues/topic whilst using the #JmsListener annotation is used to synchronously listen to messages?
Furthermore, there's a few other ListenerContainerFactory out there such as DefaultJmsListenerContainerFactory and SimpleJmsListenerContainerFactory but not sure I get the difference. I was reading https://codenotfound.com/spring-jms-listener-example.html and from what I've gathered from that is Default uses a pull model so that suggests it's async so why would it matter if we consume the message via a messageListener or the annotation? I'm a bit confused and muddled up so would like my doubts to be cleared up. Thanks!
This is the snippet of the program when sending 100 dummy messages (just noticed it's not outputting the even numbered messages..):
received message='This the 95 message.'.
In the onMessage().
ActiveMQMessage[ID:006623ca-d42a-11ea-a68e-648099ad9459]:PERSISTENT/ClientMessageImpl[messageID=24068, durable=true, address=helloworld.q,userID=006623ca-d42a-11ea-a68e-648099ad9459,properties=TypedProperties[__AMQ_CID=00651257-d42a-11ea-a68e-648099ad9459,_AMQ_ROUTING_TYPE=1]]
received message='This the 97 message.'.
In the onMessage().
ActiveMQMessage[ID:006ba214-d42a-11ea-a68e-648099ad9459]:PERSISTENT/ClientMessageImpl[messageID=24088, durable=true, address=helloworld.q,userID=006ba214-d42a-11ea-a68e-648099ad9459,properties=TypedProperties[__AMQ_CID=0069cd51-d42a-11ea-a68e-648099ad9459,_AMQ_ROUTING_TYPE=1]]
received message='This the 99 message.'.
The following configuration
#Configuration
#EnableJms
public class ReceiverConfig {
//your config code here..
}
would ensure that every time a Message is received on the Destination named "helloworld .q", Receiver.receive() is called with the content of the message.
You can read more here: https://docs.spring.io/spring/docs
Environment
Spring Boot: 1.5.13.RELEASE
Cloud: Edgware.SR3
Cloud AWS: 1.2.2.RELEASE
Java 8
OSX 10.13.4
Problem
I am trying to write an integration test for SQS.
I have a local running localstack docker container with SQS running on TCP/4576
In my test code I define an SQS client with the endpoint set to local 4576 and can successfully connect and create a queue, send a message and delete a queue. I can also use the SQS client to receive messages and pick up the message that I sent.
My problem is that if I remove the code that is manually receiving the message in order to allow another component to get the message nothing seems to be happening. I have a spring component annotated as follows:
Listener
#Component
public class MyListener {
#SqsListener(value = "my_queue", deletionPolicy = ON_SUCCESS)
public void receive(final MyMsg msg) {
System.out.println("GOT THE MESSAGE: "+ msg.toString());
}
}
Test
#RunWith(SpringRunner.class)
#SpringBootTest(properties = "spring.profiles.active=test")
public class MyTest {
#Autowired
private AmazonSQSAsync amazonSQS;
#Autowired
private SimpleMessageListenerContainer container;
private String queueUrl;
#Before
public void setUp() {
queueUrl = amazonSQS.createQueue("my_queue").getQueueUrl();
}
#After
public void tearDown() {
amazonSQS.deleteQueue(queueUrl);
}
#Test
public void name() throws InterruptedException {
amazonSQS.sendMessage(new SendMessageRequest(queueUrl, "hello"));
System.out.println("isRunning:" + container.isRunning());
System.out.println("isActive:" + container.isActive());
System.out.println("isRunningOnQueue:" + container.isRunning("my_queue"));
Thread.sleep(30_000);
System.out.println("GOT MESSAGE: " + amazonSQS.receiveMessage(queueUrl).getMessages().size());
}
#TestConfiguration
#EnableSqs
public static class SQSConfiguration {
#Primary
#Bean(destroyMethod = "shutdown")
public AmazonSQSAsync amazonSQS() {
final AwsClientBuilder.EndpointConfiguration endpoint = new AwsClientBuilder.EndpointConfiguration("http://127.0.0.1:4576", "eu-west-1");
return new AmazonSQSBufferedAsyncClient(AmazonSQSAsyncClientBuilder
.standard()
.withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials("key", "secret")))
.withEndpointConfiguration(endpoint)
.build());
}
}
}
In the test logs I see:
o.s.c.a.m.listener.QueueMessageHandler : 1 message handler methods found on class MyListener: {public void MyListener.receive(MyMsg)=org.springframework.cloud.aws.messaging.listener.QueueMessageHandler$MappingInformation#1cd4082a}
2018-05-31 22:50:39.582 INFO 16329 ---
o.s.c.a.m.listener.QueueMessageHandler : Mapped "org.springframework.cloud.aws.messaging.listener.QueueMessageHandler$MappingInformation#1cd4082a" onto public void MyListener.receive(MyMsg)
Followed by:
isRunning:true
isActive:true
isRunningOnQueue:false
GOT MESSAGE: 1
This demonstrates that in the 30 second pause between sending the message the container didn't pick it up and when I manually poll for the message it is there on the queue and I can consume it.
My question is, why isn't the listener being invoked and why is the isRunningOnQueue:false line suggesting that it's not auto started for that queue?
Note that I also tried setting my own SimpleMessageListenerContainer bean with autostart set to true explicitly (the default anyway) and observed no change in behaviour. I thought that the org.springframework.cloud.aws.messaging.config.annotation.SqsConfiguration#simpleMessageListenerContainer that is set up by #EnableSqs ought to configure an auto started SimpleMessageListenerContainer that should be polling for me message.
I have also set
logging.level.org.apache.http=DEBUG
logging.level.org.springframework.cloud=DEBUG
in my test properties and can see the HTTP calls create the queue, send a message and delete etc but no HTTP calls to receive (apart from my manual one at the end of the test).
I figured this out after some tinkering.
Even if the simple message container factory is set to not auto start, it seems to do its initialisation anyway, which involves determining whether the queue exists.
In this case, the queue is created in my test in the setup method - but sadly this is after the spring context is set up which means that an exception occurs.
I fixed this by simply moving the queue creation to the context creation of the SQS client (which happens before the message container is created). i.e.:
#Bean(destroyMethod = "shutdown")
public AmazonSQSAsync amazonSQS() {
final AwsClientBuilder.EndpointConfiguration endpoint = new AwsClientBuilder.EndpointConfiguration("http://localhost:4576", "eu-west-1");
final AmazonSQSBufferedAsyncClient client = new AmazonSQSBufferedAsyncClient(AmazonSQSAsyncClientBuilder
.standard()
.withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials("dummyKey", "dummySecret")))
.withEndpointConfiguration(endpoint)
.build());
client.createQueue("test-queue");
return client;
}
I wrote the following configuration:
#Slf4j
#Configuration
#EnableConfigurationProperties(BatchProperties.class)
public class BatchConfiguration {
#Autowired
private BatchProperties properties;
#Bean
public PollableAmqpChannel testingChannel(final RabbitTemplate rabbitTemplate) {
final PollableAmqpChannel channel = new PollableAmqpChannel(properties.getQueue(), rabbitTemplate);
channel.setLoggingEnabled(false);
return channel;
}
#Bean
#ServiceActivator(inputChannel = "testingChannel", poller = #Poller(fixedRate = "1000", maxMessagesPerPoll = "1"))
public MessageHandler messageHandler(final RabbitTemplate rabbitTemplate) {
return message -> {
log.info("Received: {}", message);
rabbitTemplate.convertAndSend(properties.getQueue(), message);
};
}
}
Message gets successfully read and requeued but I keep getting the following message:
Calling receive with a timeout value on PollableAmqpChannel. The
timeout will be ignored since no receive timeout is supported.
I am using Spring Boot 1.5.3.RELASE.
I've put a breakpoint on :
#Override
public void setLoggingEnabled(boolean loggingEnabled) {
this.loggingEnabled = loggingEnabled;
}
in the AbstractAmqpChannel class. It gets called with 'false' (as it should according to my configuration) and then it gets called again each time it polls a message and it is set to 'true'.
I checked the hashcode and it seems it is my bean, but 'loggingEnabled' gets reset to 'true' with each message.
Is there something wrong with my configuration and how can I fix this?
It looks like a bug - if you have spring-integration-jmx on the classpath, or specify #EnableIntegrationManagement on one of your config classes; the IntegrationManagementConfigurer bean sets logging enabled to true for all IntegrationManagement implementations, overwriting your setting.
Please open a JIRA Issue.
In the meantime, you could add a bean that implements SmartLifecyle to set the flag back to false (in the start() method); it will run after the IntegrationManagementConfigurer.
Of course, you could also set the log category org.springframework.integration.amqp.channel.PollableAmqpChannel to WARN to achieve the same effect much simpler.