I'm trying to setup a headers exchange with a queue where messages are routed based on a recipient header.
The exchange is of type headers.
So far the class is able to connect to the exchange and feed messages to the queue.
It's also able to subscribe to the queue and receive messages. It also closes the connection whenever the subscriber's connection is cancelled.
The current problem is that the message is not routed by the recipient's header value.
Given the following class:
import com.rabbitmq.client.*;
import lombok.extern.slf4j.Slf4j;
import reactor.core.publisher.Flux;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.TimeoutException;
#Slf4j
public class MyQueue {
private final ConnectionFactory connectionFactory;
private Channel channel;
public MyQueue(ConnectionFactory connectionFactory) {
this.connectionFactory = connectionFactory;
}
public String sendMessage(TestTextMessage message) throws UndeliverableMessageException {
try (Connection connection = connectionFactory.newConnection();
Channel channel = connection.createChannel()) {
Map<String, Object> headers = new HashMap<>();
headers.put(RabbitMqConfig.MATCH_HEADER, message.getRecipient());
AMQP.BasicProperties props = new AMQP.BasicProperties.Builder()
.deliveryMode(MessageProperties.PERSISTENT_TEXT_PLAIN.getDeliveryMode())
.priority(MessageProperties.PERSISTENT_TEXT_PLAIN.getPriority())
.headers(headers).build();
log.info("Sending message to {}", headers);
channel.basicPublish(RabbitMqConfig.EXCHANGE_NAME, "", props,
message.getMessage().getBytes(StandardCharsets.UTF_8));
log.info("RabbitMQ sent message {} to {}", message.getMessage(), message.getRecipient());
return "ok";
} catch (TimeoutException e) {
log.error("Rabbit mq timeout", e);
} catch (IOException e) {
log.error("Rabbit mq io error", e);
}
throw new UndeliverableMessageException();
}
public Flux<String> listenMessages(String recipient) throws IOException, TimeoutException {
Connection connection = connectionFactory.newConnection();
this.channel = connection.createChannel();
// The map for the headers.
Map<String, Object> headers = new HashMap<>();
headers.put("x-match", "all");
headers.put(RabbitMqConfig.MATCH_HEADER, recipient);
final String[] consumerTag = new String[1];
Flux<String> as = Flux.create(sink -> new MessageListener<String>() {
{
try {
log.info("Binding to {}", headers);
channel.queueBind(RabbitMqConfig.QUEUE_NAME, RabbitMqConfig.EXCHANGE_NAME, "",
headers);
DeliverCallback deliverCallback = (consumerTag, delivery) -> {
String message = new String(delivery.getBody(), StandardCharsets.UTF_8);
log.info("Subscriber {} received a message {} with headers {}", recipient, delivery.getEnvelope(),
delivery.getProperties().getHeaders());
sink.next(delivery.getEnvelope().getDeliveryTag() + "--" + message);
//channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
};
consumerTag[0] = channel.basicConsume(RabbitMqConfig.QUEUE_NAME,
true, deliverCallback, tag -> {
sink.complete();
});
} catch (IOException e) {
log.error("RabbitMQ IOException ", e);
}
}
});
return as.doOnCancel(() -> {
try {
if (consumerTag[0] == null) {
log.error("RabbitMQ uncloseable subscription, consumerTag is null!");
channel.close();
return;
}
channel.basicCancel(consumerTag[0]);
channel.close();
log.info("RabbitMQ CANCEL subscription for recipient {}", recipient);
} catch (IOException | TimeoutException e) {
log.error("RabbitMQ channel close error", e);
}
});
}
interface MessageListener<T> {
}
}
The exchange is declared by the following call
channel.exchangeDeclare(RabbitMqConfig.EXCHANGE_NAME, BuiltinExchangeType.HEADERS, true);
Binding recipient log:
Binding to {x-match=all, message-recipient=mary}
Binding to {x-match=all, message-recipient=james}
Binding to {x-match=all, message-recipient=john}
Bound 3 recipients with x-match:
However, messages are not matched, as if they were routed randomly
Sending message to {message-recipient=james}
RabbitMQ sent message Hey there to james
Subscriber mary received a message Envelope(deliveryTag=1, redeliver=false, exchange=my-exchange, routingKey=) with headers {message-recipient=james}
Sending message to {message-recipient=james}
RabbitMQ sent message Hey there to james
Subscriber james received a message Envelope(deliveryTag=1, redeliver=false, exchange=my-exchange, routingKey=) with headers {message-recipient=james}
Sending message to {message-recipient=james}
RabbitMQ sent message Hey there to james
Subscriber john received a message Envelope(deliveryTag=1, redeliver=false, exchange=my-exchange, routingKey=) with headers {message-recipient=james}
Why isn't x-match: all, matching?
After reading the comment posted by #Gryphon, on the subscriber side, I ended up creating a queue for each participant.
channel.queueDeclare(RabbitMqConfig.QUEUE_NAME + "-" + recipient,
true,
false,
false,
null)
On the publisher side, code remains unchanged, the messages are sent to the exchange, and the exchange will handle routing based on the x-match: all configuration, routing the messages to the appropiate queue.
Related
Currently I am making logic to consume Message using Rabbitmq. However, contrary to expectations, it takes too long to consume the message.
If you look at the Queued messages graph in the picture above, you can see Unacked and Ready stacking up.
Looking at the message rates below, the publish speed is fast, but the consumer ack speed is too slow.
I'm not sure if the Rabbitmq Configuration I've developed is wrong or if I forgot to set the listener configuration.
The rabbitmq message I receive is a callback message.
Any help would be greatly appreciated.
This is Rabbitmq configuration and RabbitListener configuration
#Configuration
#Profile({ProfileConfig.RABBITMQ})
public class RabbitmqConfig {
#Value("${rabbitmq.queue.name}")
private String queueName;
#Value("${rabbitmq.exchange.name}")
private String exchangeName;
#Value("${rabbitmq.routing.key.callback}")
private String routingKey;
#Value("${rabbitmq.fetch-count}")
private Integer fetchCount;
#Bean
Queue queue() {
return new Queue(queueName, true);
}
#Bean
DirectExchange directExchange() {
return new DirectExchange(exchangeName);
}
#Bean
Binding binding(DirectExchange directExchange, Queue queue) {
return BindingBuilder.bind(queue).to(directExchange).with(routingKey);
}
#Bean
public RabbitListenerContainerFactory<SimpleMessageListenerContainer> prefetchOneContainerFactory(
SimpleRabbitListenerContainerFactoryConfigurer configurer, ConnectionFactory factory)
{
SimpleRabbitListenerContainerFactory simpleFactory = new SimpleRabbitListenerContainerFactory();
configurer.configure(simpleFactory, factory);
simpleFactory.setPrefetchCount(fetchCount);
return simpleFactory;
}
}
#RabbitListener(queues = {"${rabbitmq.queue.name}"}, concurrency = "3", containerFactory = "prefetchOneContainerFactory")
public void receiveMessage(final String message, Channel channel, #Header(AmqpHeaders.DELIVERY_TAG) long tag) {
try {
JSONParser parser = new JSONParser();
JSONObject json = (JSONObject) parser.parse(message);
String messageType = json.get("messageType").toString();
log.debug("Receive Queue Key={}, Message = {}", messageType, message);
AsyncType asyncType = AsyncType.valueOf(messageType);
executeMessage(asyncType, message);
} catch (Exception e) {
traceService.removeTraceId();
traceService.printErrorLog(log, "Fail to deal receive message.", e, PrintStackPolicy.ALL);
} finally {
try {
channel.basicAck(tag, false);
}
catch (IOException e) {
traceService.printErrorLog(log, "Fail to send ack to RabbitMQ", e, PrintStackPolicy.ALL);
}
}
}
The goal is to consume messages to Rabbitmq faster.
However, the current consumption speed is too slow.
I try to receive a protobuf message out of RabbitMQ using Spring Integration.
My integration flow:
public class FacadeIntegrationFlowAdapter extends IntegrationFlowAdapter {
#SuppressWarnings("rawtypes")
private final Facade facade;
private final FacadeProperties facadeProperties;
#SuppressWarnings("unchecked")
#Override
protected IntegrationFlowDefinition<?> buildFlow() {
return from(facadeProperties.getQueueName())
.handle(facade::getNewMessages);
}
}
The getNewMessages method:
#Override
public ExchangeResponse getNewMessages(Message<ExchangeRequest> message) {
ExchangeRequest request = message.getPayload();
log.info("Receiving new message: " + request.toString());
This is how I send the message to the queue. It's so simple to make the test easy to follow.
ExchangeRequest request = ExchangeRequest.newBuilder()
.addAllAuthors(List.of("author1", "author2"))
.addAllBooks(List.of("book1", "book2"))
.build();
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setUsername("user");
connectionFactory.setPassword("password");
connectionFactory.setHost("localhost");
connectionFactory.setPort(24130);
try {
Connection connection = connectionFactory.newConnection();
Channel channel = connection.createChannel();
var basicProperties = new AMQP.BasicProperties().builder()
.contentType("application/x-protobuf")
.type(request.getDescriptorForType().getFullName())
.build();
channel.basicPublish(
"facade-exchange", "facade-routing-key", basicProperties, request.toByteArray());
} catch (IOException e) {
Unfortunately, I keep getting the exception:
com.google.protobuf.InvalidProtocolBufferException: Type of the Any message does not match the given class.
However, when I change the getNewMessages method to the following, all seems fine.
#Override
public ExchangeResponse getNewMessages(Message message) {
try {
Any payload = (Any) message.getPayload();
ByteString value = payload.getValue();
ExchangeRequest request = ExchangeRequest.parseFrom(value);
log.info("Receiving new message: " + request.toString());
Where do I make a mistake? Tx!
I am trying to receive JSON messages from a Solace JMS queue but I am not receiving any message. Below is my code
#Service
public class QueueConsumer {
final String QUEUE_NAME = "test.Request.Q.V01";
// Latch used for synchronizing between threads
final CountDownLatch latch = new CountDownLatch(1);
public void run(String... args) throws Exception {
String host = "test.solace.com";
String vpnName = "TEST_VPN";
String username = "testVpn";
String password = "test123";
System.out.printf("QueueConsumer is connecting to Solace messaging at %s...%n", host);
SolConnectionFactory connectionFactory = SolJmsUtility.createConnectionFactory();
connectionFactory.setHost(host);
connectionFactory.setVPN(vpnName);
connectionFactory.setUsername(username);
connectionFactory.setPassword(password);
connectionFactory.setDynamicDurables(true);
Connection connection = connectionFactory.createConnection();
Session session = connection.createSession(false, SupportedProperty.SOL_CLIENT_ACKNOWLEDGE);
System.out.printf("Connected to the Solace Message VPN '%s' with client username '%s'.%n", vpnName, username);
Queue queue = session.createQueue(QUEUE_NAME);
MessageConsumer messageConsumer = session.createConsumer(queue);
messageConsumer.setMessageListener(new MessageListener() {
#Override
public void onMessage(Message message) {
try {
if (message instanceof SolaceMsg) {
System.out.printf("TextMessage received: '%s'%n", ((SolaceMsg) message).getClass());
} else {
System.out.println("Message received.");
}
System.out.printf("Message Content:%n%s%n", SolJmsUtility.dumpMessage(message));
message.acknowledge();
latch.countDown(); // unblock the main thread
} catch (JMSException ex) {
System.out.println("Error processing incoming message.");
ex.printStackTrace();
}
}
});
System.out.println("Start receiving messages....");
connection.start();
System.out.println("Awaiting message...");
latch.await();
connection.stop();
messageConsumer.close();
session.close();
connection.close();
}
public static void main(String... args) throws Exception {
new QueueConsumer().run(args);
}
}
My message type is JSON ad below, and I have created a POJO for this.
{
"customerDetails": {
"customerID": "0001234",
"customerName": "John"
}
}
I am getting one warning saying Response - 400 Queue already exists as it is an existing queue, and I am not receiving any messages. What am I doing wrong here?
Your code snippet looks correct. You can log on to the PubSub+ Manager of your event broker to verify that the client is binding to the correct queue and that the messages were successfully published to the queue and are waiting to be consumed. You can also enable Solace JMS API logging to understand more about what the application is doing: https://docs.solace.com/Solace-JMS-API/Code-and-Compile-Guideli.htm
How to implement the following situation on Java with RabbitMQ:
where every node, send messages to all other nodes and every node receive messages from all other nodes ?
UPDATE 1:
I tried to create the above situation with the following code:
ReceiveLogs.java
public class ReciveLogs {
...
public void start() throws IOException, TimeoutException, InterruptedException {
connection = factory.newConnection();
channel = connection.createChannel();
channel.queueDeclare(coda, false, false, false, null);
channel.exchangeDeclare(exName, BuiltinExchangeType.FANOUT);
channel.queueBind(coda, exName, "");
channel.basicPublish(exName, codaX, null, message.getBytes("UTF-8"));
System.out.println(" ReceiveLogs Sent: " + message);
Consumer consumer = new DefaultConsumer(channel) {
#Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties,
byte[] body) throws IOException {
String message = new String(body, "UTF-8");
System.out.println(" ReceiveLogs RECEIVED:" + message);
}
};
channel.basicConsume(codaX, true, consumer);
}
}
EmitLog.java
public class EmitLog {
...
public void start() throws IOException, TimeoutException {
connection = factory.newConnection();
channel = connection.createChannel();
channel.exchangeDeclare(exName, BuiltinExchangeType.FANOUT);
channel.queueDeclare(codaX, false, false, false, null);
channel.queueBind(codaX, exName, "");
Consumer consumer = new DefaultConsumer(channel) {
#Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties,
byte[] body) throws IOException {
String message = new String(body, "UTF-8");
System.out.println(" ROUTER Received: '" + message);
}
};
String message = channel.basicConsume(codaX, true, consumer);
channel.basicPublish(exName, "", null, message.getBytes("UTF-8"));
System.out.println("ROUTER Sent: " + message);
channel.close();
connection.close();
}
}
You can achieve this by creating a "fanout" exchange.
This is the setup you need to do:
Create a fanount exchange
Create 3 queues, one for each node. Say Q1, Q2 and Q3 corresponding to C1, C2 and C3.
Bind all queues (Q1, Q2 and Q3) to the fanount exchange created in step 1
Listening Code:
Create a listener for each of the nodes. For example, C1 node listens for messages on Queue "Q1", C2 for "Q2" and so on.
Publishing:
- Whenever you want to send a message, publish the message on broadcast exchange you created.
There is small caveat here: If C1 publishes the message, then C1 receives the same message as well. So, if you don't any node to process the same message it published, then you can use one of the attributes in the message to filter it out.
Further documentation:
https://www.rabbitmq.com/tutorials/amqp-concepts.html#exchange-fanout
Does Apache Camel Mail Component support reuse of TCP connection for sending bulk emails?
If not, is it feasible/recommendable to write custom org.apache.camel.component.mail.JavaMailSender to do so?
I think it's not possible with the standard camel mail component and also implementing a JavaMailSender won't do the trick for you imo.
If you have a look at org.apache.camel.component.mail.MailProducer(2.13.0):
public void process(final Exchange exchange) {
try {
MimeMessage mimeMessage;
final Object body = exchange.getIn().getBody();
if (body instanceof MimeMessage) {
// Body is directly a MimeMessage
mimeMessage = (MimeMessage) body;
} else {
// Create a message with exchange data
mimeMessage = new MimeMessage(sender.getSession());
getEndpoint().getBinding().populateMailMessage(getEndpoint(), mimeMessage, exchange);
}
if (LOG.isDebugEnabled()) {
LOG.debug("Sending MimeMessage: {}", MailUtils.dumpMessage(mimeMessage));
}
sender.send(mimeMessage);
// set the message ID for further processing
exchange.getIn().setHeader(MailConstants.MAIL_MESSAGE_ID, mimeMessage.getMessageID());
} catch (MessagingException e) {
exchange.setException(e);
} catch (IOException e) {
exchange.setException(e);
}
}
There is only 1 message or exchange object send/handled.
So maybe implementing your own "BulkMailComponent" with the functionality you need can be a solution for you.