I use MQ in my project via SpringJMS, as a broker I use ActiveMQ.
I need to set expiration message-based, so I tried to used message.setJMSExpiration but without success. All messages coming to ActiveMQ have expiration=0.
Does anyone has success with setting Expiration per message using Spring?
For configuring JmsTemplate I used default value explicitQosEnabled = false; so I expected to keep expiration from my Message props. But as I see in ActiveMQSession.class this message properties will be override:
long expiration = 0L;
if (!producer.getDisableMessageTimestamp()) {
long timeStamp = System.currentTimeMillis();
message.setJMSTimestamp(timeStamp);
if (timeToLive > 0) {
expiration = timeToLive + timeStamp;
}
}
message.setJMSExpiration(expiration);
//me: timeToLive coming from default values of Producer/JmsTemplate...
What I am doing wrong ? or it is just impossible with this tools.
I don't know why Spring decided to exclude this, but you can extend JmsTemplate and overload some methods, passing a timeToLive argument.
public class MyJmsTemplate extends JmsTemplate {
public void send(final Destination destination,
final MessageCreator messageCreator, final long timeToLive)
throws JmsException {
execute(new SessionCallback<Object>() {
public Object doInJms(Session session) throws JMSException {
doSend(session, destination, messageCreator, timeToLive);
return null;
}
}, false);
}
protected void doSend(Session session, Destination destination,
MessageCreator messageCreator, long timeToLive) throws JMSException {
Assert.notNull(messageCreator, "MessageCreator must not be null");
MessageProducer producer = createProducer(session, destination);
try {
Message message = messageCreator.createMessage(session);
if (logger.isDebugEnabled()) {
logger.debug("Sending created message: " + message);
}
doSend(producer, message, timeToLive);
// Check commit - avoid commit call within a JTA transaction.
if (session.getTransacted() && isSessionLocallyTransacted(session)) {
// Transacted session created by this template -> commit.
JmsUtils.commitIfNecessary(session);
}
} finally {
JmsUtils.closeMessageProducer(producer);
}
}
protected void doSend(MessageProducer producer, Message message,
long timeToLive) throws JMSException {
if (isExplicitQosEnabled() && timeToLive > 0) {
producer.send(message, getDeliveryMode(), getPriority(), timeToLive);
} else {
producer.send(message);
}
}
}
JMSExpiration is not the way to set an expiration. See the javadocs for Message...
JMS providers set this field when a message is sent. This method can be used to change the value for a message that has been received.
In other words, it's ignored on a send - the time to live is set on the producer.send() method.
To expire a message set explicitQosEnabled to true and setTimeToLive(...).
Related
I've configured an SQS queue and an additional dead letter queue using terraform.
resource "aws_sqs_queue" "sqs_deadletter" {
name = "worker-dead-letter"
}
resource "aws_sqs_queue" "sqs" {
name = "worker"
/* TODO: If I enable this all messages goes to the dead letter queue
redrive_policy = jsonencode({
deadLetterTargetArn = aws_sqs_queue.sqs_deadletter.arn
maxReceiveCount = 4
})
*/
}
resource "aws_lambda_event_source_mapping" "sqs" {
event_source_arn = aws_sqs_queue.sqs.arn
function_name = aws_lambda_function.worker.arn
enabled = true
batch_size = var.batch_size
}
I use the below handler to process my messages.
#Introspected
class LegacyToModernRequestHandler : MicronautRequestHandler<SQSEvent, Unit>() {
private val logger = KotlinLogging.logger {}
override fun execute(input: SQSEvent) {
input.records.forEach {
handle(it)
}
}
private fun handle(message: SQSMessage) {
val key = message.body
logger.info { "LegacyToModernRequestHandler($key)" }
}
}
But all my messages goes to the DLQ. How can I indicate successful handling so that doesn't happen?
If you are not using AUTO_ACKNOWLEDGEMENT mode, you will have to explicitly acknowledge the message so that it is processed successfully. Otherwise it will go to DLQ. Can you show how have you configured your SQS queue?
I'm using BatchingRabbitTemplate to send messages in a batch to amqp endpoint. Now, on the other receiving end, I can use #RabbitListener to receive messages, but my problem is that messages are automatically de-batched so I cannot use #RabbitHandler public void receive (List<SomeObject> so). Is there any simpler way of non-de-batching messages except me doing this:
#RabbitListener(..., containerFactory = "nonDeBatchingContainerFactory")
#Bean
public RabbitListenerContainerFactory nonDeBatchingContainerFactory(){
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory());
factory.setDeBatchingEnabled(false);
factory.setMessageConverter(jackson2JsonMessageConverter());
factory.setAfterReceivePostProcessors(new NonDeBatchingMessagePostProcessor(jackson2JsonMessageConverter()));
return factory;
}
and then implementing this post-processor (that is more or less copy of existing code for de-batching).
public class NonDeBatchingMessagePostProcessor implements MessagePostProcessor {
private MessageConverter payloadConverter;
public NonDeBatchingMessagePostProcessor(MessageConverter payloadConverter) {
this.payloadConverter = payloadConverter;
}
#Override
public Message postProcessMessage(Message message) throws AmqpException {
Object batchFormat = message.getMessageProperties().getHeaders().get(MessageProperties.SPRING_BATCH_FORMAT);
if (MessageProperties.BATCH_FORMAT_LENGTH_HEADER4.equals(batchFormat)) {
List<? super Object> aggregatedObjects = new ArrayList<>();
ByteBuffer byteBuffer = ByteBuffer.wrap(message.getBody());
MessageProperties messageProperties = message.getMessageProperties();
String singleObjectTypeId = messageProperties.getHeaders().get(DEFAULT_CLASSID_FIELD_NAME).toString();
messageProperties.getHeaders().remove(MessageProperties.SPRING_BATCH_FORMAT);
while (byteBuffer.hasRemaining()) {
int length = byteBuffer.getInt();
if (length < 0 || length > byteBuffer.remaining()) {
throw new ListenerExecutionFailedException("Bad batched message received",
new MessageConversionException("Insufficient batch data at offset " + byteBuffer.position()),
message);
}
byte[] body = new byte[length];
byteBuffer.get(body);
messageProperties.setContentLength(length);
// Caveat - shared MessageProperties.
Message fragment = new Message(body, messageProperties);
Object singleObject = this.payloadConverter.fromMessage(fragment);
aggregatedObjects.add(singleObject);
}
Message aggregatedMessages = this.payloadConverter.toMessage(aggregatedObjects, messageProperties);
aggregatedMessages.getMessageProperties().getHeaders().put(DEFAULT_CONTENT_CLASSID_FIELD_NAME, singleObjectTypeId);
return aggregatedMessages;
}
return null;
}
}
I need this use case in order to receive all messages in batch on the rabbit and then do bulk indexing in elastic search. Thanks.
It might be a bit easier to do the batching at the producing application level (send a List<SomeObject>) rather than using the batching template. Then you won't need anything at all on the consumer side.
Play 2.5 Highlights states
Better control over WebSocket frames
The Play 2.5 WebSocket API gives you direct control over WebSocket frames. You can now send and receive binary, text, ping, pong and close frames. If you don’t want to worry about this level of detail, Play will still automatically convert your JSON or XML data into the right kind of frame.
However
https://www.playframework.com/documentation/2.5.x/JavaWebSockets has examples around LegacyWebSocket which is deprecated
What is the recommended API/pattern for Java WebSockets? Is using
LegacyWebSocket the only option for java websockets?
Are there any examples using new Message types ping/pong to implement a heartbeat?
The official documentation on this is disappointingly very sparse. Perhaps in Play 2.6 we'll see an update to this. However, I will provide an example below on how to configure a chat websocket in Play 2.5, just to help out those in need.
Setup
AController.java
#Inject
private Materializer materializer;
private ActorRef chatSocketRouter;
#Inject
public AController(#Named("chatSocketRouter") ActorRef chatInjectedActor) {
this.chatSocketRouter = chatInjectedActor;
}
// Make a chat websocket for a user
public WebSocket chatSocket() {
return WebSocket.Json.acceptOrResult(request -> {
String authToken = getAuthToken();
// Checking of token
if (authToken == null) {
return forbiddenResult("No [authToken] supplied.");
}
// Could we find the token in the database?
final AuthToken token = AuthToken.findByToken(authToken);
if (token == null) {
return forbiddenResult("Could not find [authToken] in DB. Login again.");
}
User user = token.getUser();
if (user == null) {
return forbiddenResult("You are not logged in to view this stream.");
}
Long userId = user.getId();
// Create a function to be run when we initialise a flow.
// A flow basically links actors together.
AbstractFunction1<ActorRef, Props> getWebSocketActor = new AbstractFunction1<ActorRef, Props>() {
#Override
public Props apply(ActorRef connectionProperties) {
// We use the ActorRef provided in the param above to make some properties.
// An ActorRef is a fancy word for thread reference.
// The WebSocketActor manages the web socket connection for one user.
// WebSocketActor.props() means "make one thread (from the WebSocketActor) and return the properties on how to reference it".
// The resulting Props basically state how to construct that thread.
Props properties = ChatSocketActor.props(connectionProperties, chatSocketRouter, userId);
// We can have many connections per user. So we need many ActorRefs (threads) per user. As you can see from the code below, we do exactly that. We have an object called
// chatSocketRouter which holds a Map of userIds -> connectionsThreads and we "tell"
// it a lightweight object (UserMessage) that is made up of this connecting user's ID and the connection.
// As stated above, Props are basically a way of describing an Actor, or dumbed-down, a thread.
// In this line, we are using the Props above to
// reference the ActorRef we've just created above
ActorRef anotherUserDevice = actorSystem.actorOf(properties);
// Create a lightweight object...
UserMessage routeThisUser = new UserMessage(userId, anotherUserDevice);
// ... to tell the thread that has our Map that we have a new connection
// from a user.
chatSocketRouter.tell(routeThisUser, ActorRef.noSender());
// We return the properties to the thread that will be managing this user's connection
return properties;
}
};
final Flow<JsonNode, JsonNode, ?> jsonNodeFlow =
ActorFlow.<JsonNode, JsonNode>actorRef(getWebSocketActor,
100,
OverflowStrategy.dropTail(),
actorSystem,
materializer).asJava();
final F.Either<Result, Flow<JsonNode, JsonNode, ?>> right = F.Either.Right(jsonNodeFlow);
return CompletableFuture.completedFuture(right);
});
}
// Return this whenever we want to reject a
// user from connecting to a websocket
private CompletionStage<F.Either<Result, Flow<JsonNode, JsonNode, ?>>> forbiddenResult(String msg) {
final Result forbidden = Results.forbidden(msg);
final F.Either<Result, Flow<JsonNode, JsonNode, ?>> left = F.Either.Left(forbidden);
return CompletableFuture.completedFuture(left);
}
ChatSocketActor.java
public class ChatSocketActor extends UntypedActor {
private final ActorRef out;
private final Long userId;
private ActorRef chatSocketRouter;
public ChatSocketActor(ActorRef out, ActorRef chatSocketRouter, Long userId) {
this.out = out;
this.userId = userId;
this.chatSocketRouter = chatSocketRouter;
}
public static Props props(ActorRef out, ActorRef chatSocketRouter, Long userId) {
return Props.create(ChatSocketActor.class, out, chatSocketRouter, userId);
}
// Add methods here handling each chat connection...
}
ChatSocketRouter.java
public class ChatSocketRouter extends UntypedActor {
public ChatSocketRouter() {}
// Stores userIds to websockets
private final HashMap<Long, List<ActorRef>> senders = new HashMap<>();
private void addSender(Long userId, ActorRef actorRef){
if (senders.containsKey(userId)) {
final List<ActorRef> actors = senders.get(userId);
actors.add(actorRef);
senders.replace(userId, actors);
} else {
List<ActorRef> l = new ArrayList<>();
l.add(actorRef);
senders.put(userId, l);
}
}
private void removeSender(ActorRef actorRef){
for (List<ActorRef> refs : senders.values()) {
refs.remove(actorRef);
}
}
#Override
public void onReceive(Object message) throws Exception {
ActorRef sender = getSender();
// Handle messages sent to this 'router' here
if (message instanceof UserMessage) {
UserMessage userMessage = (UserMessage) message;
addSender(userMessage.userId, userMessage.actorRef);
// Watch sender so we can detect when they die.
getContext().watch(sender);
} else if (message instanceof Terminated) {
// One of our watched senders has died.
removeSender(sender);
} else {
unhandled(message);
}
}
}
Example
Now whenever you want to send a client with a websocket connection a message you can do something like:
ChatSenderController.java
private ActorRef chatSocketRouter;
#Inject
public ChatSenderController(#Named("chatSocketRouter") ActorRef chatInjectedActor) {
this.chatSocketRouter = chatInjectedActor;
}
public static void sendMessage(Long sendToId) {
// E.g. send the chat router a message that says hi
chatSocketRouter.tell(new Message(sendToId, "Hi"));
}
ChatSocketRouter.java
#Override
public void onReceive(Object message) throws Exception {
// ...
if (message instanceof Message) {
Message messageToSend = (Message) message;
// Loop through the list above and send the message to
// each connection. For example...
for (ActorRef wsConnection : senders.get(messageToSend.getSendToId())) {
// Send "Hi" to each of the other client's
// connected sessions
wsConnection.tell(messageToSend.getMessage());
}
}
// ...
}
Again, I wrote the above to help out those in need. After scouring the web I could not find a reasonable and simple example. There is an open issue for this exact topic. There are also some examples online but none of them were easy to follow. Akka has some great documentation but mixing it in with Play was a tough mental task.
Please help improve this answer if you see anything that is amiss.
I am adding messages to rabbitmq queue using spring amqp template in my spring batch item writer.
public class AmqpAsynchRpcItemWriter<T> implements ItemWriter<T> {
protected String exchange;
protected String routingKey;
protected String queue;
protected String replyQueue;
protected RabbitTemplate template;
BlockingQueue<Object> blockingQueue;
public void onMessage(Object msgContent) {
try {
blockingQueue.put(msgContent);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
#Override
public void write(List<? extends T> items) throws Exception {
for (T item : items) {
Message message = MessageBuilder
.withBody(item.toString().getBytes())
.setContentType(MessageProperties.CONTENT_TYPE_TEXT_PLAIN)
.setReplyTo(this.replyQueue)
.setCorrelationId(item.toString().getBytes()).build();
template.send(this.exchange, this.routingKey, message);
}
for (T item : items) {
Object msg = blockingQueue.poll(60, TimeUnit.SECONDS);
if (msg instanceof Exception) {
throw (Exception) msg;
} else if (msg == null) {
System.out.println("reply timeout...");
break;
}
}
}
}
Messages are going to be processed on different remote servers. I am trying to handle the use case where if my message processing is failed (due to some exception) the step execution will be stopped.
I want to purge all the remaining messages in that queue so that remaining messages in queue should not be consumed and processed as they will also be failed.
If the step is failed, my item writer will again queue all the messages, so I need to purge all remaining message on any exception.
How can I purge the queue using spring amqp ?
I could do it using
admin.purgeQueue(this.queue, true);
I would use RabbitAdmin instead
http://docs.spring.io/autorepo/docs/spring-amqp-dist/1.3.4.RELEASE/api/org/springframework/amqp/rabbit/core/RabbitAdmin.html#purgeQueue%28java.lang.String,%20boolean%29
#Autowired private RabbitAdmin admin;
...
admin.purgeQueue("queueName", false);
you can use
AMQP.Queue.PurgeOk queuePurge(java.lang.String queue)
"See queuePurge:
http://www.rabbitmq.com/amqp-0-9-1-quickref.html#queue.purge"
I am new to FIX. I have a FIX message:
8=FIX.4.4|9=122|35=D|34=215|49=CLIENT12|52=20100225-19:41:57.316|56=B|1=Marcel|11=13346|21=1|40=2|44=5|54=1|59=0|60=20100225-19:39:52.020|10=072|
and I am using quickfixJ.
Here is my class code:
public String getYear(Message aMessage, SessionID aSessionID){
try {
crack(aMessage, aSessionID);
} catch (Exception e) {
e.printStackTrace();
}
String year = String.valueOf(mUTCCal.get(Calendar.YEAR));
String begin = String.valueOf(BeginString);
return year + " " + begin;
}
and when I call this method I 2012 null
I tried all sorts of methods for different fields and I get null. I am confused about why I do not get null for the date and how do I make it interpret correctly the other fields?
quickfix.fix44.NewOrderSingle message;
message = new quickfix.fix44.NewOrderSingle();
SessionID session = new SessionID("beginString", "senderCompID", "targetCompID");
MyApp app = new MyApp("", "", "");
String result = app.myMessage(message, session);
System.out.println(result);
I do not understand where to input the string I have (up top) into message
public void onMessage(Message message, SessionID sessionID) throws FieldNotFound {
Header header = message.getHeader();
String FIX = header.getString(8);
System.out.println(FIX);
}
public void onMessage(quickfix.fix44.NewOrderSingle message, SessionID sessionID) throws FieldNotFound, UnsupportedMessageType, IncorrectTagValue {
Header header = message.getHeader();
String FIX = header.getString(8);
String a = message.getString(1);
System.out.println(a);
System.out.println(FIX);}
In order to correctly get and parse FIX messages via QuickFIX, you must:
Create your Application: http://www.quickfixengine.org/quickfix/doc/html/application.html
Implement FromApp(Message message, SessionID sessionID) method
Implement the cracked method for ALL your message types you will receive from your counterparty
The FromApp method can be very simple:
public void fromApp(Message message, SessionID sessionID)
{
crack(message, sessionID);
}
Now, in your example you have a message FIX 4.4 of type 35=D [NewOrderSingle]
Therefore, you MUST implement a method as follows:
public override void onMessage(QuickFix44.NewOrderSingle message, SessionID session)
{
base.onMessage(message, session);
}
Now into your method you can easily work with all the fields you need:
public override void onMessage(QuickFix44.NewOrderSingle message, SessionID session)
{
base.onMessage(message, session);
ClOrdID ordid = new ClOrdID();
message.get(ordid);
}
Please also take a look here: http://www.quickfixengine.org/quickfix/doc/html/receiving_messages.html