We Keep Coding

How do the Sender know that the Receiver isn't available in JMS using Spring Boot?

I have a Spring Boot app, there are 2 microservices and these microservices communicate asynchronous using JMS and ActiveMq. So the Sender (ms1) sends a message to the Receiver (ms2), the Sender will put the message on the queue and if the Receiver isn't available, the message will stay on the queue until the Receiver is available.
I want to ask you how could the Sender knows if the Receiver is available or not? I want to know that because I want to use Hystrix and if the Receiver is available the Sender will show a message like this: "Transaction successfully completed!", but if the Receiver isn't available the Sender will show other message, something like this: "The Receiver service isn't currently availble, the message is added to the queue and will be sended to the Receiver when it'll be available".
This code is from the Sender service:
#HystrixCommand(fallbackMethod="sendMessageToProducerFail")
private ResponseEntity sendMessageToProducer(String jsonStr) {
jmsTemplate.convertAndSend(queue, jsonStr);
return new ResponseEntity("Transaction successfully completed!", HttpStatus.CREATED);
}
private ResponseEntity sendMessageToProducerFail(String jsonStr) {
// "The Receiver service isn't currently availble, the message is added to the queue..."
}

There's no simple way to achieve this. Identifying whether the consumer is up or not bring some of the design choices. Also, there's no out of box solution for your problem. You can use the concept of a heartbeat to notify producers about consumer state.
+----------+ +-------------+ +--------+
| |<--Heartbeat---| |---Message--->| |
| Producer | | AMQP Server | |Consumer|
| |----Message--->| |<--Heartbeat--| |
+----------+ +-------------+ +--------+
Your setup would look somewhat like this, In this producer(ms1) will send messages to AMQP server and will consume Heartbeat events from the consumer(s), to identify whether the consumer(s) is/are alive. This will become more tricky when you have more than one producer/consumer for the same topic.
Once you have more than once consumers, multiple heartbeats would be sent on the same topic/queue. You need to identify which one of them is alive, you need to also consider consumers going down/up.
Each heartbeat can have timestamp when it was generated so that you can make a consumer alive decision at a more granular level. The next question would be when to send a heartbeat? As you're using Spring boot you will not have the flexibility of sending heartbeat outside of the message listener. If you're using the MessageListener interface then do something like this.
Create a heartbeat publisher class
#Component
class HeartBeatPublisher {
public void registerMessageListener( String listenerName,
String topicOrQueueName,
Long intervals) {
// store this detail in a map
// do lazy init of threads to send heartbeat
}
}
Your message listener class would like
#Component
class MyMessageListener implements MessageListener {
#Autowired HeartBitPublisher heartBeatPublisher;
#PostConstruct
public void init(){
// 30 seconds interval
heartBeatPublisher.registerMessageListener("MyMessageListener",
"MyMessageListener",
30*1000 );
}
void onMessage(Message message){
// consume message here
}
}
In case you're not using MessageListener still you can send heartbeats, in that case, I would recommend adding one listener per component.
#Component
class MyMessageListener{
#Autowired HeartBeatPublisher heartBeatPublisher;
#PostConstruct
public void init(){
// 30 seconds interval
heartBeatPublisher.registerMessageListener("MyMessageListener",
"MyMessageListener",
30*1000 );
}
#RabbitListener(queues="myQueue")
public void onMessage(Object message) {
// Consume message
}
}
On the producer side, you need to add a listener(s) to check which consumer(s) is/are active using a heartbeat. Once you know which consumer(s) is/are active you can use that to restrict message publish.
#Component
class HeartBeatListener {
private List<String> queues;
#PostConstruct
public void init(){
// initialize queue and consumer status to inactive
// at certain intervals checks for missing heartbeat
// if you find heartbeats are not being sent that means either
// that consumer has died or there's a delay in heart bit
// publish in that case mark that consumer/topic/queue inactive
}
#RabbitListener(queues="myQueue-HeartBeat")
public void onMessage(Object message) {
// update consumer status
// Record heart beat for a given consumer/topic
}
}
Instead of doing all this, you can use Consul, Zookeeper, or Etcd to move some of the works to these systems.

Amount parallel processing Simple Queue Service (SQS)

I am using Spring Cloud to consume Simple Queue Service (SQS). I have the following configurations for parallel processing:
#Bean
public SimpleAsyncTaskExecutor simpleAsyncTaskExecutor() {
SimpleAsyncTaskExecutor simpleAsyncTaskExecutor = new SimpleAsyncTaskExecutor();
simpleAsyncTaskExecutor.setConcurrencyLimit(50);
return simpleAsyncTaskExecutor;
}
#Bean
public SimpleMessageListenerContainerFactory simpleMessageListenerContainerFactory(
SimpleAsyncTaskExecutor simpleAsyncTaskExecutor) {
SimpleMessageListenerContainerFactory factory = new SimpleMessageListenerContainerFactory();
factory.setAutoStartup(true);
factory.setTaskExecutor(simpleAsyncTaskExecutor);
factory.setWaitTimeOut(20);
factory.setMaxNumberOfMessages(10);
return factory;
}
I need to process 50 messages in 50 threads (configuration in the bean SimpleAsyncTaskExecutor), but is processing only 10 messages in parallel (maxNumberOfMessages returned from SQS)
How can I process 50 messages instead 10?

I found the solution.
It's necessary to annotate the method with #Async, change deletionPolicy to NEVER, and delete the message when finalizing execution.
In this way, the queue consume will respect the configured number of threads. For example, if you have 50 threads, will make 5 requests in the SQS queue (10 messages per request), thus processing a total of 50 messages in parallel.
The code looks like this:
#Async
#SqsListener(value = "sqsName", deletionPolicy = SqsMessageDeletionPolicy.NEVER)
public void consume(String message, Acknowledgment acknowledgment) throws InterruptedException, ExecutionException {
//your code
acknowledgment.acknowledge().get(); //To delete message from queue
}

I wouldn't be into specific numbers (like 50 messages for 50 threads) too much. Try performance testing it instead (build something to push the expected number of messages in peak-hours to the queue, and let your service handle them, to see if it bottlenecks).
As per your actual question, you can't. AWS SQS simply doesnt support fetching more than 10 messages pr. request. see http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html for reference. (it's in the 1st paragraph).

Asynchronous event in Spring taking lot of time to execute(get its turn)

I have an application where I need to trigger email whenever a REST call is made for an endpoint. The design is that whenever a REST call is invoked, I save the data in the db, emit an asynchrnous event and return.
My problem is that due to huge no of requests that keep coming, the async events which are emitted do not get a chance for quite lot of time. Sometimes as the server is up for some weeks, the delay keeps increasing.
The scenario
Server endpoint is invoked
Server saves data to db, emits a Spring Asynchronous event
Returns from the endpoint
The call 2 is getting delayed as the listener is invoked quite late sometimes.
public class DataController {
#Inject
ApplicationEventPublisher eventPublisher;
#RequestMapping(value = "data", method = RequestMethod.POST)
#ResponseStatus(HttpStatus.NO_CONTENT)
public void addData(#RequestBody DataDTO data) {
dataService.addData(data);
eventPublisher.publishEvent(new DataRequest(new DataDTO());
}
}
public class DataRequest extends ApplicationEvent {
private DataDTO dataDTO;
public DataRequest(DataDTO dataDTO) {
super(dataDTO);
this.dataDTO = dataDTO;
}
}
#Component
public class DataListener {
#EventListener
#Async
private void dataListener(DataDTO dataDTO) {
// Send email
}
}
Since it is an Async event , the JVM gives the dataListener chance to execute very late. Sometimes
the events triggered earlier gets chance late than the ones that were triggered after that.
So 2 fundamental problems
Emails delayed. Delay can range from 1 min, to 4 hours to 8 days etc
If an event is triggered at 12 PM to send email to xyz#gmail.com, and another at 12:15 PM which sends email to abc#gmail.com, then there are chances of abc#gmail.com receiving email before xyz#gmail.com.
Appreciate your help

Spring Asynchronous event is limited to the size of the Thread pool and as soon as the incoming requests are higher than the size of active threads there will be delays.
You need to use a Message Queue like RabbitMQ, Kafka, etc. Your architecture should be changed to do the following;
Serialize a JSON message in the REST Endpoint with all information like to email address, database entry data, etc and just store that JSON message in the message queue and return a status code
There must be consumers for message queue (separate Java applications) which poll or get notified when there is data in the message queue.
These consumers should de-serialize the JSON message, save an entry in the database and send an email.
With this architecture you can increase consumers at times of high load and thus scale as required.

Listen to RabbitMQ queue and get notifications on events

I am currently developing an Java application using Spring AMQP with RabbitMQ and would like to monitor my queues and being informed if some events happen, like
element was added to queue,
element was removed, or
element was but in the queue again (rollback).
How can I listen to such events, or what is the RabbitMQ-way of doing such things?

To receive a message asynchronously from a queue is to use the annotated listener endpoint infrastructure. In a nutshell, it allows you to expose a method of a managed bean as a Rabbit listener endpoint.
#Component
public class MyService {
#RabbitListener(queues = "myQueue")
public void processOrder(String data) {
...
}
}
Check this
Whenever there a message is pushed to the queue myQueue processOrder method is triggered.
For your other requirements you can also use spring events to monitor any operation performed on the queue. Just right before performing any operation on the queue a respective event will be triggered.

JMS and MDB with setRollbackOnly

I have a java class which consumes messages from a queue, sending HTTP calls to some urls. I have made some search on google and also on stackoverflow (and really sorry if i have missed any sources mentioning about the problem) but couldnt find anything in details about setRollbackOnly call.
My question is... in case I rollback, the message which is consumed from the queue will be blocking the rest of the queue and will be looping until it is processed successfully or it will be requeued at the end of the current queue?
My code which I use for consuming from the queue and sending HTTP calls is below and the whole application is running on Glassfish server:
public class RequestSenderBean implements MessageListener
{
#Resource
private MessageDrivenContext mdbContext;
public RequestSenderBean(){}
public void onMessage(final Message message)
{
try
{
if(message instanceof ObjectMessage)
{
String responseOfCall=sendHttpPost(URL, PARAMS_FROM_MESSAGE);
if(responseOfCall.startsWith("Success"))
{
//Everything is OK, do some stuff
}
else if(responseOfCall.startsWith("Failure"))
{
//Failure, do some other stuff
}
}
catch(final Exception e)
{
e.printStackTrace();
mdbContext.setRollbackOnly();
}
}
}

This is fundamental JMS/messaging knowledge.
Queues implement "load balancing" scenarios, whereby a message hits a queue and is dequed to be processed by one consumer. Increasing the number of consumers increases potential throughput of that queue's processing. Each message on a queue will be processed by one and only one consumer.
Topics provide publish-subscribe semantics: all consumers of a topic will receive the message that is pushed to the topic.
With that in mind, once a message is dequed and handed (transactionally) to a consumer, it is by no means blocking the rest of the queue if it is asynchronous (as is the case with MDBs).
As the Java EE Tutorial states:
Message Consumption
Messaging products are inherently asynchronous: There is no fundamental timing dependency between the production and the consumption of a message. However, the JMS specification uses this term in a more precise sense. Messages can be consumed in either of two ways:
Synchronously: A subscriber or a receiver explicitly fetches the message from the destination by calling the receive method. The receive method can block until a message arrives or can time out if a message does not arrive within a specified time limit.
Asynchronously: A client can register a message listener with a consumer. A message listener is similar to an event listener. Whenever a message arrives at the destination, the JMS provider delivers the message by calling the listener’s onMessage method, which acts on the contents of the message.
Because you use a MessageListener which is by definition asynchronous, you are not blocking the queue or its subsequent processing.
Also from the tutorial is the following:
Using Session Beans to Produce and to Synchronously Receive Messages
An application that produces messages or synchronously receives them can use a session bean to perform these operations. The example in An Application That Uses the JMS API with a Session Bean uses a stateless session bean to publish messages to a topic.
Because a blocking synchronous receive ties up server resources, it is not a good programming practice to use such a receive call in an enterprise bean. Instead, use a timed synchronous receive, or use a message-driven bean to receive messages asynchronously. For details about blocking and timed synchronous receives, see Writing the Clients for the Synchronous Receive Example.
As for message failure, it depends on how your queue is configured. You can set error-queues (in the case of containers like Glassfish or Weblogic) that failed messages are pushed to for later inspection. In your case, you're using setRollbackOnly which is handled thus:
7.1.2 Coding the Message-Driven Bean: MessageBean.java
The message-driven bean class, MessageBean.java, implements the
methods setMessageDrivenContext, ejbCreate, onMessage, and ejbRemove.
The onMessage method, almost identical to that of TextListener.java,
casts the incoming message to a TextMessage and displays the text. The
only significant difference is that it calls the
MessageDrivenContext.setRollbackOnly method in case of an exception.
This method rolls back the transaction so that the message will be
redelivered.
I recommend you read the Java EE Tutorial as well as the Enterprise Integration Patterns book which covers messaging concepts in good detail that's also product/technology-agnostic.