We Keep Coding

how we should design communication between different internal Microservices

I am new to Microservices. I am currently developing an application using Microservices and using both synchronous and asynchronous communication.
Recently I have read many articles saying that you shouldn't use synchronous(HTTP) communication and should only use asynchronous(message broker). A few have mentioned - If the Microservices are communicating via REST, then you still have, in effect, a monolithic application.
Consider a scenario where we have 2 Microservices (MS) :
CurrencyConversion MS - We will pass input to this MS as we want to convert $100 to INR. CurrencyConversion MS will execute a GET call to CurrencyExchange MS to get exchange rate for $ to INR.
CurrencyExchange MS - We will pass input to this MS as $ to INR and CurrencyExchange MS will return the exchange rate as 75 i.e. $1 = 75 INR.
In such cases, CurrencyConversion can't work independently and if CurrencyExchange is failing, CurrencyConversion is also going to fail.
So my first question is - Is synchronous communication between services an anti-pattern in Microservices?
The second question is - If synchronous communication is not a preferred way then what is the best way to design communication between different internal services where one service is going to execute a GET call to get some dependent data for example the scenario I have mentioned above.
How do we overcome this without using synchronous communication?

When you are on a microservices project, it is very frequent that microservices need other microservices. As you said, there are several ways to communicate between them: synchronously or asynchronously.
For my part, I think that there is no good or bad choice between synchronous and asynchronous, what you need to do is to choose what best meets your needs.
In the case you mention, I would personally choose a synchronous HTTP call simply because if you made an asynchronous call, it would be more difficult to know if your MS has received the request and especially when it will answer it. This could force you to block the call from your client for a while because he is calling you synchronously in HTTP on a REST resource.
However, if your client does not expect an immediate response to his call, you can very well start with an asynchronous call and provide a notification system to inform your client that the response to his request is ready.
In any case, synchronous calls between microservices should not be considered as anti-patterns. Synchronous and asynchronous calls each meet different needs, so you have to choose which one is more appropriate in your case.
Finally, whether you do synchronous or asynchronous, there are still several ways to do it. Here is a link that explains, I think, quite well the different possibilities for these two solutions : https://dzone.com/articles/patterns-for-microservices-sync-vs-async

Synchronous communication between services is not an anti-pattern in Microservices. But it's important to choose a appropriate communication style depending on the specified quality requirements. Microservices.io describes some communication patterns with pros & cons, tradeoffs and examples.
In such cases, CurrencyConversion can't work independently and if CurrencyExchange is failing, CurrencyConversion is also going to fail.
In your example the two MS are highly coupled cause they need to work together in a synchronous transaction to answer the user request. Assuming that the user wants a response within a specific time interval (lets say 50ms), synchronous communication seems appropriate. Cascading errors can be counteracted with resilience patterns (circuit breaker, bulk head, etc.). In my opinion the example functionality should get deployed in just one MS (Currency-Service). The two described operations and the underlaying domain model seem highly cohesive. That's a strong signal you should not split the functionality into multiple MS. Communication problems solved :)

Lagom PubSubRef subscriber drops messages

[Attention] The question is Lagom framework specific!
In my current project, the problem with cutting the list of messages from Source to Kafka topic publisher has been observed when upstream is of high speed and looks like downstream can't handle all messages in time. As realized, the cutting is related to the behavior of PubSubRef.subscribe() method https://github.com/lagom/lagom/blob/master/pubsub/javadsl/src/main/scala/com/lightbend/lagom/javadsl/pubsub/PubSubRef.scala#L85
The full method definition:
def subscriber(): Source[T, NotUsed] = {
scaladsl.Source.actorRef[T](bufferSize, OverflowStrategy.dropHead)
.mapMaterializedValue { ref =>
mediator ! Subscribe(topic.name, ref)
NotUsed
}.asJava
}
There's OverflowStrategy.dropHead is used. Can it be changed to use back-pressure strategy?
UPD#1:
The use case is pretty simple, when a query request is published into command topic, get it and query objects from DB table, the resulting list is pushed into result Kafka topic. Code snippet:
objectsResultTopic = pubSub.refFor(TopicId.of(CustomObject.class, OBJECTS_RESULT_TOPIC));
objectQueryTopic().subscribe().atLeastOnce(
Flow.fromSinkAndSource(
Flow.fromFunction(this::deserializeCommandAndQueryObjects)
.mapAsync(concurrency, objects -> objects)
.flatMapMerge(concurrency, objects -> objects)
.alsoTo(Sink.foreach(event -> LOG.trace("Sending object {}", object)))
.to(objectsResultTopic.publisher()),
Source.repeat(Done.getInstance())
)
)
In case of objects stream generated by deserializeCommandAndQueryObjects function is more than default buffer-size = 1000 it starts cutting the elements (our case is ~ 2.5k objects).
UPD#2:
The source of objects data is:
// returns CompletionStage<Source<CustomObject, ?>>
jdbcSession.withConnection(
connection -> Source.from(runQuery(connection, rowConverter))
)
And there's a subscription to Kafka objectsResultTopic:
TopicProducer.singleStreamWithOffset(
offset -> objectsResultTopic.subscriber().map(gm -> {
JsonNode node = mapper.convertValue(gm, JsonNode.class);
return Pair.create(node, offset);
}));

It sounds like Lagom's distributed publish-subscribe feature may not be the best tool for the job you have.
Your question mentions Kafka, but this feature does not make use of Kafka. Instead, it works by directly broadcasting messages to all subscribers in the cluster. This is an "at most once" messaging transport that may indeed lose messages, and is intended for consumers who care more about keeping up with recent messages than processing every single one. The overflow strategy is not customizable, and you would not want to use back-pressure in these use cases, as it would mean that one slow consumer could slow down delivery to all of the other subscribers.
There are a few other options that you have:
If you do want to use Kafka, you should use Lagom's message broker API. This supports "at least once" delivery semantics, and can be used to ensure that each consumer processes every message (at the cost of possibly increasing latency).
In this case, Kafka acts as a giant durable buffer, so it's even better than back-pressure: the producer and consumer can proceed at different paces, and (when used with partitioning) you can add consumers in order to scale out and process messages more quickly when needed.
The message broker API can be used when producers and consumers are all in the same service, but it is particularly suitable for communication between services.
If the messages you are sending are persistent entity events, and the consumers are part of the same service, then a persistent read-side processor might be a good option.
This also provides "at least once" delivery, and if the only effects of processing messages are database updates, then the built-in support for Cassandra read-side databases and relational read-side databases provide "effectively once" semantics, where the database updates are run transactionally to ensure that failures that occur during event processing cannot result in partial updates.
If the messages you are sending are persistent entity events, the consumers are part of the same service, but you want to process the events as a stream, you can access a raw stream of events.
If your use case does not fit into one of the use cases that Lagom supports explicitly, you can use lower-level Akka APIs, including distributed publish-subscribe, to implement something more tailored to your needs.
The best choice will depend on the specifics of your use case: the source of the messages and the types of consumers you want. If you update your question with more details and add a comment to this answer, I can edit the answer with more specific suggestions.

If someone is interested, finally we solved that problem by using Akka Producer API, like:
ProducerSettings<String, CustomObject> producerSettings = ProducerSettings.create(system, new StringSerializer(), new CustomObjectSerializer());
objectQueryTopic().subscribe().atLeastOnce(
Flow.fromSinkAndSource(
Flow.fromFunction(this::deserializeCommandAndQueryObjects)
.mapAsync(concurrency, objects -> objects)
.flatMapMerge(concurrency, objects -> objects)
.alsoTo(Sink.foreach(object -> LOG.trace("Sending event {}", object)))
.map(object -> new ProducerRecord<String, CustomObject>(OBJECTS_RESULT_TOPIC, object))
.to(Producer.plainSink(producerSettings)),
Source.repeat(Done.getInstance())));
It works without buffering, just makes the pushing into Kafka topic.

Spring JMS Template - Concurrent Calls

Can I make concurrent calls using Spring JMSTemplate?
I want to make 4 external service calls in parallel and am exploring using Spring's JMSTemplate to perform these calls in parallel and wait for the execution to complete.
The other option that I am looking at is to use ExecutorService.
Is there any advantage using one over the other?

JMSTemplate is thread-safe, so making parallel calls to it is not a problem.
Messaging services are usually fast enough for most tasks and can receive your messages with minimal latency, so adding an ExecutorService doesn't seem as the first thing you usually need. What you really need is to correctly configure your JMS connections pool and give it enough open connections (four in your case) so it could handle your parallel requests with no blocking.
You only need ExecutorService in case you don't care about guaranteed delivery and your program needs extremely high speed that your messaging service cannot deliver, which is highly unlikely.
As for receiving replies from your external service, you need to use JMS Request/Reply pattern (you can find examples in this article). Happily, as you're using Spring, you could make Spring Integration do lots of work for you. You need to configure outbound-gateway to send messages and inbound-gateway to receive responses. Since version 2.2 you can also use reply-listener to simplify things on your client side. All these components are covered in the official documentation (with examples as well).

So need to talk to more than two JMS queues (send and or receive) parallel using asynchronous methods. Best option is usng #Asynch at method level
This example contains RestTemplate , But in your case create JmsTemplate beans.
Prerequisites:- Please create proper JMS Beans to connect to the queue. Proper use of this will help to invoke two queues paralleley. Its works for sure because already I have implemented. I just giving the skeleton due to Copyright issues.
More details: Spring Boot + Spring Asynch
https://spring.io/guides/gs/async-method/
Step1: Create a Service Class where JMS Queue
#EnableAsynch
public class JMSApplication {
#Autowired
JmsService jmsService;
public void invokeMe(){
// Start the clock
long start = System.currentTimeMillis();
// Kick of multiple, asynchronous lookups
Future<Object> queue1= jmsService.findqueue1();
Future<Object> queue2= jmsService.findqueue2();
// Wait until they are all done
while (!(queue1.isDone() && queue2.isDone())) {
Thread.sleep(10); //10-millisecond pause between each check
}
// Print results, including elapsed time
System.out.println("Elapsed time: " + (System.currentTimeMillis() - start));
System.out.println(queue1.get());
System.out.println(queue2.get());
}
}
Step2: Write the Service Method which will contain the business logic
for Jms
#Service
public Class JmsService{
#Asynch
public Object findqueue1(){
//Logic to invoke the JMS queue
}
#Asynch
public Object findqueue2(){
//Logic to invoke the JMS queue
}
}

Differences Message Driven Application between simple method calls

I've been reading something about, found some libraries which really messed with my thoughts, like Akka, Quasar, Reactor and Disruptor, Akka and Quasar implements the Actor Pattern and the Disruptor is a Inter-Thread Messaging library, and Reactor is based on . So what are the advantages, use cases for using a message driven architecture over simple method calls?
Given a RabbitMQ queue listener, I receive a message from the method, decide which Type the RabbitMQ message is (NewOrder,Payment,...).
With a Message Driven library I could do.
Pseudo code:
actor.tell('decider-mailbox',message)
Which basically says "I'm putting this message here, when you guys can handle it, do it") and so on until it gets saved.
And the actor is ready again to receive another message
But with directly calling the method like messageHandler.handle(message), wouldn't be better and less abstracted ?

The Actors Model looks a lot like people working together; it is based on message-passing but there is much more to it and I'd say that not all message-passing models are the same, for example Quasar actually supports not only Erlang-like actors but also Go-like channels which are simpler but don't provide a fault-tolerance model (and fibers BTW, that are just like threads but much more lightweight, which you can use even without any message-passing at all).
Methods/functions follow a strict, nestable call-return (so request-response) discipline and usually don't involve any concurrency (at least in imperative and non-pure functional languages).
Message passing instead, very broadly speaking, allows looser coupling because doesn't enforce a request-response discipline and allows the communicating parties to execute concurrently, which also helps in isolating failures and in hot-upgrades and generally maintenance (for example, the Actors Model offers these features). Often message passing will also allow looser data contracts by using a more dynamic typing for messages (this is especially true for the Actors Model where each party, or actor, has a single incoming channel, that is his mailbox).
Other than that the details depends a lot on the messaging model/solution you're considering, for example the communication channels can synchronize the interacting parts or have limited/unlimited buffering, allow multiple source and/or multiple producers and consumers etc.
Note that RPC is really message passing but with a strict request-response communication discipline.
This means that, depending on the situation, one or the other may suit you better: methods/functions are better when you're in a call-return discipline and/or you're simply making your sequential code more modular. Message-passing is better when you need a network of potentially concurrent, autonomous "agents" that communicate but not necessarily in a request-response discipline.
As for the Actors Model I think you can build more insight about it for example by reading the first part of this blog post (notice: I'm the main author of the post and I'm part of the Parallel Universe - and Quasar - development team):
The actor model is a design pattern for fault-tolerant and highly scalable systems. Actors are independent worker-modules that communicate with other actors only through message-passing, can fail in isolation from other actors but can monitor other actors for failure and take some recovery measures when that happens. Actors are simple, isolated yet coordinated, concurrent workers.
Actor-based design brings many benefits:
Adaptive behaviour: interacting only through a message-queue makes actors loosely coupled and allows them to:
Isolate faults: mailboxes are decoupling message queues that allow actor restart without service disruption.
Manage evolution: they enable actor replacement without service disruption.
Regulate concurrency: receiving messages very often and discarding overflow or, alternatively, increasing mailbox size can maximize concurrency at the expense of reliability or memory usage respectively.
Regulate load: reducing the frequency of receive calls and using small mailboxes reduces concurrency and increases latencies, applying back-pressure through the boundaries of the actor system.
Maximum concurrency capacity:
Actors are extremely lightweight both in memory consumption and
management overhead, so it’s possible to spawn even millions in a
single box.
Because actors do not share state, they can safely run in parallel.
Low complexity:
Each actor can implements stateful behaviour by mutating its private state without worrying about concurrent modification.
Actors can simplify their state transition logic by selectively receiving messages from the mailbox in logical, rather than arrival order.

The difference is that the processing takes place in a different thread so the current one is ready to receive and forwards the next message. When you call the handler from the current thread it is blocked until processing is finished.

In a way it is just a matter of defining abstractions. Some say that originally object oriented programming was actually supposed to be based on message passing, and calling a method on an object would have the semantics of sending it a message (with similar async non-blocking behavior as in actors).
The way we implemented OO in most popular languages is such that it became what it is today - a "synchronous blocking order" to an object, controlled and ran from the same execution context (thread/process) as the caller. This is nice because it is easy to understand, but it has its limitations when designing concurrent systems.
In theory, you could create a language with similar syntax as Java, but give it different semantics - making object.method(arg) actually internally be something similar to actor.tell(msg). There are a lot of idioms that try to hide asynchronous calling and message passing behind simple method invocations, but as always it depends on the use case.
Akka provides a nice new syntax which makes it clear that what we are doing is something completely different than invoking methods on an object, in part to cause less confusion and make the message passing more explicit. In the end, you are stating the same thing - you are sending a message to an actor in the system, but you are doing it with less constraints than if you were calling one of its methods directly.

Distributed event handling mechanism for Java

I'm looking for a reasonably fast event handling mechanism in Java to generate and handle events across different JVMs running on different hosts.
For event handling across multiple threads in a single JVM, I found some good candidates like Jetlang. But in my search for a distributed equivalent , I couldn't find anything that was lightweight enough to offer good performance.
Does anyone know of any implementations that fit the bill?
Edit:
Putting numbers to indicate performance is a bit difficult. But for example, if you implement a heartbeating mechanism using events and the heartbeat interval is 5 seconds, the heartbeat receiver should receive a sent heartbeat within say a second or two.
Generally, a lightweight implementation gives good performance. A event handling mechanism involving a web server or any kind of centralized hub requiring powerful hardware (definitely not lightweight) to give good performance is not what I'm looking for.

Hazelcast Topic is a distributed pub-sub messaging solution.
public class Sample implements MessageListener {
public static void main(String[] args) {
Sample sample = new Sample();
Topic topic = Hazelcast.getTopic ("default");
topic.addMessageListener(sample);
topic.publish ("my-message-object");
}
public void onMessage(Object msg) {
System.out.println("Message received = " + msg);
}
}
Hazelcast also supports events on distributed queue, map, set, list. All events are ordered too.
Regards,
-talip
http://www.hazelcast.com

Depending on your use case, Terracotta may be an excellent choice.

AMQP(Advanced Message Queuing Protocol ) -- more details :
http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol is probably what you're looking for.
It is used by financial service companies for their high performance requirements -- apache has an implementation going -- http://cwiki.apache.org/qpid/
OpenAMQ - http://www.openamq.org/ is an older REFERENCE IMPLEMENTATION .

For distributed Event processing you could use Esper.It could process up to 500 000 event/s on a dual CPU 2GHz Intel based hardware.It's very stable because many banks use this solution. It supports JMS input and output adapter based on Spring JMS templates. So you could use any JMS implementation for event processing, i.e. ActiveMQ.

ZeroMQ - http://www.zeromq.org/
Although this is a transport layer, it can be tailored for event handling.

Whichever tool you use I'd recommend hiding the middleware APIs from your application logic. For example if you used the Apache Camel approach to hiding middleware you could then easily switch from AMQP to SEDA to JMS to ActiveMQ to JavaSpaces to your own custom MINA transport based on your exact requirements.
If you want to use a message broker I'd recommend using Apache ActiveMQ which is the most popular and powerful open source message broker with the largest most active community behind it both inside Apache and outside it.

Take a look at akka (http://akka.io/). It offers a distributed actor model in the same vein as erlang for the JVM with both java and scala APIs.

You need to implement Observer Design pattern for distributed event handling in java. I am using event Streaming using MongoDB capped collection and Observers to achieve this.
You can make an architecture in which your triggers a publish a document in capped collection and your observer thread waits for it using a tailable cursor.
If you did not understand what I have said above you need to brush up your MongoDB and java skills

If a JMS implementation isn't for you, then you may be interested in an XMPP approach. There are multiple implementations, and also have a Publish-Subscribe extension.

The Avis event router might be suitable for your needs. It's fast enough for near-real-time event delivery, such as sending mouse events for remote mouse control (an application we use it for daily).
Avis is also being used for chat, virtual presence, and smart room automation where typically 10-20 computers are communicating over an Avis-based messaging bus. Its commercial cousin (Mantara Elvin) is used for high-volume commercial trade event processing.