I am new to Microservices. I am currently developing an application using Microservices and using both synchronous and asynchronous communication.
Recently I have read many articles saying that you shouldn't use synchronous(HTTP) communication and should only use asynchronous(message broker). A few have mentioned - If the Microservices are communicating via REST, then you still have, in effect, a monolithic application.
Consider a scenario where we have 2 Microservices (MS) :
CurrencyConversion MS - We will pass input to this MS as we want to convert $100 to INR. CurrencyConversion MS will execute a GET call to CurrencyExchange MS to get exchange rate for $ to INR.
CurrencyExchange MS - We will pass input to this MS as $ to INR and CurrencyExchange MS will return the exchange rate as 75 i.e. $1 = 75 INR.
In such cases, CurrencyConversion can't work independently and if CurrencyExchange is failing, CurrencyConversion is also going to fail.
So my first question is - Is synchronous communication between services an anti-pattern in Microservices?
The second question is - If synchronous communication is not a preferred way then what is the best way to design communication between different internal services where one service is going to execute a GET call to get some dependent data for example the scenario I have mentioned above.
How do we overcome this without using synchronous communication?
When you are on a microservices project, it is very frequent that microservices need other microservices. As you said, there are several ways to communicate between them: synchronously or asynchronously.
For my part, I think that there is no good or bad choice between synchronous and asynchronous, what you need to do is to choose what best meets your needs.
In the case you mention, I would personally choose a synchronous HTTP call simply because if you made an asynchronous call, it would be more difficult to know if your MS has received the request and especially when it will answer it. This could force you to block the call from your client for a while because he is calling you synchronously in HTTP on a REST resource.
However, if your client does not expect an immediate response to his call, you can very well start with an asynchronous call and provide a notification system to inform your client that the response to his request is ready.
In any case, synchronous calls between microservices should not be considered as anti-patterns. Synchronous and asynchronous calls each meet different needs, so you have to choose which one is more appropriate in your case.
Finally, whether you do synchronous or asynchronous, there are still several ways to do it. Here is a link that explains, I think, quite well the different possibilities for these two solutions : https://dzone.com/articles/patterns-for-microservices-sync-vs-async
Synchronous communication between services is not an anti-pattern in Microservices. But it's important to choose a appropriate communication style depending on the specified quality requirements. Microservices.io describes some communication patterns with pros & cons, tradeoffs and examples.
In such cases, CurrencyConversion can't work independently and if CurrencyExchange is failing, CurrencyConversion is also going to fail.
In your example the two MS are highly coupled cause they need to work together in a synchronous transaction to answer the user request. Assuming that the user wants a response within a specific time interval (lets say 50ms), synchronous communication seems appropriate. Cascading errors can be counteracted with resilience patterns (circuit breaker, bulk head, etc.). In my opinion the example functionality should get deployed in just one MS (Currency-Service). The two described operations and the underlaying domain model seem highly cohesive. That's a strong signal you should not split the functionality into multiple MS. Communication problems solved :)
Related
I have to coordinate 5 separate microservices e.g. A,B,C,D,E
I need to create a coordinator which might monitor a queue for new jobs for A. If A completes ok then a rest request should be sent to B then if everything is ok (happy path) then C is called all the way down to E.
However B,C etc might fail for one reason or another e.g. end point is down or credentials are insufficient causing the flow to fail at a particular stage. I'd like to be able to create something that could check the status of failed job and rerun again e.g. lets try B again, ok now it works the flow would then continue.
Any tips or advice for patterns / frameworks to do this. I'd like something fairly simple and not over complex.
I've already looked briefly at Netflix Conductor / Camunda but ideally I'd like something a bit less complex.
Thanks
W
Any tips or advice for patterns / frameworks to do this. I'd like something fairly simple and not over complex.
What you describe is the good ol' domain of A,B,C,D and E. Because the dependencies and engagement rules between the letters are complex enough, it's good to create a dedicated service for this domain. It could be as simple as this overarching service just being triggered by queue events.
The only other alternative is to do more on the client side and organize the service calls from there. But that isn't feasible in every domain for security reasons or other issues.
And since it sounds like you already got an event queue going, I'll not recommend one (Kafka).
One way apart from Camunda, Conductor is to send a event from Service A on some Messaging Queue (eg. lets say kafka ) which provides at least once delivery semantics.
Then write a consumer which receive the event and do the orchestration part (talking to service B,C,D,E).
As these all operations needs to be idempotent.First before starting orchestration create a RequestAgg. for the event from A and keep updating its state to represent where you reach in your orchestration journey.
Now even if the other services are down or your node goes down. This should either reach the end or you should write functions to rollback as well.
And to check the states and debug , you could see the read model of RequestAgg.
Angular 4 application sends a list of records to a Java spring MVC application that has been deployed in Websphere 8 Servlet container. The list is then inserted into to a temp table. After the batch insert, a procedure call is made in order to do some calculations and return results. Depending on the size of the list that was inserted into temp table it may take anywhere between: 3000ms( N ~ 500 ), 6000ms( N ~ 1000 ), 50,000+ms ( N > 2000 ).
My asendach would be to create chunks of data and simultaneously send them to database for processing. After threads (Futures) return results I would aggregate them and return back to the client. To sum up, I would split a synchronous call into multiple asynchronous processes(simultaneously executed) and return back to the client over the same thread that initiated HTTP call - landed into my controller.
Everything would be fine and I would not be asking this questions if a more experienced colleague of mine was not strongly disagreeing with this approach. His reasoning is that using this approach is prone to exceptions due to thread interrupts / timeouts / semaphores and so on. Hi is going as far as saying that multithreading should be avoided within a web container because it can crash the Servlet container in case it runs out of threads.
He proposes that we should have the browser send multiple AJAX requests and aggregates/present data in chunks.
Can you please help me understand which approach is better and why?
I would say that your approach is much better.
Threads created by application logic aren't application container threads and limited only by operating system. While each AJAX request uses a thread from application container. So the second approach reduces throughput and increases the possibility of reaching application container limit while and the first one not. Performance also should be considered because it's much cheaper to create a thread than to send a request over network. Plus each network requests uses additional resources for authentication/authorization/encryption etc.
It's definetely harder to write correct multithread code and it can easily prone to errors. However it shouldn't stop you from doing it because concurrency can significantly increase your performance. It's pretty straightforward to handle interrupts and timeouts using Future and you for sure don't need semaphores here.
Exposing this logic to client looks like breaking of encapsulation. Imagine that you use rest api which forces you to send multiple request by splitting you data in chunks. What chunk size should i use? How to deal with timeouts/interrupts? How many requests should i sent? etc. You will have almost the same challenges in both approaches, but it's much easier to deal with them using specially designed for this libraries like ExecutorService and Future.
What I am doing:
I am using play 2.5.7 (java) and trying to build a REST application.
When I get a call on my controller I ask the first actor, this actor can only solve part of the problem (getting additional data), which needs to be forwarded to another actor which uses the request data and additional data to update some more data, send an async void call (tell) to another actor and respond to the controller. All these (4) actors are #Injected in other actors or controller with Guice.
Flow of calls:
controller --(Patterns.ask)--> actor1 --(actor.forward)--> actor2 --(actor.forward)--> actor3 (-tell-> actor4) and --(sender().tell)--> controller.
Issue:
This works for first 4 calls. Then on actor1.forward keeps failing on every consecutive request; Patterns.ask times out. System.out on the line before actor1.forward works but not the actual forward. No matter the timeout value (tried even 20s). No change done in the request; I just hit the send button in postman every time.
I have two questions:
Why 4? Why does it fail after 4th request? Is it some config? What should I look for in config?
Is what I am doing with actors correct way to build a REST web service?
Update: I found the issue; it was caused due to consumption of Redis connections through the pool and never freeing them. But the second question I had still remains, is what I am doing here advisable?
Sure, this could be a reasonable design. But I would consider though whether it would be more maintainable to work with Future returning methods, unless your workflow requires some complex protocol between multiple moving pieces or internal state. It may also be worth considering Akka Streams, if your processing doesn't map well to async method calls.
Basically, actors are a pretty low-level tool. To the extent that you need them, I would try to minimize the surface area of your application where they are being directly used. Higher-level abstractions are better, where possible.
Let's imagine a problem:
I have a REST service, which is implemented using Java/MySQL/Spring and HTTP/JSON technologies.
The clients of the REST service are mobile applications.
So it's possible that someone will decompile the code and will get the API of the REST service.
( Yes, the code is obfuscated etc, but anyway ).
Problem: there is a POST method to send money to other user of the application.
I am worried, that someone can get the API, write a bot and make this POST request 500 or 5,000 or even 50,000 times per second.
As a result, he might send more money than he actually has, because if 1000 requests are processed simultaneously then the balance checking might be
successful for all 1000 request, however the real amount of money on an account might be enough only for, lets say, 50 requests.
So, basically, it's more like the standard "race" condition with multiple threads.
The problem is, that I have multiple servers and they are not related with each other anyhow.
So, 300 request can come to server A, 300 requests can come to server B and rest requests can come to server C.
The best idea what I have is to use something like "SELECT ... FOR UPDATE" and synchronize on database level.
However, I would like to consider another solutions.
Any ideas or suggestions?
You have a few options:
Rely on ACID implementation of the database (MySQL in your case). Assuming you are using InnoDB engine, you need to choose the right transaction isolation level (SET TRANSACTION syntax) in combination with the right locking reads mechanism (SELECT ... FOR UPDATE and SELECT ... LOCK IN SHARE MODE Locking Reads). You need to understand these concepts well in order to do the right choice. It might be possible that simply using the right isolation level will already prevent the race condition even without the locking reads. The cons are you are trading off consistency for scalability and tying your application to RDBMS database thus it will be more difficult for you to move to NoSQL for example.
Decompose your back end into web tier and service tier (option suggested by atk in the comments). This will allow you to scale the web tier instances independently while keeping a single service tier instance. Having a single service tier instance makes it possible to use Java synchronization mechanisms such as synchronised blocks or ReadWriteLock. Although this solution will work I wouldn't recommend it since it reduces the scalability of your service tier.
This is an enhancement of the previous option. You can use a Distributed lock manager instead of built-in java synchronization mechanisms. It will allow you to scale your web tier and service tier independently.
For mission critical applications, it's best to have multiple levels of locking mechanisms.
"SELECT ... FOR UPDATE" is a good way to do so, but they're pretty expensive, and when you try to bombard this with Charles, you'll see that your upper API stack will suffer, and that simple mechanism will cripple your infrastructure pretty easily, much like a DDoS event.
Implement it at the Load Balancer/Proxy layer first, to throttle N-number of requests per specified intervals from a single IP address.
Then apply a shared cache layer lock, where all your boxes go to synchronize on certain keys depending on which critical transaction you want to lockdown. For example, you can use Redis GETSET or INCR functionality to atomically set a flag, before entering the critical code path. Reject anything else quickly to avoid those bad actors from holding on to CPU/memory.
You can also implement things like an APC cache (before hitting your Redis / Memcache cluster) to do similar locking on a per box basis. This is quicker as there's no network latency involved.
The above are necessary on top of using "SELECT ... FOR UPDATE"
I have a requirement in my java web application where I need to send email alerts for certain conditions. For this I have used javax mail api and sending email works just fine. But the problem is the programs executions waits until the methods for sending the email are executed. As there are hundreds of email to be sent at various points ... this reduces the performance significantly.
I am using spring and have also used spring aop. Can anyone suggest me how can I separate my business logic and sending email functionality. It should be like -
Sending emails is my advice which gets executed when xyz method is called - So main execution should not wait for advice to finish its execution rather it should return back and execute further business logic thus email sending executed separately.
Here creating new threads seems obvious choice. But I think there could be some better way, is there? Thanks.
You can make the mail sending method #Async. This way Spring will execute this in a seperate thread. Read this blog post about it: Creating Asynchronous Methods
What you describe is asynchronous execution and natural way to do async execution is Java is to use threads.
You can introduce some Executor, e.g., Executors.newFixedThreadPool(), and use it to offload mailing task into separate threads.
Aspect itself is a unsuitable place for this, since this would introduce state into aspect, for example, you may want to check if mail task was successful by using returned Future:
class Mailer {
private final ExecutorService executor = Executors.newFixedThreadPool(maxMailingThreads);
//...
public void doMail(MailTask anEmail) {
Future<MailTaskResult> future = executor.submit(new MailTask(anEmail));
future.get().isSuccessful(); // handle success or failure somehow
}
Better move this logic into separate class and call it from aspect somehow.
Treat the email sending functionality like an IO device. Make it a plugin to your business logic. Do not allow any knowledge of the fact that you're even talking to the email code into your business logic. Make the email logic depend on the business logic. Never the other way around.
Here's a very good talk about this kind of architecture:
https://vimeo.com/97530863
Here's a series debating it:
https://www.youtube.com/watch?v=z9quxZsLcfo
Here's a ruby master demonstrating it with real code. We miss him.
https://www.youtube.com/watch?v=tg5RFeSfBM4
If your business rules are interesting enough to be worth respecting than this is the way to make them the masters of your application. Express them only using java. Don't accept any help. No spring, no weird annotations, just business rules. Push all that "help" out to the mail code.
Do this and your app will scale well. I think this is the best way to put it:
That's from a hexagonal architecture post. But the idea of giving your business rules a safe place to live removed from implementation detail shows up in many architectures. This answer rounds them up nicely.
Use a localhost MTA (like OpenSMTPD) and then relay to your real SMTP server, like Amazon SES ("Satellite" mode). It won't block.
I did a test, and sent 1000 emails in 2.8 seconds this way
It's simpler than doing async in java, and is useful across multiple applications.
As for separating logic, raise a Spring Application Event when needed, and make another class to listen to it, and send your email from there. Or consider something like Guava's EventBus
Consider creating a separate thread to send emails within your application. This will allow parallel execution(application+email sending).
If you would want another approach you can create a separate back end application that only sends emails. Although you will need to submit the email messages to the application. An asynchronous way to do this is to send a JMS message to the email application.