I have an application which reads from a kafka queue and and goes on like this.
validate->convert->enrich->persist->notify
In the flow, I'm gathering some performance and other data points into a ThreadLocal container.
In the happy scenario I'm sending these information to a service to be later used in reporting. But the pipeline can stop in any step if one of the step fails due to a known error (eg, convert failed so flow should stop there). I do not like each of these processors to have a code that sends the information in the ThreadLocal to reporting service if the execution resulted in error, as that would couple those services with information not related to its task.
It would be nice to have a way to execute a service at the end of the flow to send this information out, no matter which step the pipeline stops moving forward. Also there could be scenarios some code did throw an exception that was not known or other issue that break the flow.
Is there a way that a final operation to be executed no matter the result of the pipeline so that it can be used to send this information similar to a finally block in java?
The integration flow is like a simple Java try...catch...finally. It is really more about a distributed computation and loosely-coupling principle between components. So, even if you tie endpoints with channels in between, there really have nothing to know about the previous and next step: everything is done in the current endpoint with its input and output channels. Therefore your request about something like finally in the flow does not fit to the EIP concepts and cannot be implement as some primitive in the framework.
You are just lucky in your use-case that you can rely on the ThreadLocal for your flow logic, but you should keep in mind that it is not a way to deal with messaging. It really has to be stateless and have scope only of the message traveling from one endpoint to another. Therefore it might be better to revise your logic in favor of storing such a tracing information into headers of that message on each step. This way in the future you can make the flow fully async or even distributed in the network.
This is just my concern for a design you have so far.
For the current error handling problem consider to have that "final" step behind some well-know channel, so you will be free to send a message to that endpoint from whatever place you need. For example you can wrap problematic endpoints into an ExpressionEvaluatingRequestHandlerAdvice. Handle an error over there and send it to the mentioned channel. This way your business method will be free from error handling and so. See more in docs: https://docs.spring.io/spring-integration/docs/current/reference/html/messaging-endpoints.html#expression-advice
If your flow starts from some gateway or inbound channel adapter, you can have an errorChannel configured there to be able to catch all the downstream errors in one central place. And again: send the handling result to the mentioned channel.
But no. No finally in the framework at the moment and I doubt it would even be suitable in the future. For the messaging and async reason I explained before.
Related
I have to coordinate 5 separate microservices e.g. A,B,C,D,E
I need to create a coordinator which might monitor a queue for new jobs for A. If A completes ok then a rest request should be sent to B then if everything is ok (happy path) then C is called all the way down to E.
However B,C etc might fail for one reason or another e.g. end point is down or credentials are insufficient causing the flow to fail at a particular stage. I'd like to be able to create something that could check the status of failed job and rerun again e.g. lets try B again, ok now it works the flow would then continue.
Any tips or advice for patterns / frameworks to do this. I'd like something fairly simple and not over complex.
I've already looked briefly at Netflix Conductor / Camunda but ideally I'd like something a bit less complex.
Thanks
W
Any tips or advice for patterns / frameworks to do this. I'd like something fairly simple and not over complex.
What you describe is the good ol' domain of A,B,C,D and E. Because the dependencies and engagement rules between the letters are complex enough, it's good to create a dedicated service for this domain. It could be as simple as this overarching service just being triggered by queue events.
The only other alternative is to do more on the client side and organize the service calls from there. But that isn't feasible in every domain for security reasons or other issues.
And since it sounds like you already got an event queue going, I'll not recommend one (Kafka).
One way apart from Camunda, Conductor is to send a event from Service A on some Messaging Queue (eg. lets say kafka ) which provides at least once delivery semantics.
Then write a consumer which receive the event and do the orchestration part (talking to service B,C,D,E).
As these all operations needs to be idempotent.First before starting orchestration create a RequestAgg. for the event from A and keep updating its state to represent where you reach in your orchestration journey.
Now even if the other services are down or your node goes down. This should either reach the end or you should write functions to rollback as well.
And to check the states and debug , you could see the read model of RequestAgg.
I have written a Rest application based on Spring MVC wherein I am required to do some validations, some of the validations are Hard rules and some of them are Soft rules. Soft rules if they fail generate a warning, but if the hard rules fail they generate an error.
First I am checking the hard rules if any fail then, at that time only, I am returning back the response, but let the process continue to process the subsequent Soft rules.
Herein I would like to know how to create two parallel threads in Spring to do this?
OR How to publish a custom event and asynchronously handle it in another thread and let the original thread continue its work in Spring?
I know about #async and SpringTaskExecutor, but how to best use them here.
I am seeking design and architectural guidelines and ideas to handle this task in best possible way.
As mentioned soft rule(s) validation failure only generates warning it can be handled in a separate background process. This way main thread can focus solely on hard rule(s) without bothering itself about soft rule(s).
For above behavior below points need to be implemented
For every request persist the relevant data, for soft rule processing, with flag processed=false and preferably time stamp (for insertion and processed).
Post persisting the data, let the main thread continue with hard rule processing.
Introduce a scheduled service (via #Scheduled) which will periodically fetch the unprocessed data and mark them as processed=true post soft rule processing along with relevant processed time stamp. (This will as act as the background process which will periodically poll the data for unprocessed rule(s))
Do ensure that the respective transactions viz. soft rule data insertion and processing are well isolated. Also the error handling should be robust in terms of system failures when rule processing are in progress.
Let know in comments if more information is required.
What I am doing:
I am using play 2.5.7 (java) and trying to build a REST application.
When I get a call on my controller I ask the first actor, this actor can only solve part of the problem (getting additional data), which needs to be forwarded to another actor which uses the request data and additional data to update some more data, send an async void call (tell) to another actor and respond to the controller. All these (4) actors are #Injected in other actors or controller with Guice.
Flow of calls:
controller --(Patterns.ask)--> actor1 --(actor.forward)--> actor2 --(actor.forward)--> actor3 (-tell-> actor4) and --(sender().tell)--> controller.
Issue:
This works for first 4 calls. Then on actor1.forward keeps failing on every consecutive request; Patterns.ask times out. System.out on the line before actor1.forward works but not the actual forward. No matter the timeout value (tried even 20s). No change done in the request; I just hit the send button in postman every time.
I have two questions:
Why 4? Why does it fail after 4th request? Is it some config? What should I look for in config?
Is what I am doing with actors correct way to build a REST web service?
Update: I found the issue; it was caused due to consumption of Redis connections through the pool and never freeing them. But the second question I had still remains, is what I am doing here advisable?
Sure, this could be a reasonable design. But I would consider though whether it would be more maintainable to work with Future returning methods, unless your workflow requires some complex protocol between multiple moving pieces or internal state. It may also be worth considering Akka Streams, if your processing doesn't map well to async method calls.
Basically, actors are a pretty low-level tool. To the extent that you need them, I would try to minimize the surface area of your application where they are being directly used. Higher-level abstractions are better, where possible.
I have a requirement in my java web application where I need to send email alerts for certain conditions. For this I have used javax mail api and sending email works just fine. But the problem is the programs executions waits until the methods for sending the email are executed. As there are hundreds of email to be sent at various points ... this reduces the performance significantly.
I am using spring and have also used spring aop. Can anyone suggest me how can I separate my business logic and sending email functionality. It should be like -
Sending emails is my advice which gets executed when xyz method is called - So main execution should not wait for advice to finish its execution rather it should return back and execute further business logic thus email sending executed separately.
Here creating new threads seems obvious choice. But I think there could be some better way, is there? Thanks.
You can make the mail sending method #Async. This way Spring will execute this in a seperate thread. Read this blog post about it: Creating Asynchronous Methods
What you describe is asynchronous execution and natural way to do async execution is Java is to use threads.
You can introduce some Executor, e.g., Executors.newFixedThreadPool(), and use it to offload mailing task into separate threads.
Aspect itself is a unsuitable place for this, since this would introduce state into aspect, for example, you may want to check if mail task was successful by using returned Future:
class Mailer {
private final ExecutorService executor = Executors.newFixedThreadPool(maxMailingThreads);
//...
public void doMail(MailTask anEmail) {
Future<MailTaskResult> future = executor.submit(new MailTask(anEmail));
future.get().isSuccessful(); // handle success or failure somehow
}
Better move this logic into separate class and call it from aspect somehow.
Treat the email sending functionality like an IO device. Make it a plugin to your business logic. Do not allow any knowledge of the fact that you're even talking to the email code into your business logic. Make the email logic depend on the business logic. Never the other way around.
Here's a very good talk about this kind of architecture:
https://vimeo.com/97530863
Here's a series debating it:
https://www.youtube.com/watch?v=z9quxZsLcfo
Here's a ruby master demonstrating it with real code. We miss him.
https://www.youtube.com/watch?v=tg5RFeSfBM4
If your business rules are interesting enough to be worth respecting than this is the way to make them the masters of your application. Express them only using java. Don't accept any help. No spring, no weird annotations, just business rules. Push all that "help" out to the mail code.
Do this and your app will scale well. I think this is the best way to put it:
That's from a hexagonal architecture post. But the idea of giving your business rules a safe place to live removed from implementation detail shows up in many architectures. This answer rounds them up nicely.
Use a localhost MTA (like OpenSMTPD) and then relay to your real SMTP server, like Amazon SES ("Satellite" mode). It won't block.
I did a test, and sent 1000 emails in 2.8 seconds this way
It's simpler than doing async in java, and is useful across multiple applications.
As for separating logic, raise a Spring Application Event when needed, and make another class to listen to it, and send your email from there. Or consider something like Guava's EventBus
Consider creating a separate thread to send emails within your application. This will allow parallel execution(application+email sending).
If you would want another approach you can create a separate back end application that only sends emails. Although you will need to submit the email messages to the application. An asynchronous way to do this is to send a JMS message to the email application.
Let me try explaining the situation:
There is a messaging system that we are going to incorporate which could either be a Queue or Topic (JMS terms).
1 ) Producer/Publisher : There is a service A. A produces messages and writes to a Queue/Topic
2 ) Consumer/Subscriber : There is a service B. B asynchronously reads messages from Queue/Topic. B then calls a web service and passes the message to it. The webservice takes significant amount of time to process the message. (This action need not be processed real-time.)
The Message Broker is Tibco
My intention is : Not to miss out processing any message from A. Re-process it at a later point in time in case the processing failed for the first time (perhaps as a batch).
Question:
I was thinking of writing the message to a DB before making a webservice call. If the call succeeds, I would mark the message processed. Otherwise failed. Later, in a cron job, I would process all the requests that had initially failed.
Is writing to a DB a typical way of doing this?
Since you have a fail callback, you can just requeue your Message and have your Consumer/Subscriber pick it up and try again. If it failed because of some problem in the web service and you want to wait X time before trying again then you can do either schedule for the web service to be called at a later date for that specific Message (look into ScheduledExecutorService) or do as you described and use a cron job with some database entries.
If you only want it to try again once per message, then keep an internal counter either with the Message or within a Map<Message, Integer> as a counter for each Message.
Crudely put that is the technique, although there could be out-of-the-box solutions available which you can use. Typical ESB solutions support reliable messaging. Have a look at MuleESB or Apache ActiveMQ as well.
It might be interesting to take advantage of the EMS platform your already have (example 1) instead of building a custom solution (example 2).
But it all depends on the implementation language:
Example 1 - EMS is the "keeper" : If I were to solve such problem with TIBCO BusinessWorks, I would use the "JMS transaction" feature of BW. By encompassing the EMS read and the WS call within the same "group", you ask for them to be both applied, or not at all. If the call failed for some reason, the message would be returned to EMS.
Two problems with this solution : You might not have BW, and the first failed operation would block all the rest of the batch process (that may be the desired behavior).
FYI, I understand it is possible to use such feature in "pure java", but I never tried it : http://www.javaworld.com/javaworld/jw-02-2002/jw-0315-jms.html
Example 2 - A DB is the "keeper" : If you go with your "DB" method, your queue/topic customer continuously drops insert data in a DB, and all records represent a task to be executed. This feels an awful lot like the simple "mapping engine" problem every integration middleware aims to make easier. You could solve this with anything from a custom java code and multiples threads (DB inserter, WS job handlers, etc.) to an EAI middleware (like BW) or even a BPM engine (TIBCO has many solutions for that)
Of course, there are also other vendors... EMS is a JMS standard implementation, as you know.
I would recommend using the built in EMS (& JMS) features,as "guaranteed delivery" is what it's built for ;) - no db needed at all...
You need to be aware that the first decision will be:
do you need to deliver in order? (then only 1 JMS Session and Client Ack mode should be used)
how often and in what reoccuring times do you want to retry? (To not make an infinite loop of a message that couldn't be processed by that web service).
This is independent whatever kind of client you use (TIBCO BW or e.g. Java onMessage() in a MDB).
For "in order" delivery: make shure only 1 JMS Session processes the messages and it uses Client acknolwedge mode. After you process the message sucessfully, you need to acknowledge the message with either calling the JMS API "acknowledge()" method or in TIBCO BW by executing the "commit" activity.
In case of an error you don't execute the acknowledge for the method, so the message will be put back in the Queue for redelivery (you can see how many times it was redelivered in the JMS header).
EMS's Explicit Client Acknolwedge mode also enables you to do the same if order is not important and you need a few client threads to process the message.
For controlling how often the message get's processed use:
max redelivery properties of the EMS queue (e.g. you could put the message in the dead
letter queue afer x redelivery to not hold up other messages)
redelivery delay to put a "pause" in between redelivery. This is useful in case the
Web Service needs to recover after a crash and not gets stormed by the same message again and again in high intervall through redelivery.
Hope that helps
Cheers
Seb