I have very huge camel pipeline which starts from consuming message from SQS.
The time requires for whole process is vary. From 5 sec to 30 min, it is hard to guess here.
What I want to achieve:
Do not guess visibility timeout size, and just delete message from SQS as soon as message is consumed.
What I already tried:
Tried Camel option deleteAfterRead=true -> doesn't help, because as stated in the doc: Delete message from SQS after it has been read (and processed by the route). And I have huge pipeline. So processed requirement fails here.
Tried to increase visibility timeout, but as I stated, it is just a guessing game, and I want to develop more reliable solution.
Thank you for help!
You should certainly NOT delete the message as soon as you consume it if you are not finished processing it because you will lose the message if your application crashes.
What you should do instead is extend the visibility timeout manually if you realize that your processing time gets close to the original visibility timeout. Not sure how you would implement it in Camel but we previously integrated into the default springframework.cloud.aws.messaging: https://github.com/Mercateo/sqs-utils
If, like the OP has stated, you don't mind about losing the messages in case e.g. of a server crash, there is a workaround consisting in sending the message to a wireTap and doing your processing there. This allows the main route to end early and a deleteMessage to be sent to the queue.
Another interesting, realted situation is that of an exception: if you have an exception handler in place and mark the exception as handled calling handled(true) on the OnExceptionDefinition, your message will be deleted too.
Related
I have implemented a retry mechanism which works well based on the following:
https://github.com/spring-projects/spring-integration-samples/issues/237
The application consumes events from kafka, transforms those events and sends them as an HTTP request to a remote service, so it's in the integration flow that sends the HTTP request where the retry mechanism is implemented.
I was worried about sending the requests to the remote service in the same order as they come in from kafka during a temporary failure (network glitch) to avoid an overriding, but fortunately it looks like the order is kept, keep me honest here.
It seems that during the retry process all events coming in are "put on hold" and once the remote service is back up before the last try, all events are sent.
I would like to know two things here:
Am I correct with my assumption? Is this how the retry mechanism works by default?
I'm worried about the events getting back (or stack) up due to the amount of time it takes to finish the current flow execution. Is there something here I should take into consideration?
I think I might use an ExecutorChannel so that events could get processed in parallel, but by doing that I wouldn't be able to keep the order of the events.
Thanks.
Your assumption is correct. The retry is done withing the same thread and it is blocked for the next event until the send is successful or retry is exhausted. And it is really done in the same Kafka consumer thread, so new records are not pulled from the topic until retry is done.
It is not a correct architecture to shift the logic into a new thread, e.g. using an ExecutorChannel since Kafka is based on an offset commit which cannot be done out of order.
We are developing a SpringBoot application where messages are produced into a Kafka queue, and are pulled by Listeners (across multiple servers, multiple threads can be processing on each server).
When message is processed successfully, we would like to do something like Acknowledgement.acknowledge() in the Listener, to mark the message as processed.
However, according to documentation:
https://docs.spring.io/spring-kafka/api/org/springframework/kafka/support/Acknowledgment.html:
Calling this method implies that all the previous messages in the partition have been processed already
I can see this causing a problem with multi threads and race conditions...
Similar unanswered question:
Spring Kafka is Acknowledgement.acknowledge thread safe?
So, I guess I am asking if Acknowledgement.acknowledge() is not the right way to do it, what is? There has to be a way to mark specific message (not the whole offset) as processed...
Thanks!
I am using vert.x to read a file and transform and then push to kafka.
I am using 2 verticles, without using any worker thread (I dont want to change the order of logs in the file).
Verticle 1 : Read the file and filter
Verticle 2 : Publish to kafka
Each files contain approximately 120000 lines
However, I observed that after sometime i stop observing logs from verticle 1.
I am suspecting that event bus is getting full, so Consumer is still consuming, but producer thread is waiting for event bus to get empty.
So My questions are
1. What is the default size of event bus? In Docs it says
DEFAULT_ACCEPT_BACKLOG
The default accept backlog = 1024
2. How do I confirm my suspicion that publisher thread is blocked?
VertX uses Netty's SingleThreadEventLoop internally for its event bus, maximum pending tasks allowed is Integer.MAX_VALUE which is probably 2 billion messages.
You may have to try VertxOptions.setWarningExceptionTime(long warningExceptionTime) to set the value lower than default (5sec) to see if there is any warning about blocked thread.
To complement #iwat answer, in the version I am using, it looks like the max size is read from a system property:
protected static final int DEFAULT_MAX_PENDING_TASKS = Math.max(16, SystemPropertyUtil.getInt("io.netty.eventLoop.maxPendingTasks", 2147483647));
So you can control the size of the queues in front of the Verticles by setting that system property.
If the event bus is full (the queue in NioEventLoop reaches the max size), the task will be rejected. So if you hit that, you should start to see error responses to your messages, you should not see any blocked producers.
I'm not sure the accept-backlog setting has any effect on the eventbus, given the documentation it might have something to do with the netserver, but from a short scan of the code I haven't found any use in the eventbus.
The event bus however does deliver the message immediately, messages don't get queued up somewhere (at least that's what I understand from the code). So regarding your first question, it doesn't have any size, at least not when running locally (don't know about the clustered version, but I assume that doesn't apply in your case anyway)
To confirm an (eventloop) thread is actually blocked is easy, there should be tons of exceptions in your log stating the event loop is blocked.
I guess your problem is somewhere else, but that's actually hard to tell without any code or meaningful logs.
I am using Apache Camel with ActiveMQ and wanting to implement guaranteed message delivery.
I have been reading through the Camel in Action book as well as the Apache Camel Developer's Cookbook.
I am hoping someone here can advise me in my approach. I am not asking for code samples.
The way I envisioned the implementation is as follows:
1. Message is received on an endpoint
2. I inspect the message
3. I use the Wiretap pattern to drop it immediately on my "GuaranteedMessages" queue if the message asks for guaranteed delivery
4. I route the message to its proper destination
5. If no exceptions were encountered, I remove the message manually from the "GuaranteedMessages" queue
Sounds easy enough. However, I have been reading about the "Dead Letter Channel" pattern in Camel.
My understanding is using this pattern's implementation implies that instead of automatically dropping each (guaranteed) message to my "GuaranteedMessages" queue, I drop that approach and instead, I set the redelivery options (max attempts, exponential delay, redelivery delay, etc.). Then, I rely on Camel to try redelivering and simply drop it off in the dead letter channel delay if it never goes through.
Then, I would have a separate route that uses this dead letter queue as it's source. Again, it would be the same pattern. If it cannot succeed, send the message back to the dead letter queue.
Does this sound like a realistic implementation for a production system?
If instead, I decide to drop every incoming message (that needs to be guaranteed) on my own "GuaranteeMessage" queue, is it realistic to believe that I can later go and remove that specific message manually from the queue? My understanding is that I would have to manually browse the queue, iterate through any number of messages, and then consume that message manually. I am not sure how scalable such an architecture really is.
Presumably the final destination is not another ActiveMQ queue, it something that can through exception. Your idea of the wiretap is functionally the same as using the DLQ so you might as well use the Camel functionality, which works fine, to do as much work as possible.
However, two points. Firstly I would use an explicit queue to hold the messages that need retrying, rather than the DLQ, as there is only one DLQ per broker and you don't want anything else unexpected appearing on it.
Secondly if you are just going to take a message from the retry queue and resubmit it, why not just increase the retry count and delay in Camel exception handling? That way your retry queue just has messages that probably require some manual intervention. So when a message is on the retry queue, you manually check/fix whatever the underlying cause is and manually move the message to the input queue.
Let me try explaining the situation:
There is a messaging system that we are going to incorporate which could either be a Queue or Topic (JMS terms).
1 ) Producer/Publisher : There is a service A. A produces messages and writes to a Queue/Topic
2 ) Consumer/Subscriber : There is a service B. B asynchronously reads messages from Queue/Topic. B then calls a web service and passes the message to it. The webservice takes significant amount of time to process the message. (This action need not be processed real-time.)
The Message Broker is Tibco
My intention is : Not to miss out processing any message from A. Re-process it at a later point in time in case the processing failed for the first time (perhaps as a batch).
Question:
I was thinking of writing the message to a DB before making a webservice call. If the call succeeds, I would mark the message processed. Otherwise failed. Later, in a cron job, I would process all the requests that had initially failed.
Is writing to a DB a typical way of doing this?
Since you have a fail callback, you can just requeue your Message and have your Consumer/Subscriber pick it up and try again. If it failed because of some problem in the web service and you want to wait X time before trying again then you can do either schedule for the web service to be called at a later date for that specific Message (look into ScheduledExecutorService) or do as you described and use a cron job with some database entries.
If you only want it to try again once per message, then keep an internal counter either with the Message or within a Map<Message, Integer> as a counter for each Message.
Crudely put that is the technique, although there could be out-of-the-box solutions available which you can use. Typical ESB solutions support reliable messaging. Have a look at MuleESB or Apache ActiveMQ as well.
It might be interesting to take advantage of the EMS platform your already have (example 1) instead of building a custom solution (example 2).
But it all depends on the implementation language:
Example 1 - EMS is the "keeper" : If I were to solve such problem with TIBCO BusinessWorks, I would use the "JMS transaction" feature of BW. By encompassing the EMS read and the WS call within the same "group", you ask for them to be both applied, or not at all. If the call failed for some reason, the message would be returned to EMS.
Two problems with this solution : You might not have BW, and the first failed operation would block all the rest of the batch process (that may be the desired behavior).
FYI, I understand it is possible to use such feature in "pure java", but I never tried it : http://www.javaworld.com/javaworld/jw-02-2002/jw-0315-jms.html
Example 2 - A DB is the "keeper" : If you go with your "DB" method, your queue/topic customer continuously drops insert data in a DB, and all records represent a task to be executed. This feels an awful lot like the simple "mapping engine" problem every integration middleware aims to make easier. You could solve this with anything from a custom java code and multiples threads (DB inserter, WS job handlers, etc.) to an EAI middleware (like BW) or even a BPM engine (TIBCO has many solutions for that)
Of course, there are also other vendors... EMS is a JMS standard implementation, as you know.
I would recommend using the built in EMS (& JMS) features,as "guaranteed delivery" is what it's built for ;) - no db needed at all...
You need to be aware that the first decision will be:
do you need to deliver in order? (then only 1 JMS Session and Client Ack mode should be used)
how often and in what reoccuring times do you want to retry? (To not make an infinite loop of a message that couldn't be processed by that web service).
This is independent whatever kind of client you use (TIBCO BW or e.g. Java onMessage() in a MDB).
For "in order" delivery: make shure only 1 JMS Session processes the messages and it uses Client acknolwedge mode. After you process the message sucessfully, you need to acknowledge the message with either calling the JMS API "acknowledge()" method or in TIBCO BW by executing the "commit" activity.
In case of an error you don't execute the acknowledge for the method, so the message will be put back in the Queue for redelivery (you can see how many times it was redelivered in the JMS header).
EMS's Explicit Client Acknolwedge mode also enables you to do the same if order is not important and you need a few client threads to process the message.
For controlling how often the message get's processed use:
max redelivery properties of the EMS queue (e.g. you could put the message in the dead
letter queue afer x redelivery to not hold up other messages)
redelivery delay to put a "pause" in between redelivery. This is useful in case the
Web Service needs to recover after a crash and not gets stormed by the same message again and again in high intervall through redelivery.
Hope that helps
Cheers
Seb