I have a pretty easy route where I pickup files from a directory and send it to a bean:
from("file:/mydir?delete=true").bean(MyProcessor.class);
It can happen that an exception occures in MyProcessor.class and so I want to delay the processing of that file again. How can I setup a redelivery for that as I tried already different things with
onException().redeliveryDelay(10000);
but it didn't work and right after the exception the same file gets processed again.
Did you do onException() before Processing?
Example:
errorHandler(defaultErrorHandler()
.maximumRedeliveries(2)
.redeliveryDelay(5000)
.retryAttemptedLogLevel(LoggingLevel.WARN));
// exception handler for specific exceptions
onException(IOException.class).maximumRedeliveries(1).redeliveryDelay(5000);
//only the failed record send write to error folder
onException(CsvRecordException.class)
.to("file:/app/dev/dataland/error");
onCompletion()
.log("global thread: ${threadName}")
.to("file:/app/dev/dataland/archive");
from("file:/path?noop=true?delay=3000")
.startupOrder(1)
.log("start to process file: ${header.CamelFileName}")
.bean(CsvFilePreLoadChecker.class, "validateMetaData")
.end()
.log("Done processing file: ${header.CamelFileName}");
When an error occurs in MyProcessor.class the route processing is failed and therefore the file consumer does not delete the file.
Since the route processing is completed, the file consumer simply reads the (still present) file again.
If you want to move files with processing errors out of your way, you can use the moveFailed option of the file consumer. You would then have to move them back periodically to retry.
If you want to decouple file reading and MyProcessor.class you need to split the route into 2 routes. One that reads read the files and sends its messages to a queue or similar. The other consumes that queue and processes the messages.
Related
I try to use Camel to deliver files from one folder to a rest call and Im trying to achieve that on Error it's tried to redeliver twice and then moved to an error folder if the second redelivery fails as well. My code in the RouteBuilder's configure method looks like this:
errorHandler(deadLetterChannel("file:///home/camelerror").useOriginalMessage());
from("file:///home/camelefiles")
.onException(RetryableException.class)
.log("RetryableException handled")
.maximumRedeliveries(2)
.end()
.routeId(port.id())
.throwException(new RetryableException());
I get the "RetryableException handled" logs so I guess the exception is handled correctly but it redelivers the message an infinite number of times.
What am I doing wrong and how can I achieve that the message is only redelivered twice and then the deadLetterChannel is used?
I have a camel application where, I am reading files from FTP source.
Then the file goes through multiple routes, like one route goes to cassandra for storage, one route process the data and push pivoted data to Kafka topic etc
I want to mark the file processed when it goes through all routes and reaches till the end. This way I can build a processing completed log based on file name.
One way, That I can think of is to implement aggregator, where each route will send completion notification in exchange header and then based on the completion criteria logic in aggregator, I will mark that file as processed.
How will I write such aggregator in java?
You could try using multicast.
from("direct:start")
.multicast()
.to("direct:a","direct:b")
.end()
// Won't run until the sub routes are complete
.process(new MarkFileAsCompletedProcessor())
.log("Finished multicast");
from("direct:a")
.log("Processing a")
.to("mock:endOfA");
from("direct:b")
.log("Processing b")
.to("mock:endOfB");
I want to put continue behaviour in route, my route is like following
from("file:D:\\?fileName=abc.csv&noop=true").split().unmarshal().csv()
.to("direct:insertToDb").end();
from("direct:insertToDb")
.to("direct:getDataId")
.to("direct:getDataParameters")
.to("direct:insertDataInDb");
from("direct:getDataId")
.to("sql:SELECT id FROM data WHERE name = :#name)
.choice()
.when(header("id").isGreaterThan(0) )
.setProperty("id", header("id"))
.otherwise()
.log("Error for")
.endChoice().end();
I want that if direct:getDataId dont find any record , my execution of route for current record from CSV get skip and program process next request. it would be equal to continue keyword.
How i can achieve this in Apache Camel route?
You can modify your routes like this:
from("file:D:\\?fileName=abc.csv&noop=true").split().unmarshal().csv()
.to("sql:SELECT id FROM data WHERE name = :#name?outputHeader=id&outputType=SelectOne)
.choice().when(header("id").isGreaterThan(0))
.to("direct:getDataParameters")
.to("direct:insertDataInDb")
.end();
Have you got a test for this? I suggest you try using CamelTestSupport because what you want is how camel will execute by default.
From Camel Split Docs:
stopOnException
default:false
description: Whether or not to stop continue processing immediately when an exception occurred. If disable, then Camel continue splitting and process the sub-messages regardless if one of them failed. You can deal with exceptions in the AggregationStrategy class where you have full control how to handle that.
I'm trying to use a Camel poll-once route that will use a file if it's present and log an error if not.
By default the route does nothing if the file does not exist so I've started by adding consumer.sendEmptyMessageWhenIdle=true to the URI. I then check for null body to decide whether to log an exception or continue:
from(theFileUri)
.onCompletion()
.onCompleteOnly()
.log("SUCCESS")
.bean(theOtherAction, "start")
.end()
.onException(Exception.class)
.logStackTrace(true)
.log(ERROR, "Failed to load file")
.handled(true)
.end()
.choice()
.when(body().isNotNull())
.to(NEXT_ROUTE_URI)
.endChoice()
.otherwise()
.throwException(new FileNotFoundException(theFileUri))
.endChoice();
There are two problems with this:
if the file is missing, the success log still happens (but not the bean!)
the stack trace is not printed
If there is a better way to do this then I'd welcome suggestions but I'd also like to know what I'm doing wrong in this method.
It's still not completely clear to me what is going on. However, I believe that the onException call needs to be separated from the chain. It looks to me that logStackTrace applies only to redelivery attempts. The number of attempts defaults to 0 and this is what I want. The only way to access the exception from the Java DSL appears to be a custom Processor. The getException() method will return null if you are using handled(true) so you must use Exchange.getProperty(Exchange.EXCEPTION_CAUGHT, Exception.class).
I also suspect that the log message from the onCompletion is due to it running in parallel before the exception aborts:
Camel 2.13 or older - On completion runs in separate thread Icon The
onCompletion runs in a separate thread in parallel with the original
route. It is therefore not intended to influence the outcome of the
original route. The idea for on completion is to spin off a new thread
to eg send logs to a central log database, send an email, send alterts
to a monitoring system, store a copy of the result message etc.
Therefore if you want to do some work that influence the original
route, then do not use onCompletion for that. Notice: if you use the
UnitOfWork API as mentioned in the top of this page, then you can
register a Synchronization callback on the Exchange which is executed
in the original route. That way allows you to do some custom code when
the route is completed; this is how custom components can enlist on
completion services which they need, eg the File component does that
for work that moves/deletes the original file etc.
Since I want to not run this code on exception, I think I can just abort the route with the exception.
I currently have this:
onException(Exception.class)
.handled(true)
.process(new Processor()
{
#Override
public void process(Exchange anExchange) throws Exception
{
Exception myException = anExchange.getProperty(Exchange.EXCEPTION_CAUGHT, Exception.class);
LOGGER.error("Failed to load", myException);
}
});
from(theFileUri)
.choice()
.when(body().isNotNull())
.to(NEXT_ROUTE_URI)
.log("SUCCESS")
.bean(theOtherAction, "start")
.endChoice()
.otherwise()
.throwException(new FileNotFoundException(theFileUri));
I have web service method that is supposed to process a very large file and output several files to the server. However, this web service will just timeout and there will be no way for the invoker to get the CREATED status. I am just wondering whether there is a way to run the processing job (starting a new thread or something) and return the status without waiting for the process to be done.
public Response processFile(InputStream inputStream){
//I want to process the file here
//but I dont want the invoker to wait for that process to finish
//I just want the response to be returned right away
return Response.status(Response.Status.CREATED).build();
}
The file comes from the input stream, right? So if you'll send back a CREATED status (effectually closing the connection) you might sever the connection before you receive the entirety of the input file?
That's what i thought anyways... In which case you'll just want to set the timeout to a lengthier value.
If that's not the case, then i guess it would be fine to start a new thread, process everything there in good time and send back the CREATED status.