I have a listener to the JMS. Once I read the message then I convert to my custom object
public IntegrationFlow queueProcessorFlow() {
return IntegrationFlows.from(Jms.inboundAdapter(jmsTemplate)
.destination("test_queue"),
c -> c.poller(Pollers.fixedDelay(5000L)
.maxMessagesPerPoll(1)))
//convert json to our custom object
.transform(new JsonToQueueEventConverterTransformer(springBeanFactory))
.transform(new CustomTransformer(springBeanFactory))
.handle(o -> {
}).get();
}
The transformer
public class CustomerTransformer implements GenericTransformer<CustomPojo, CustomPojo> {
private final QueueDataProcessorSpringBeanFactory factory;
#Override
public CustomPojo transform(CustomPojo CustomPojo) {
try {
//do something e.g. service call
throw new Exception("This failed mate !! SOS");
} catch (Exception e) {
//ISSUE here
//e contains the original payload in the stack trace
throw new RuntimeException(e);
}
return CustomPojo;
}
Now when I throw my custom exception the stack trace contains everything. It even contains the payload. I am not interested in the payload in case of exception.
How do I update not to include payload?
** Update **
After changing as per answer I still see the issue
org.springframework.integration.transformer.MessageTransformationException: Failed to transform Message; nested exception is org.springframework.messaging.MessageHandlingException: nested exception is org.springframework.integration.transformer.MessageTransformationException: Error initiliazing the :; nested exception is CustomException Error lab lab lab , failedMessage=GenericMessage [payload=
my error handler
#Bean
public IntegrationFlow errorHandlingFlow() {
return IntegrationFlows.from("errorChannel")
.handle(message -> {
try {
ErrorMessage e = (ErrorMessage) message;
if (e.getPayload() instanceof MessageTransformationException) {
String stackTrace = ExceptionUtils.getStackTrace(e.getPayload());
LOG.info("Exception trace {} ", stackTrace);
Not sure what is the business purpose to lose a payload in the stack trace, but you can achieve that throwing a MessageTransformationException instead of that RuntimeException.
To avoid a message in stack trace with the mentioned payload, you need to use one of these constructors:
public MessageTransformationException(String description, Throwable cause) {
super(description, cause);
}
public MessageTransformationException(String description) {
super(description);
}
Instead of those based on the Message<?>.
This way a wrapping MessageTransformingHandler will do an appropriate logic:
protected Object handleRequestMessage(Message<?> message) {
try {
return this.transformer.transform(message);
}
catch (Exception e) {
if (e instanceof MessageTransformationException) {
throw (MessageTransformationException) e;
}
throw new MessageTransformationException(message, "Failed to transform Message", e);
}
}
UPDATE
It turned out that MessageTransformationException is not enough since the AbstractMessageHandler checks for the MessageHandlingException for wrapping in the IntegrationUtils.wrapInHandlingExceptionIfNecessary(). Therefore I suggest to throw a MessageHandlingException from your code instead. And use this constructor with the null for the message arg:
MessageHandlingException(Message<?> failedMessage, Throwable cause)
I had almost the same issue maybe this can help you. If you use the default errorChannel Bean this already has been subscribed to a LoggingHandler which prints the the full message, if you want avoid printing the payload you can create your own errorChannel by this way you'll override the default behavior
#Bean
#Qualifier(IntegrationContextUtils.ERROR_CHANNEL_BEAN_NAME)
public MessageChannel errorChannel() {
return new PublishSubscribeChannel();
}
If your problem is when you use the .log() handler you can always use a function to decide which part of the Message you want to show
#Bean
public IntegrationFlow errorFlow(IntegrationFlow
createOutFileInCaseErrorFlow) {
return
IntegrationFlows.from(IntegrationContextUtils.ERROR_CHANNEL_BEAN_NAME)
.log(LoggingHandler.Level.ERROR, m -> m.getHeaders())
.<MessagingException>log(Level.ERROR, p -> p.getPayload().getMessage())
.get();
}
Related
I had to customized Sftp Inbound default handler LoggingHandler and using my own CustomizedErrorHandler which extends ErrorHandler. But I can't return any message to my controller after handling exceptions.
I was researching couple of days and I found nothing to show my customized message to my UI using Controller. Below are some code snippet from my CustomizedErrorHandler, SftpInboundConfiguration.
SftpInboundConfiguration
public IntegrationFlow fileFlow() {
SftpInboundChannelAdapterSpec spec = Sftp
.inboundAdapter(getSftpSessionFactory())
.preserveTimestamp(true)
.remoteDirectory(getSourceLocation())
.autoCreateLocalDirectory(true)
.deleteRemoteFiles(false)
.localDirectory(new File(getDestinationLocation()));
return IntegrationFlows
.from(spec, e -> e.id(BEAN_ID)
.autoStartup(false)
.poller(sftpPoller())
)
.channel(sftpReceiverChannel())
.handle(sftpInboundMessageHandler())
.get();
}
... ... ...
public PollerMetadata sftpPoller() {
PollerMetadata pollerMetadata = new PollerMetadata();
List<Advice> adviceChain = new ArrayList<>();
pollerMetadata.setErrorHandler(customErrorMessageHandler());
pollerMetadata.setTrigger(new PeriodicTrigger(5000));
return pollerMetadata;
}
... ... ...
private CustomErrorMessageHandler customErrorMessageHandler() {
return new CustomErrorMessageHandler(
controlChannel(),
BEAN_ID
);
}
CustomErrorMessageHandler
public class CustomErrorMessageHandler implements ErrorHandler {
private final MessageChannel CONTROL_CHANNEL;
private final String BEAN_ID;
public CustomErrorMessageHandler(
MessageChannel controlChannel,
String beanID
) {
this.CONTROL_CHANNEL = controlChannel;
this.BEAN_ID = beanID;
}
public void handleError(#NotNull Throwable throwable) {
final Throwable rootCause = ExceptionUtils.getRootCause(throwable);
if (rootCause instanceof MessagingException) {
log.error("MessagingException : {} ", rootCause.getMessage());
} else if (rootCause instanceof SftpException) {
log.error("SftpException : {}", rootCause.getMessage());
} ... ... ...
else {
log.error("Unknown : Cause : {} , Error : {}",
rootCause, rootCause.getMessage());
}
log.info("Stopping SFTP Inbound");
boolean is_stopped = CONTROL_CHANNEL.send(
new GenericMessage<>("#" + BEAN_ID + ".stop()"));
if (is_stopped) {
log.info("SFTP Inbound Stopped.");
} else {
log.info("SFTP Inbound Stop Failed.");
}
}
}
Now I want to save some customized message from if-else statements and need to show it in UI. Is there any way to save the message and show it using Route or Controller ?
Don't customize the error handler, use poller.errorChannel("myErrorChannel") instead.
Then add an error channel flow
#Bean
IntegrationFlow errors() {
return IntegrationFLows.from("myErrorChannel")
.handle(...)
...
.get();
The message sent to the handler is an ErrorMessage with a MessagingException payload, with cause and failedMessage which was the message at the point of the failure and originalMessage which is the original message emitted by the adapter.
After handling the exception, you can simply call a method on your controller to tell it the state.
I have a rest controller that has an endpoint:
#GET
#Path("/reindex-record")
public String reindexRecord(#QueryParam("id") String id) {
if (StringUtils.isEmpty(id)) {
CompletableFuture.runAsync(
() -> runWithException(Reindexer::reindexAll));
} else {
CompletableFuture.runAsync(() -> runWithException(
() -> Reindexer.reindexOne(id)));
}
// return "ok" or throw WebApplciationException from runWithException method below
}
and here is my wrapper method - both methods - reindexAll and reindexOne throw checked exceptions so decided to use wrapper method and interface:
public interface RunnableWithException {
void run() throws Exception;
}
private void runWithException(RunnableWithException task) {
try {
task.run();
} catch (Exception e) {
log.error("Error occured during async task execution", e);
throw new WebApplicationException(
Response.status(Response.Status.INTERNAL_SERVER_ERROR)
.entity("Internal error occurred").build());
}
}
The problem is that I want to run this task asnychronously using CompleteableFuture and give a response only after given task is done or if there was an error throw WebApplicationException with INTERNAL_SERVER_ERROR status.
How would you implement that in my use-case with if/else?
EDIT:
As of now I have this method:
#GET
#Path("/reindex-record")
public String reindexRecord(#QueryParam("id") String id) throws ExecutionException,
InterruptedException {
CompletableFuture<Void> task;
if (StringUtils.isEmpty(id)) {
task = CompletableFuture.runAsync(
() -> runWithException(Reindexer::reindexAll));
} else {
task = CompletableFuture.runAsync(() -> runWithException(
() -> Reindexer.reindexOne(id)));
}
return task.thenApply(x -> "ok")
.exceptionally(throwable -> {
log.error("Error occured during async task execution", throwable);
throw new WebApplicationException(Response.status(Response.Status.SERVICE_UNAVAILABLE)
.entity("Internal error occurred. Try again later")
.build());
}).get();
}
But if error is thrown by any of Reindexer methods I'm still getting status 500 with data:
{
"code": 500,
"message": "There was an error processing your request. It has been logged (ID 03f09a62b62b1649)."
}
Instead of 503 defined in my exceptionally block.
Using dropwizard with JAX-RS if that matters.
You can change the body of your method to this:
#GET
#Path("/reindex-record")
public String reindexRecord(#QueryParam("id") String id) {
final CompletableFuture<Void> future;
if (StringUtils.isEmpty(id)) {
future = CompletableFuture.runAsync(
() -> runWithException(Reindexer::reindexAll));
} else {
future = CompletableFuture.runAsync(
() -> runWithException(() -> Reindexer.reindexOne(id)));
}
// this will block
future.get();
return "ok";
}
By storing the future, you can then call the get() method on it, which will block until the future is finished.
From the javadoc of CompletableFuture.get():
Waits if necessary for this future to complete, and then returns its result.
The problem is that you are using exceptionally() for something it wasn't intended. CompletableFutureis intented to be used in a chain of CompletableFutures where the output of one feeds into the next one. What happens if one of the CompletableFutures throws an exception? You can use exceptionally to catch that and return a new fallback ComletableFuture for the next step of the chain to use instead.
You're not doing that, you just throw an WebApplicationException. The CompleteableFuture views that as a failure in the chain and wraps your WebApplicationException inside an ExecutionException. Dropwizard only sees the ExecutionException (it doesn't inspect any wrapped ones) and throws the generic 500 response.
The answer is just do the future.get(); in #Lino's answer, but wrapped in a try...catch block for ExecutionException, and then throw WebApplicationException from the catch.
try {
// this will block
future.get();
} catch (ExecutioException) {
throw new WebApplicationException(
Response.status(Response.Status.INTERNAL_SERVER_ERROR)
.entity("Internal error occurred").build());
}
return "ok";
You might be able to shorten the whole throw new WebApplicationException(... to simple throw new InternalServerErrorException()
I have following configuration for creation of two channels (by using the JmsChannelFactoryBean):
#Bean
public JmsChannelFactoryBean jmsChannel(ActiveMQConnectionFactory activeMQConnectionFactory) {
JmsChannelFactoryBean fb = new JmsChannelFactoryBean(true);
fb.setConnectionFactory(activeMQConnectionFactory);
fb.setDestinationName("something.queue");
fb.setErrorHandler(t -> log.error("something went wrong on jms channel", t));
return fb;
}
#Bean
public JmsChannelFactoryBean jmsChannelDLQ(ActiveMQConnectionFactory activeMQConnectionFactory) {
JmsChannelFactoryBean fb = new JmsChannelFactoryBean(true);
fb.setConnectionFactory(activeMQConnectionFactory);
fb.setDestinationName("something.queue.DLQ");
fb.setErrorHandler(t -> log.error("something went wrong on jms channel", t));
return fb;
}
The something.queue is configured to put the dead letter on something.queue.DLQ. Im using mostly Java DSL to configure the app, and if possible - would like to keep this.
Case is: the message is taken from jmsChannel put to sftp outbound gateway, if there is a problem on sending the file, the message is put back into the jmsChannel as not delivered. After some retries it is designed as poisonus, and put to something.queue.DLQ.
Is it possbile to have the info on error channel when that happens?
What is best practice to handle errors when using JMS backed message channels?
EDIT 2
The integration flow is defined as:
IntegrationFlows.from(filesToProcessChannel).handle(outboundGateway)
Where filesToProcessChannel is the JMS backed channel and outbound gateway is defined as:
#Bean
public SftpOutboundGateway outboundGateway(SftpRemoteFileTemplate sftpRemoteFileTemplate) {
SftpOutboundGateway gateway = new SftpOutboundGateway(sftpRemoteFileTemplate, AbstractRemoteFileOutboundGateway.Command.PUT.getCommand(), EXPRESSION_PAYLOAD);
ArrayList<Advice> adviceChain = new ArrayList<>();
adviceChain.add(errorHandlingAdvice());
gateway.setAdviceChain(adviceChain);
return gateway;
}
Im trying to grab exception using advice:
#Bean
public Advice errorHandlingAdvice() {
RequestHandlerRetryAdvice advice = new RequestHandlerRetryAdvice();
RetryTemplate retryTemplate = new RetryTemplate();
SimpleRetryPolicy retryPolicy = new SimpleRetryPolicy();
retryPolicy.setMaxAttempts(1);
retryTemplate.setRetryPolicy(retryPolicy);
advice.setRetryTemplate(retryTemplate);
advice.setRecoveryCallback(new ErrorMessageSendingRecoverer(filesToProcessErrorChannel));
return advice;
}
Is this the right way?
EDIT 3
There is certanly something wrong with SFTPOutboundGateway and advices (or with me :/):
I used the folowing advice from the spring integration reference:
#Bean
public Advice expressionAdvice() {
ExpressionEvaluatingRequestHandlerAdvice advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setSuccessChannelName("success.input");
advice.setOnSuccessExpressionString("payload + ' was successful'");
advice.setFailureChannelName("failure.input");
advice.setOnFailureExpressionString(
"payload + ' was bad, with reason: ' + #exception.cause.message");
advice.setTrapException(true);
return advice;
}
#Bean
public IntegrationFlow success() {
return f -> f.handle(System.out::println);
}
#Bean
public IntegrationFlow failure() {
return f -> f.handle(System.out::println);
}
And when I use :
return IntegrationFlows.from(filesToProcessChannel)
.handle((GenericHandler<File>) (payload, headers) -> {
if (payload.equals("x")) {
return null;
}
else {
throw new RuntimeException("some failure");
}
}, spec -> spec.advice(expressionAdvice()))
It gets called, and i get error message printed out (and that is expected), but when I try to use:
return IntegrationFlows.from(filesToProcessChannel)
.handle(outboundGateway, spec -> spec.advice(expressionAdvice()))
The advice is not called, and the error message is put back to JMS.
The app is using Spring Boot v2.0.0.RELEASE, Spring v5.0.4.RELEASE.
EDIT 4
I managed to resolve the advice issue using following configuration, still don't understand why the handler spec will not work:
#Bean
IntegrationFlow files(SftpOutboundGateway outboundGateway,
...
) {
return IntegrationFlows.from(filesToProcessChannel)
.handle(outboundGateway)
...
.log(LoggingHandler.Level.INFO)
.get();
}
#Bean
public SftpOutboundGateway outboundGateway(SftpRemoteFileTemplate sftpRemoteFileTemplate) {
SftpOutboundGateway gateway = new SftpOutboundGateway(sftpRemoteFileTemplate, AbstractRemoteFileOutboundGateway.Command.PUT.getCommand(), EXPRESSION_PAYLOAD);
ArrayList<Advice> adviceChain = new ArrayList<>();
adviceChain.add(expressionAdvice());
gateway.setAdviceChain(adviceChain);
return gateway;
}
#Bean
public ExpressionEvaluatingRequestHandlerAdvice expressionAdvice() {
ExpressionEvaluatingRequestHandlerAdvice advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setSuccessChannelName("success.input");
advice.setOnSuccessExpressionString("payload + ' was successful'");
advice.setFailureChannelName("failure.input");
advice.setOnFailureExpressionString(
"payload + ' was bad, with reason: ' + #exception.cause.message");
advice.setTrapException(true);
return advice;
}
#Bean
public IntegrationFlow success() {
return f -> f.handle(System.out::println);
}
#Bean
public IntegrationFlow failure() {
return f -> f.handle(System.out::println);
}
Since the movement to the DLQ is performed by the broker, the application has no mechanism to log the situation - it is not even aware that it happened.
You would have to catch the exceptions yourself and publish the message the the DLQ yourself, after some number of attempts (JMSXDeliveryCount header), instead of using the broker policy.
EDIT
Add an Advice to the .handle() step.
.handle(outboundGateway, e -> e.advice(myAdvice))
Where myAdvice implements MethodInterceptor.
In the invoke method, after a failure, you can check the delivery count header and, if it exceeds your threshold, publish the message to the DLQ (e.g. send it to another channel that has a JMS outbound adapter subscribed) and log the error; if the threshold has not been exceeded, simply return the result of the invocation.proceed() (or rethrow the exception).
That way, you control publishing to the DLQ rather than having the broker do it. You can also add more information, such as the exception, to headers.
EDIT2
You need something like this
public class MyAdvice implements MethodInterceptor {
#Autowired
private MessageChannel toJms;
public Object invoke(MethodInvocation invocation) throws Throwable {
try {
return invocation.proceed();
}
catch Exception(e) {
Message<?> message = (Message<?>) invocation.getArguments()[0];
Integer redeliveries = messasge.getHeader("JMXRedeliveryCount", Integer.class);
if (redeliveries != null && redeliveries > 3) {
this.toJms.send(message); // maybe rebuild with additional headers about the error
}
else {
throw e;
}
}
}
}
(it should be close, but I haven't tested it). It assumes your broker populates that header.
I'm always getting an unhandled exception when google+ responses with error json
retrofit2.HttpException: HTTP 404
at retrofit2.RxJavaCallAdapterFactory$SimpleCallAdapter$1.call(RxJavaCallAdapterFactory.java:159)
at retrofit2.RxJavaCallAdapterFactory$SimpleCallAdapter$1.call(RxJavaCallAdapterFactory.java:154)
at rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:54)
at retrofit2.RxJavaCallAdapterFactory$CallOnSubscribe.call(RxJavaCallAdapterFactory.java:109)
at retrofit2.RxJavaCallAdapterFactory$CallOnSubscribe.call(RxJavaCallAdapterFactory.java:88)
at rx.Observable$2.call(Observable.java:162)
at rx.Observable$2.call(Observable.java:154)
at rx.Observable$2.call(Observable.java:162)
at rx.Observable$2.call(Observable.java:154)
....
In that code:
Observable.create(new Observable.OnSubscribe<String>() {
#Override
public void call(Subscriber<? super String> strSub) {
// Getting ID
strSub.onNext(AccountUtils.getAccountId(appContext));
strSub.onCompleted();})
.subscribeOn(Schedulers.io())
// Get Google+ Image through Retrofit2
.flatMap(str -> createGPlusUserObservable(str, AccountUtils.ANDROID_API_KEY))
.map(this::setprofileImage) // I don't see Timber.d message inside that method!
.compose(binder)
.observeOn(AndroidSchedulers.mainThread())
.subscribe(subscriber);
In createGPlusUserObservable I use Retrofit 2 to get google+ image
private Observable<GPlusUser> createGPlusUserObservable(String userId, String apiKey) {
//try {
GoogleApiService service = ServiceFactory.getInstance().createJsonRetrofitService(
GoogleApiService.class,
GoogleApiService.SERVICE_ENDPOINT
);
Observable<GPlusUser> result = service.getGPlusUserInfo(userId, apiKey);
Timber.d("Here1!"); // I see that in console!
return result; // It always returns result!
/*} catch (Throwable e) { - it doesn't catch anything!
Timber.d("Here!");
}*/
}
And subscriber is:
new Subscriber<GPlusUser>() {
#Override
public void onCompleted() {
Timber.d("GPlusUserSubscriber ON COMPLETED");
}
#Override
public void onError(Throwable e) {
if (e instanceof HttpException) {
Timber.d("RETROFIT!"); // I see that in console!
}
}
#Override
public void onNext(GPlusUser gPlusUser) {
setupAccountBox();
}
};
UPDATE: setprofileImage method
private GPlusUser setprofileImage(GPlusUser gPlusUser) {
Timber.d("FOUR"); // As I've said, it doesn't appear in console
AccountUtils.setProfileImage(appContext, gPlusUser.image.url);
Timber.d("Setting profile image: %s", gPlusUser.image.url);
return gPlusUser;
}
So the question is - why I'm getting unhandled exception if I handle it in subscriber's onError(Throwable e)
Thanks!
I think it is because error happens in retrofit factory logic, while converting from pure html string to my GPlusUser class.
I've eliminated that annoying exception in console log by working with pure html through Observable<Response<ResponseBody>> response and it's response.isSuccess()
I have completed a "happy-path" (as below).
How I can advise a .transform call to have it invoke an error flow (via errorChannel) w/o interrupting the mainFlow?
Currently the mainFlow terminates on first failure occurrence in second .transform (when payload cannot be deserialized to type). My desired behavior is that I'd like to log and continue processing.
I've read about ExpressionEvaluatingRequestHandlerAdvice. Would I just add a second param to each .transform call like e -> e.advice(myAdviceBean) and declare such a bean with success and error channels? Assuming I'd need to break up my mainFlow to receive success from each transform.
On some commented direction I updated the original code sample. But I'm still having trouble taking this "all the way home".
2015-09-08 11:49:19,664 [pool-3-thread-1] org.springframework.integration.handler.ServiceActivatingHandler DEBUG handler 'ServiceActivator for [org.springframework.integration.dsl.support.BeanNameMessageProcessor#5f3839ad] (org.springframework.integration.handler.ServiceActivatingHandler#0)' produced no reply for request Message: ErrorMessage [payload=org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice$MessageHandlingExpressionEvaluatingAdviceException: Handler Failed; nested exception is org.springframework.integration.transformer.MessageTransformationException: failed to transform message; nested exception is com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "hasDoaCostPriceChanged" (class com.xxx.changehistory.jdbc.data.RatePlanLevelRestrictionLog), not marked as ignorable (18 known properties: "supplierUpdateDate", "fPLOSMaskArrival", "createDate", "endAllowed", "sellStateId", "ratePlanLevel", "ratePlanId", "startAllowed", "stayDate", "doaCostPriceChanged", "hotelId", "logActionTypeId" [truncated]])
at [Source: java.util.zip.GZIPInputStream#242017b8; line: 1, column: 32] (through reference chain: com.xxx.changehistory.jdbc.data.RatePlanLevelRestrictionLog["hasDoaCostPriceChanged"]), headers={id=c054d976-5750-827f-8894-51aba9655c77, timestamp=1441738159660}]
2015-09-08 11:49:19,664 [pool-3-thread-1] org.springframework.integration.channel.DirectChannel DEBUG postSend (sent=true) on channel 'errorChannel', message: ErrorMessage [payload=org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice$MessageHandlingExpressionEvaluatingAdviceException: Handler Failed; nested exception is org.springframework.integration.transformer.MessageTransformationException: failed to transform message; nested exception is com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "hasDoaCostPriceChanged" (class com.xxx.changehistory.jdbc.data.RatePlanLevelRestrictionLog), not marked as ignorable (18 known properties: "supplierUpdateDate", "fPLOSMaskArrival", "createDate", "endAllowed", "sellStateId", "ratePlanLevel", "ratePlanId", "startAllowed", "stayDate", "doaCostPriceChanged", "hotelId", "logActionTypeId" [truncated]])
at [Source: java.util.zip.GZIPInputStream#242017b8; line: 1, column: 32] (through reference chain: com.xxx.changehistory.jdbc.data.RatePlanLevelRestrictionLog["hasDoaCostPriceChanged"]), headers={id=c054d976-5750-827f-8894-51aba9655c77, timestamp=1441738159660}]
2015-09-08 11:49:19,664 [pool-3-thread-1] org.springframework.integration.channel.DirectChannel DEBUG preSend on channel 'mainFlow.channel#3', message: GenericMessage [payload=java.util.zip.GZIPInputStream#242017b8, headers={id=b80106f9-7f4c-1b92-6aca-6e73d3bf8792, timestamp=1441738159664}]
2015-09-08 11:49:19,664 [pool-3-thread-1] org.springframework.integration.aggregator.AggregatingMessageHandler DEBUG org.springframework.integration.aggregator.AggregatingMessageHandler#0 received message: GenericMessage [payload=java.util.zip.GZIPInputStream#242017b8, headers={id=b80106f9-7f4c-1b92-6aca-6e73d3bf8792, timestamp=1441738159664}]
2015-09-08 11:49:19,665 [pool-3-thread-1] org.springframework.integration.channel.DirectChannel DEBUG preSend on channel 'errorChannel', message: ErrorMessage [payload=org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.integration.aggregator.AggregatingMessageHandler#0]; nested exception is java.lang.IllegalStateException: Null correlation not allowed. Maybe the CorrelationStrategy is failing?, headers={id=24e3a1c7-af6b-032c-6a29-b55031fba0d7, timestamp=1441738159665}]
2015-09-08 11:49:19,665 [pool-3-thread-1] org.springframework.integration.handler.ServiceActivatingHandler DEBUG ServiceActivator for [org.springframework.integration.dsl.support.BeanNameMessageProcessor#5f3839ad] (org.springframework.integration.handler.ServiceActivatingHandler#0) received message: ErrorMessage [payload=org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.integration.aggregator.AggregatingMessageHandler#0]; nested exception is java.lang.IllegalStateException: Null correlation not allowed. Maybe the CorrelationStrategy is failing?, headers={id=24e3a1c7-af6b-032c-6a29-b55031fba0d7, timestamp=1441738159665}]
2015-09-08 11:49:19,665 [pool-3-thread-1] com.xxx.DataMigrationModule$ErrorService ERROR org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.integration.aggregator.AggregatingMessageHandler#0]; nested exception is java.lang.IllegalStateException: Null correlation not allowed. Maybe the CorrelationStrategy is failing?
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:84)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:101)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:97)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:287)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:245)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:95)
at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutput(AbstractMessageProducingHandler.java:231)
at org.springframework.integration.handler.AbstractMessageProducingHandler.produceOutput(AbstractMessageProducingHandler.java:154)
at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutputs(AbstractMessageProducingHandler.java:102)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:105)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:101)
at org.springframework.integration.dispatcher.UnicastingDispatcher.access$000(UnicastingDispatcher.java:48)
at org.springframework.integration.dispatcher.UnicastingDispatcher$1.run(UnicastingDispatcher.java:92)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Null correlation not allowed. Maybe the CorrelationStrategy is failing?
at org.springframework.util.Assert.state(Assert.java:385)
at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.handleMessageInternal(AbstractCorrelatingMessageHandler.java:369)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
... 22 more
UPDATED (09-08-2015)
code sample
#Bean
public IntegrationFlow mainFlow() {
// #formatter:off
return IntegrationFlows
.from(
amazonS3InboundSynchronizationMessageSource(),
e -> e.poller(p -> p.trigger(this::nextExecutionTime))
)
.transform(unzipTransformer())
.split(f -> new FileSplitter())
.channel(MessageChannels.executor(Executors.newCachedThreadPool()))
.transform(Transformers.fromJson(persistentType()), , e -> e.advice(handlingAdvice()))
// #see http://docs.spring.io/spring-integration/reference/html/messaging-routing-chapter.html#agg-and-group-to
.aggregate(a ->
a.releaseStrategy(g -> g.size() == persistenceBatchSize)
.expireGroupsUponCompletion(true)
.sendPartialResultOnExpiry(true)
.groupTimeoutExpression("size() ge 2 ? 10000 : -1")
, null
)
.handle(jdbcRepositoryHandler())
// TODO add advised PollableChannel to deal with possible persistence issue and retry with partial batch
.get();
// #formatter:on
}
#Bean
public ErrorService errorService() {
return new ErrorService();
}
#Bean
public MessageChannel customErrorChannel() {
return MessageChannels.direct().get();
}
#Bean
public IntegrationFlow customErrorFlow() {
// #formatter:off
return IntegrationFlows
.from(customErrorChannel())
.handle("errorService", "handleError")
.get();
// #formatter:on
}
#Bean
ExpressionEvaluatingRequestHandlerAdvice handlingAdvice() {
ExpressionEvaluatingRequestHandlerAdvice advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setOnFailureExpression("payload");
advice.setFailureChannel(customErrorChannel());
advice.setReturnFailureExpressionResult(true);
advice.setTrapException(true);
return advice;
}
protected class ErrorService implements ErrorHandler {
private final Logger log = LoggerFactory.getLogger(getClass());
#Override
public void handleError(Throwable t) {
stopEndpoints(t);
}
private void stopEndpoints(Throwable t) {
log.error(ExceptionUtils.getStackTrace(t));
}
}
Turns out I had things wrong in a few places, like:
I had to autowire a Jackson2 ObjectMapper (that I get from Sprint Boot auto-config) and construct an instance of JsonObjectMapper to be added as second arg in Transformers.fromJson; made for more lenient unmarshalling to persistent type (stops UnrecognizedPropertyException); and thus waived need for ExpressionEvaluatingRequestHandlerAdvice
Choosing the proper variant of .split method in IntegrationFlowDefinition in order to employ the FileSplitter, otherwise you don't get this splitter rather a DefaultMessageSplitter which pre-maturely terminates flow after first record read from InputStream
Moved transform, aggregate, handle to a its own pubsub channel employing an async task executor
Still not 100% of what I need, but it's much further along.
See what I ended up w/ below...
#Configuration
#EnableIntegration
#IntegrationComponentScan
public class DataMigrationModule {
private final Logger log = LoggerFactory.getLogger(getClass());
#Value("${cloud.aws.credentials.accessKey}")
private String accessKey;
#Value("${cloud.aws.credentials.secretKey}")
private String secretKey;
#Value("${cloud.aws.s3.bucket}")
private String bucket;
#Value("${cloud.aws.s3.max-objects-per-batch:1024}")
private int maxObjectsPerBatch;
#Value("${cloud.aws.s3.accept-subfolders:false}")
private String acceptSubFolders;
#Value("${cloud.aws.s3.remote-directory}")
private String remoteDirectory;
#Value("${cloud.aws.s3.local-directory-ref:java.io.tmpdir}")
private String localDirectoryRef;
#Value("${cloud.aws.s3.local-subdirectory:target/s3-dump}")
private String localSubdirectory;
#Value("${cloud.aws.s3.filename-wildcard:}")
private String fileNameWildcard;
#Value("${app.persistent-type:}")
private String persistentType;
#Value("${app.repository-type:}")
private String repositoryType;
#Value("${app.persistence-batch-size:2500}")
private int persistenceBatchSize;
#Value("${app.persistence-batch-release-timeout-in-milliseconds:5000}")
private int persistenceBatchReleaseTimeoutMillis;
#Autowired
private ListableBeanFactory beanFactory;
#Autowired
private ObjectMapper objectMapper;
private final AtomicBoolean invoked = new AtomicBoolean();
private Class<?> repositoryType() {
try {
return Class.forName(repositoryType);
} catch (ClassNotFoundException cnfe) {
log.error("Unknown repository implementation!", cnfe);
System.exit(0);
}
return null;
}
private Class<?> persistentType() {
try {
return Class.forName(persistentType);
} catch (ClassNotFoundException cnfe) {
log.error("Unsupported type!", cnfe);
System.exit(0);
}
return null;
}
public Date nextExecutionTime(TriggerContext triggerContext) {
return this.invoked.getAndSet(true) ? null : new Date();
}
#Bean
public FileToInputStreamTransformer unzipTransformer() {
FileToInputStreamTransformer transformer = new FileToInputStreamTransformer();
transformer.setDeleteFiles(true);
return transformer;
}
#Bean
public MessageSource<?> amazonS3InboundSynchronizationMessageSource() {
AWSCredentials credentials = new BasicAWSCredentials(this.accessKey, this.secretKey);
AmazonS3InboundSynchronizationMessageSource messageSource = new AmazonS3InboundSynchronizationMessageSource();
messageSource.setCredentials(credentials);
messageSource.setBucket(bucket);
messageSource.setMaxObjectsPerBatch(maxObjectsPerBatch);
messageSource.setAcceptSubFolders(Boolean.valueOf(acceptSubFolders));
messageSource.setRemoteDirectory(remoteDirectory);
if (!fileNameWildcard.isEmpty()) {
messageSource.setFileNameWildcard(fileNameWildcard);
}
String directory = System.getProperty(localDirectoryRef);
if (!localSubdirectory.startsWith("/")) {
localSubdirectory = "/" + localSubdirectory;
}
if (!localSubdirectory.endsWith("/")) {
localSubdirectory = localSubdirectory + "/";
}
directory = directory + localSubdirectory;
FileUtils.mkdir(directory);
messageSource.setDirectory(new LiteralExpression(directory));
return messageSource;
}
#Bean
public IntegrationFlow mainFlow() {
// #formatter:off
return IntegrationFlows
.from(
amazonS3InboundSynchronizationMessageSource(),
e -> e.poller(p -> p.trigger(this::nextExecutionTime))
)
.transform(unzipTransformer())
.split(new FileSplitter(), null)
.publishSubscribeChannel(new SimpleAsyncTaskExecutor(), p -> p.subscribe(persistenceSubFlow()))
.get();
// #formatter:on
}
#Bean
public IntegrationFlow persistenceSubFlow() {
JsonObjectMapper<?, ?> jsonObjectMapper = new Jackson2JsonObjectMapper(objectMapper);
ReleaseStrategy releaseStrategy = new TimeoutCountSequenceSizeReleaseStrategy(persistenceBatchSize,
persistenceBatchReleaseTimeoutMillis);
// #formatter:off
return f -> f
.transform(Transformers.fromJson(persistentType(), jsonObjectMapper))
// #see http://docs.spring.io/spring-integration/reference/html/messaging-routing-chapter.html#agg-and-group-to
.aggregate(
a -> a
.releaseStrategy(releaseStrategy)
.correlationStrategy(m -> m.getHeaders().get("id"))
.expireGroupsUponCompletion(true)
.sendPartialResultOnExpiry(true)
, null
)
.handle(jdbcRepositoryHandler());
// #formatter:on
}
#Bean
public JdbcRepositoryHandler jdbcRepositoryHandler() {
return new JdbcRepositoryHandler(repositoryType(), beanFactory);
}
protected class JdbcRepositoryHandler extends AbstractMessageHandler {
#SuppressWarnings("rawtypes")
private Insertable repository;
public JdbcRepositoryHandler(Class<?> repositoryClass, ListableBeanFactory beanFactory) {
repository = (Insertable<?>) beanFactory.getBean(repositoryClass);
}
#Override
protected void handleMessageInternal(Message<?> message) {
repository.insert((List<?>) message.getPayload());
}
}
protected class FileToInputStreamTransformer extends AbstractFilePayloadTransformer<InputStream> {
#Override
protected InputStream transformFile(File payload) throws Exception {
return new GZIPInputStream(new FileInputStream(payload));
}
}
}
Yes, you are correct. To advice the handle() method of Transformer's MessageHandler you should use exactly that e.advice method of the second parameter of .transform() EIP-method. And yes: you should define ExpressionEvaluatingRequestHandlerAdvice bean for your purpose.
You can reuse that Advice bean for different goals to handle successes and failures the same manner.
UPDATE
Although it isn't clear to me how you'd like to continue the flow with the wrong message, but you you can use onFailureExpression and returnFailureExpressionResult=true of the ExpressionEvaluatingRequestHandlerAdvice to return something after the unzipErrorChannel().
BTW the failureChannel logic doesn't work without onFailureExpression:
if (this.onFailureExpression != null) {
Object evalResult = this.evaluateFailureExpression(message, actualException);
if (this.returnFailureExpressionResult) {
return evalResult;
}
}