We Keep Coding

Exchange.GROUPED_EXCHANGE not available after aggregation in Apache Camel 3.7.2

After upgrading from Apache Camel 2.x to 3.7.2 in our Java 11 project we experience some crudities in a route configuration class, extending the RouteBuilder (Documentation) class. With Camel 2 I used the SimpleExpression ${property[" + Exchange.GROUPED_EXCHANGE + "]}, which has now been renamed in Camel 3.x to exchangeProperty (Migration Guide). So far, so good.
#Override
public void configure() throws Exception {
final SimpleExpression documentAppendix =
new SimpleExpression("${body.appendix}");
final SimpleExpression groupedExchanges =
new SimpleExpression("${exchangeProperty[" + Exchange.GROUPED_EXCHANGE + "]}");
from("direct://input")
.choice()
.when(documentAppendix)
.split(documentAppendix, new GroupedExchangeAggregationStrategy())
.to("direct://input.splitted")
.end()
.setBody(groupedExchanges)
.endChoice()
.otherwise()
.setBody(constant(Lists.newArrayList()))
.end();
// ... omitted
}
After running the tests, everything failed because the body did not contain the expected number of appendices. At first, we thought there might be a problem with exchangeProperty but after some time spent with the debugger we've found out the following trail where the property gets "lost":
GroupedExchangeAggregationStrategy
|
v
AbstractListAggregationStrategy
(This class sets the "CamelGroupedExchange" property!)
|
v
AggregateProcessor
doAggregation(...)
ExchangeHelper.preparteAggregation(...)
The expected return after aggregation should be a list of objects, accessible via the CamelGroupedExchange or ${exchangeProperty[" + Exchange.GROUPED_EXCHANGE + "]} but it gets overridden with the newExchange in ExchangeHelper.preparteAggregation.
As we did not find more evidence, could someone explain this weird behaviour after upgrading Camel to 3.7.2? Maybe there have been breaking changes in ExchangeHelper and the available ExchangePattern/MEP Pattern (CAMEL-13286) but we were not able to break down the issue.
Thanks for your help guys!

We figured out that the AbstractListAggregationStrategy in Camel 3.7.2 now sets the property as In body on completion:
public abstract class AbstractListAggregationStrategy<V> implements AggregationStrategy {
public AbstractListAggregationStrategy() {
}
public abstract V getValue(Exchange exchange);
public boolean isStoreAsBodyOnCompletion() {
return true;
}
public void onCompletion(Exchange exchange) {
if (exchange != null && this.isStoreAsBodyOnCompletion()) {
List<V> list = (List)exchange.removeProperty("CamelGroupedExchange");
if (list != null) {
exchange.getIn().setBody(list);
}
}
}
// omitted
}
With this change our .setBody(groupedExchanges) is redundant and we are able to access the list of exchanges via getIn().

Override #override with AOP

I'm giving AOP a try for the first time. I want to do AOP on the #Override notation to write logs. It seems to be almost working but:
I have to import my own Override class in every class. Is this normal? I thought that it would magically go through my #Override decorator first and then through the Java one.
I have this very weird behavior where depending on the endpoint that I call first, it keeps working after or it only works for this one endpoint. Say, I have /a and /b, if I call /b first, it shows my logs, then I'll call /a and it won't show anything, if after that I call /b it will show logs. However, if I call /a first, it works, then I call /b and it works, and it just keeps working for all of them. It makes no sense to me.
This is my OverrideInterceptor:
#Slf4j
public class OverrideInterceptor implements MethodInterceptor<Object, Object> {
#Override
public Object intercept(MethodInvocationContext<Object, Object> context) {
String prettyMethod = context.getDeclaringType().getSimpleName() + "." + context.getName();
log.debug("{} with params {}", prettyMethod, context.getParameterValueMap());
long start = System.nanoTime();
Object result = context.proceed();
long end = System.nanoTime() - start;
log.debug("Execution of " + prettyMethod + " took: " + (end/1000) + "ms.");
return result;
}
}
And my annotation:
#Documented
#Retention(RUNTIME)
#Target({ElementType.TYPE, ElementType.METHOD})
#Around
#Type(OverrideInterceptor.class)
public #interface Override {
}
Both classes are in the package: package com.time.infrastructure.config;
And I'm using this #Override annotation in packages under com.time.infrastructure.db, com.time.infrastructure.rest, com.time.application.repository, etc.

For point 1: The built-in #Override annotation is a normal #interface - it resides in the java.lang package. What you've done here is that you've created a custom annotation with the name Override in the package com.time.infrastructure.config, which has nothing to do with java.lang.Override. So in that sense, it's "normal", but it's probably not doing what you think. You can't subtype annotations in Java, unfortunately.

JEE: how to pass parameter to an interceptor

In my JEE application, running on glassfish 3, I have the following situation:
MyFacade class
#Interceptors(SomeInterceptor.class)
public void delete(Flag somethingForTheInterceptor, String idToDelete) {
.......
}
#Interceptors(SomeInterceptor.class)
public void update(Flag somethingForTheInterceptor, MyStuff newStuff) {
.......
}
The variable somethingForTheInterceptor is not used in these methods, it is only used in the interceptor:
SomeInterceptor class
#AroundInvoke
public Object userMayAccessOutlet(InvocationContext ctx) throws Exception {
Flag flag = extractParameterOfType(Arrays.asList(ctx.getParameters()), Flag.class);
// some checks on the flag
}
Somehow it doesn't feel good to have a parameter that is not used in the method. Is there another way to "send" somethingForTheInterceptor to the interceptor?
UPDATE: The callers of delete() and update() have different ways of calculating the somethingForTheInterceptor variable. And this is not a constant. The information needed to calculate it is in the REST call. But the 2 REST methods have different input objects so it is not enough to inject the http request.
These are the callers:
MyResource class
#DELETE
#Path("/delete/{" + ID + "}")
public Response delete(#PathParam(ID) final String id) {
Flag flag = calculateFlagForInterceptor(id);
facade.delete(flag, id);
}
#POST
#Path("/update")
#Consumes(MediaType.APPLICATION_JSON + RestResourceConstants.CHARSET_UTF_8)
public Response update(final WebInputDTO updateDetails) throws ILeanException {
Flag flag = calculateFlagForInterceptor(updateDetails);
facade.update(flag, convertToMyStuff(updateDetails));
}
I was thinking - is it possible for the methods in the Resource to set the flag in some kind of Context, that can be later injected in the Interceptor?

In Java EE, Interceptors allow to add pre and post processings to a method.
So, the context of the Interceptor execution is the context of the method.
I was thinking - is it possible for the methods in the Resource to set
the flag in some kind of Context, that can be later injected in the
Interceptor?
Staless Service should be privileged when you may. So, you should avoid storing data on the server (ThreadLocal, Session, etc..).
The information needed to calculate it is
in the REST call.
Why ?
A Rest controller has no vocation to do computations and logic.
To solve your problem, are you sure you could not move the flag computation in your interceptor ?
By enhancing the interceptor responsibilities, you would have not need anly longer to transit the flag :
#AroundInvoke
public Object userMayAccessOutlet(InvocationContext ctx) throws Exception {
Flag flag = calculFlag(Arrays.asList(ctx.getParameters()));
// some checks on the flag
}

Camel: failure to add routes dynamically

I'm using Apache-Camel 2.15.2.
I am trying to add routes to a CamelContext dynamically, but I came across a problem that baffles me.
As far as I can tell, I do add the routes to the correct CamelContext, and it seems like their configure() is called without throwing exceptions. However when I try to execute the main route, I get a run time Exception telling me that the route I added dynamically does not exist.
Here is a simplified version of my code:
public class MainRouteBuilder extends RouteBuilder
{
public static CamelContext camelContext;
public static boolean returnLabel = true;
public static RouteBuilder nestedRouteBuilder;
#Override
public void configure() throws Exception
{
System.out.println("Building main route!");
System.out.println("Context: " + getContext());
camelContext = getContext();
from("direct:mainRoute")
//3. I do not actually want to instantiate RouteContainer like this each time I call this route.
//I would simply want to reuse a reference to an instance I created outside of configure()...
.to(new RouteContainer().getMyRoute(2))
;
returnLabel = false;
//configure direct:myRoute.2
includeRoutes(nestedRouteBuilder);
}
}
public class RouteContainer extends RouteBuilder
{
public Route route;
RouteContainer() {
super(MainRouteBuilder.camelContext);
}
String getMyRoute(final int n) {
if (MainRouteBuilder.returnLabel && route == null) {
route = new Route() {
#Override
public void configure()
{
System.out.println("Building nested route!");
System.out.println("Context: " + getContext());
from("direct:myRoute." + n)
.transform()
.simple("number: " + n)
.to("stream:out")
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
Response response = new Response();
response.setStatus(Status.SUCCESS);
exchange.getOut().setBody(response);
}
});
}
};
}
//1. works:
MainRouteBuilder.nestedRouteBuilder = this;
//2. does not work:
// RouteContainer routeContainer = new RouteContainer();
// routeContainer.route = this.route;
// MainRouteBuilder.nestedRouteBuilder = routeContainer;
return "direct:myRoute." + n;
}
#Override
public void configure() throws Exception {
if (route != null) {
route.configure();
}
}
public abstract static class Route {
abstract public void configure();
}
}
Requests that are sent to direct:mainRoute work.
During Camel startup I see in the console:
Building main route!
Context: SpringCamelContext(camel-1) with spring id org.springframework.web.context.WebApplicationContext:/sample-route
Building nested route!
Context: SpringCamelContext(camel-1) with spring id org.springframework.web.context.WebApplicationContext:/sample-route
and when I send a request to direct:mainRoute, the output is:
{"status":"SUCCESS"}
HOWEVER, if I comment out (1) above, and uncomment (2), Camel starts up with the same output to the console, but when I send a request to direct:mainRoute, the execution of the route fails with the exception:
org.apache.camel.component.direct.DirectConsumerNotAvailableException: No consumers available on endpoint: Endpoint[direct://myRoute.2].
To Clarify: my problem is because I would actually like NOT to instantiate RouteContainer each time I call its route, as I do in (3). This is why I instantiate them at point (2) and plug the Route instance into it...
So I would like MainRouteBuilder to look like this:
public class MainRouteBuilder extends RouteBuilder
{
public static CamelContext camelContext;
public static boolean returnLabel = true;
public static RouteBuilder nestedRouteBuilder;
RouteContainer routeContainer = new RouteContainer();
#Override
public void configure() throws Exception
{
System.out.println("Building main route!");
System.out.println("Context: " + getContext());
camelContext = getContext();
from("direct:mainRoute")
.to(routeContainer.getMyRoute(2))
//I may want to call it again like this:
//.to(routeContainer.getMyRoute(3))
;
returnLabel = false;
//configure direct:myRoute.2
includeRoutes(nestedRouteBuilder);
}
}
My assumption is that maybe the nested route direct:myRoute.2 is created in the wrong CamelContext, but the console output tells me it is not so.
Any idea what I am doing wrong here?

Route configuration != route execution
It seems that you are confusing route configuration with route execution. We've all been there ;-)
When you configure the RouteBuilder in MainRouteBuilder#configure(), the method is only is only executed once when your Camel app bootstraps, in order to set up the routing logic. The DSL creates the plumbing for the route (Processors, Interceptors, etc.) and that's what the route runtime will be.
Point to bring home: The DSL is not executed over and over again with every Exchange.
In other words, Camel does not do what you point out in (3). It doesn't execute new RouteContainer().getMyRoute(2) for every single Exchange. Think about it: the bytecode for configure() is only executed when configuring Camel, and the bytecode instantiates an object of class RouteContainer and it invokes the getMyRoute with argument 2. The resulting object is fed into the SendProcessor that the to() DSL generates.
Analysis of your code
Now, with regards to why your code doesn't yield the result you expect.
You have a problem with the state-keeping of RouteContainer. Every time you call getMyRoute you overwrite the instance variable route. So it's impossible for your current code to call getMyRoute several times (with different ns) and then call includeRoutes just once at the end, because only the most recently generated route will be added.
I also don't like masking the Camel Route class with a class of your own, just to act as a placeholder, but that brings up a different discussion you didn't ask for.
Simpler solution
Instead of your RouteContainer class, here's a RouteGenerator class that creates routes and returns the direct: endpoint to the caller. It keeps track of all the routes in an internal Set.
public class RouteGenerator {
private Set<RouteBuilder> routeBuilders = new HashSet<>();
public String generateRoute(final int n) {
routeBuilders.add(new RouteBuilder() {
#Override public void configure() throws Exception {
System.out.println("Building nested route!");
System.out.println("Context: " + getContext());
from("direct:myRoute." + n)
.transform() .simple("number: " + n)
.to("stream:out")
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
Response response = new Response();
response.setStatus(Status.SUCCESS);
exchange.getOut().setBody(response);
}
});
}
});
return "direct:myRoute." + n;
}
public Set<RouteBuilder> getRouteBuilders() {
return routeBuilders;
}
}
And here is your MainRouteBuilder, which instantiates the RouteGenerator only once, and can generate as many routes as you wish.
Once you finish configuring your routes, you just iterate over the accumulated RouteBuilders and include them:
public class MainRouteBuilder extends RouteBuilder {
public static CamelContext camelContext;
public static RouteGenerator routeGenerator = new RouteGenerator();
#Override
public void configure() throws Exception {
System.out.println("Building main route!");
System.out.println("Context: " + getContext());
camelContext = getContext();
from("direct:mainRoute")
.to(routeGenerator.generateRoute(2));
for (RouteBuilder routeBuilder : routeGenerator.getRouteBuilders()) {
includeRoutes(routeBuilder);
}
}
}
EDIT: Why doesn't your option (2) work?
After debugging for some time, I realised why you're seeing this effect.
Extracted from the Java Tutorial:
As with instance methods and variables, an inner class is associated
with an instance of its enclosing class and has direct access to that
object's methods and fields.
In (2), you create an instance of Route within the scope of the initial RouteContainer object, acting as the outer object. The Route object retains the outer RouteContainer as its outer object. The from() and subsequent methods are therefore being invoked on that initial RouteContainer (RouteBuilder) object, not on the new RouteContainer you create later, which is the one you provide to the upper RouteBuilder (which is associated to the CamelContext).
That's why your direct:myRoute.2 is not being added to the Camel Context, because it's being created in a different route builder.
Also note that the console output of (2):
Building main route!
Context: CamelContext(camel-1)
Building nested route!
Context: CamelContext(camel-2)
Added!
The second route is being added to a different context camel-2. This new context is created by Camel lazily when it adds the route to the old RouteBuilder, which hasn't been associated to any Camel Context yet.
Note that the the Camel Context of the initial RouteContainer (created in the instance variable initialization) is null, because you assign the MainRouteBuilder.camelContext property later on.
You can see how two different route builders are being used by adding the following println statements:
Inside Route#configure:
System.out.println("RouteContainer to which route is added: " + RouteContainer.this.hashCode());
Inside MainRouteBuilder#configure, just before includeRoutes:
System.out.println("RouteContainer loaded into Camel: " + nestedRouteBuilder.hashCode());
With (1), the hashcode is the same. With (2), the hashcode is different, clearly showing that there are two different RouteBuilders in play (one which contains the route, and the one that's loaded into the Context, which does not include the route).
Source: I'm a Apache Camel PMC member and committer.

Apache Camel listener

I have created a router class and marked as a #Bean in #Configuration. One thing I am not very much sure is how frequently camel would be making a database call to get the select result? As soon as I have a new entry in the database, camel retrieve and process it.
public class SQLRouteBuilderForNewUserProcessing extends RouteBuilder {
#Override
public void configure() throws Exception {
//files refer camel files endpoint
//noop will not move or delete the files
from("sql:" +
"select id from users where status=" + Status.NEW.ordinal() +
"?" +
"consumer.onConsume=update users set status = " + Status.PROCESSING.ordinal()
" where id = :#id")
.bean(UserDataTranslator.class, "transformToUserData")
.to("log:uk.co.infogen.users?level=INFO");
}
}

by default, the sql consumer pool the database every 500ms. You can configure this with consumer.delay
from("sql:select ... &consumer.delay=5000")
.to(...)
see the documentation of the sql component

consumer.delay
long
500
Camel 2.11: SQL consumer only: Delay in milliseconds between each poll.
from http://camel.apache.org/sql-component.html