We Keep Coding

Apache Camel: get consumer endpoint information in bean method called by the Camel route

I have multiple databases, all containing the same table Data. I want to read from them, input all Data elements into the MyBean method #Handler public Data updateData(Data data) and write back the output of the method.
from("jpa://Data?persistenceUnit=persUnit1").to("direct:collector");
from("jpa://Data?persistenceUnit=persUnit2").to("direct:collector");
from("jpa://Data?persistenceUnit=persUnit3").to("direct:collector");
...
from("direct:collector").bean(new MyBean()).to("jpa://Data?persistenceUnit=destinationUnit");
However I need the information from which source the Data element came (e.g. the name of the persistence unit) within the bean for validation. What's the best way to do so?

You could set a header
from("jpa://Data?persistenceUnit=persUnit1")
.setHeader("dataSource", constant("dataSource1"))
.to("direct:collector");
from("jpa://Data?persistenceUnit=persUnit2")
.setHeader("dataSource", constant("dataSource2"))
.to("direct:collector");

The exchange object provides the necessary information to about the "from" endpoint (of current exchange):
https://www.javadoc.io/doc/org.apache.camel/camel-api/latest/org/apache/camel/Exchange.html#getFromRouteId()
It's quite usual to put id on routes (but also on endpoints)
from("jpa://Data?persistenceUnit=persUnit1")
.routeId("ComingFromRouteA")
.to("direct:collector");
This way, you can know where you come from, using:
exchange.getFromRouteId()

What does the spring.cloud.stream.source really do?

I am trying to understand how the new functional model of Spring Cloud Streams works and how the configuration actually works under the hood.
One of the properties I am unable to figure out is spring.cloud.stream.source.
What does this property actually signify ?
I could not understand the documentation :
Note that preceding example does not have any source functions defined
(e.g., Supplier bean) leaving the framework with no trigger to create
source bindings, which would be typical for cases where configuration
contains function beans. So to trigger the creation of source binding
we use spring.cloud.stream.source property where you can declare the
name of your sources. The provided name will be used as a trigger to
create a source binding.
What if I did not need a Supplier ?
What exactly is a source binding and why is it important ?
What if I only wanted to produce to a messaging topic ? Would I still need this property ?
I also could not understand how it is used in the sample here.

Spring cloud stream looks for java.util Function<?, ?, Consumer<?>, Supplier<?> beans and creates bindings for them.
In the supplier case, the framework polls the supplier (each second by default) and sends the resulting data.
For example
#Bean
public Supplier<String> output() {
return () -> "foo";
}
spring.cloud.stream.bindings.output-out-0.destination=bar
will send foo to destination bar each second.
But, what if you don't need a polled source, but you want to configure a binding to which you can send arbitrary data. Enter spring.cloud.stream.source.
spring.cloud.stream.source=output
spring.cloud.stream.bindings.output-out-0.destination=bar
allows you to send arbitrary data to the stream bridge
bridge.send("output-out-0", "test");
In other words, it allows you to configure one or more ouput bindings that you can use in the StreamBridge; otherwise, when you send to the bridge, the binding is created dynamically.

NiFi custom processor expression language

I'm trying to make a custom processor in Apache NiFi that can add an attribute/string to the JSON object in the flowfile content. At the moment it works when I just use a string but it's not working when I use NiFi's expression language although I have it supported in my code.
The expression language is 100% correct as it works in another processor and I've also tried different attributes to make sure it's not the attribute.
The property:
public static final PropertyDescriptor ADD_ATTRIBUTE = new PropertyDescriptor
.Builder().name("Add Attribute")
.description("Example Property")
.required(true)
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.expressionLanguageSupported(true)
.build();
Later in my code when I want to get the value and put in the JSON object I use:
jsonObject.put("hostname", context.getProperty(ADD_ATTRIBUTE).evaluateAttributeExpressions().getValue());
I also made a Unit Test and it works when I assign a text value to the testrunner.setProperty. However I don't know how I can assign an attribute to the testrunner or how I can use expression language in my test.
Thanks in advance for any suggestions or a solution!

I'll put my answer from Hortonworks Community Connection here too FWIW:
If the expression refers to attributes on a flow file, you will need to pass a reference to the flow file into evaluateAttributeExpressions:
FlowFile flowFile = session.get();
jsonObject.put("hostname", context.getProperty(ADD_ATTRIBUTE).evaluateAttributeExpressions(flowFile).getValue());
If the property contains an attribute name (rather than an Expression containing an attribute name) and you want the value from the flow file:
jsonObject.put("hostname", flowFile.getAttribute(context.getProperty(ADD_ATTRIBUTE).getValue()));
If the attribute value itself contains Expression Language and you want to evaluate it, take a look at the following class:
org.apache.nifi.attribute.expression.language.Query

Regarding testing...
Assuming you are evaluating the expression language against an incoming FlowFile (evaluateAttributeExpressions(flowFile)) then you can do the following:
runner.setProperty(ADD_ATTRIBUTE, "${my.attribute}");
Then create an attribute Map that has my.attribute in it:
final Map<String,String> attributes = new HashMap<>();
attributes.put("my.attribute", myAttribute);
Then enqueue some content with the attributes:
runner.enqueue(fileIn, attributes);
runner.run();
An example from the code base:
https://github.com/apache/nifi/blob/1e56de9521e4bc0752b419ffc7d62e096db1c389/nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/test/java/org/apache/nifi/processors/solr/TestPutSolrContentStream.java#L243

Camel Rest DSL response encoding

I am currently working on a REST based java application using the new Camel REST DSL as the foundation.
It mostly works except that I noticed when calling the URLs through a REST client (instead of say a browser) the JSON response is "garbled" and comes through with what I assume is the wrong encoding
MyRouteBuilder.java
#Component
public class MyRouteBuilder extends RouteBuilder{
#Autowired
LocalEnvironmentBean environmentBean;
#Override
public void configure() throws Exception {
restConfiguration().component("jetty").host("0.0.0.0").port(80)
.bindingMode(RestBindingMode.auto);
rest("/testApp")
.get("/data").route()
.to("bean:daoService?method=getData")
.setProperty("viewClass", constant(CustomeJsonViews.class))
.marshal("customDataFormat").endRest()
.get("/allData").route()
.to("bean:daoService?method=getDatas")
.setProperty("viewClass", constant(CustomeJsonViews.class))
.marshal("customDataFormat").endRest();
}
}
CustomeDataFormat.java
public class CustomDataFormat implements DataFormat{
private ObjectMapper jacksonMapper;
public CustomDataFormat(){
jacksonMapper = new ObjectMapper();
}
#Override
public void marshal(Exchange exchange, Object obj, OutputStream stream) throws Exception {
Class view = (Class) exchange.getProperty("viewClass");
if (view != null)
{
ObjectWriter w = jacksonMapper.writerWithView(view);
w.writeValue(stream, obj);
}
else
stream.write(jacksonMapper.writeValueAsBytes(obj));
}
#Override
public Object unmarshal(Exchange exchange, InputStream stream) throws Exception {
return null;
}
}
A full working version can be found here:
https://github.com/zwhitten/camel-rest-test
When going to the URL, {host}/testApp/data, in Chrome for example the response comes through as:
{
data: "Sherlock",
value: "Holmes",
count: 10
}
However using the Postman browser plugin as the client returns:
"W3siZGF0YSI6ImRhdGE6OjAiLCJ2YWx1ZSI6InZhbHVlOjowIiwiY291bnQiOjB9LHsiZGF0YSI6ImRhdGE6OjEiLCJ2YWx1ZSI6InZhbHVlOjoxIiwiY291bnQiOjF9LHsiZGF0YSI6ImRhdGE6OjIiLCJ2YWx1ZSI6InZhbHVlOjoyIiwiY291bnQiOjJ9LHsiZGF0YSI6ImRhdGE6OjMiLCJ2YWx1ZSI6InZhbHVlOjozIiwiY291bnQiOjN9LHsiZGF0YSI6ImRhdGE6OjQiLCJ2YWx1ZSI6InZhbHVlOjo0IiwiY291bnQiOjR9LHsiZGF0YSI6ImRhdGE6OjUiLCJ2YWx1ZSI6InZhbHVlOjo1IiwiY291bnQiOjV9LHsiZGF0YSI6ImRhdGE6OjYiLCJ2YWx1ZSI6InZhbHVlOjo2IiwiY291bnQiOjZ9LHsiZGF0YSI6ImRhdGE6OjciLCJ2YWx1ZSI6InZhbHVlOjo3IiwiY291bnQiOjd9LHsiZGF0YSI6ImRhdGE6OjgiLCJ2YWx1ZSI6InZhbHVlOjo4IiwiY291bnQiOjh9LHsiZGF0YSI6ImRhdGE6OjkiLCJ2YWx1ZSI6InZhbHVlOjo5IiwiY291bnQiOjl9XQ=="
The problem seems to be with the REST bind mode being "auto" and using a custom marshaller.
If I set the binding mode to "json" then both the browser and client responses get garbled.
If I set the binding mode to "json" and bypass the custom marshallers everything works correctly.
Is there a way to configure the route to use a custom marshaller and encode the responses correctly regardless of the client?

I think the solution is to use the default binding option(off) since you are using custom marshallers.

You have two ways to achieve it:
Turn off the RestBindingMode, because otherwise the RestBindingMarshalOnCompletion in RestBindingProcessor will be registered and manually (un)marshal.
Register your own DataFormat and use it within the RestBinding automatically. You configure the REST configuration via jsonDataFormat to set the custom data format.
Map<String, DataFormatDefinition> dataFormats = getContext().getDataFormats();
if (dataFormats == null) {
dataFormats = new HashMap<>();
}
dataFormats.put("yourFormat", new DataFormatDefinition(new CustomDataFormat()));
restConfiguration()....jsonDataFormat("yourFormat")

You can also create your own dataformat like so:
in your restconfiguration it will look sthg like this (see json-custom)
builder.restConfiguration().component("jetty")
.host(host(propertiesResolver))
.port(port(propertiesResolver))
.bindingMode(RestBindingMode.json)
.jsonDataFormat("json-custom")
;
You must create a file "json-custom"
that's the name of the file and that file should contain the class name that implements your own way to marshal and unmarshal...
it must be located in your jar : META-INF\services\org\apache\camel\dataformat
so the content of the file should be:
class=packageofmyclass.MyOwnDataformatter

The response you were receiving is JSON, but it had been encoded to base64. Taking the String from your post, I was able to decode it as:
[{"data":"data::0","value":"value::0","count":0},{"data":"data::1","value":"value::1","count":1},{"data":"data::2","value":"value::2","count":2},{"data":"data::3","value":"value::3","count":3},{"data":"data::4","value":"value::4","count":4},{"data":"data::5","value":"value::5","count":5},{"data":"data::6","value":"value::6","count":6},{"data":"data::7","value":"value::7","count":7},{"data":"data::8","value":"value::8","count":8},{"data":"data::9","value":"value::9","count":9}]
The answers above stop the response body being encoded to base64. The documentation from Apache Camel on bindingMode is illusive as to why it behaves that way when combined with explicit marshalling. Removing the explicit marshalling will return a JSON body, but you may also notice that it contains the any class names in the body. The documentation suggests that bindingMode is more for the transportation of classes and that you specifiy a type(Pojo.class) and optionally outType(Pojo.class) of your requests/responses. See http://camel.apache.org/rest-dsl.html (section Binding to POJOs Using) for more details.
Base64 is the safest way of transferring JSON across networks to ensure it is received exactly as the server sent it, according to some posts I've read. It is then the responsibility of the client to then decode the response.
The answers above do solve the problem. However, I'm not entirely convinced that mixing the data format in the service routes is such as good thing and should ideally be at a higher level of abstraction. This would then allow the data format to be changed in one place, rather than having to change it on every route that produces JSON. Though, I must admit, I've never seen a service that has change data format in its lifetime, so this really is a mute point.

We were also facing the same issue.
Our DataFormat was Json .Once we implented our own custom marshaller. Camel was encoding the data to base64.I tried the approach provided by Cshculz but our CustomDataFormatter was not getting called for some reason which i couldn't figure out.
So We added .marshal(YourDataFormatter) after every Bean call.This provided us with the formatted json but in the encoded form so in the end of the route we added .unmarshal().json(JsonLibrary.Jackson) to return a raw json to the client.
sample snippet :-
.to("xxxxxxx").marshal(YourDataFormatterBeanRef)
.to("xxxxxxx").marshal(YourDataFormatterBeanRef)
.to("xxxxxxx").marshal(YourDataFormatterBeanRef)
.to("xxxxxxx").marshal(YourDataFormatterBeanRef)
.end().unmarshal().json(JsonLibrary.Jackson)

Simple AggregationStrategy Apache Camel

I would like to know if Camel provides any standard aggregation strategies out-of-the-box. I have been researching but I have only be able to find some of the in the unit test. Those ones cannot be used from the actual code. I am trying to use it with Apache Camel 2.12.1 and Spring DSL.
In particular, I am looking for this one:
public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
if (oldExchange == null) {
return newExchange;
}
String oldBody = oldExchange.getIn().getBody(String.class);
String newBody = newExchange.getIn().getBody(String.class);
oldExchange.getIn().setBody(oldBody + "+" + newBody);
return oldExchange;
}
If they are not included in the package, is there any solution to do the same in Spring DSL without having to create the AggregationStrategy in code?
Thanks for the help!

A generic aggregator would not work. The example you have works if both bodies are Strings and assumes they don't need to be separated with any delimiters. But what about JSON or XML? Simple concatenating those would not work as you'd end up with 2 top level elements. I believe that's why Camel has you implement your own.

At least in version 2.13.1 (the one I'm using now), there exists a GroupedExchangeAggregationStrategy which extends AbstractListAggregationStrategy<Exchange>. The GroupedExchangeAggregationStrategy concatenates all members into a List<T>. After that, it's a matter of converting a list to another type, probably by using a POJO.