NiFi custom processor expression language - java

I'm trying to make a custom processor in Apache NiFi that can add an attribute/string to the JSON object in the flowfile content. At the moment it works when I just use a string but it's not working when I use NiFi's expression language although I have it supported in my code.
The expression language is 100% correct as it works in another processor and I've also tried different attributes to make sure it's not the attribute.
The property:
public static final PropertyDescriptor ADD_ATTRIBUTE = new PropertyDescriptor
.Builder().name("Add Attribute")
.description("Example Property")
.required(true)
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.expressionLanguageSupported(true)
.build();
Later in my code when I want to get the value and put in the JSON object I use:
jsonObject.put("hostname", context.getProperty(ADD_ATTRIBUTE).evaluateAttributeExpressions().getValue());
I also made a Unit Test and it works when I assign a text value to the testrunner.setProperty. However I don't know how I can assign an attribute to the testrunner or how I can use expression language in my test.
Thanks in advance for any suggestions or a solution!

I'll put my answer from Hortonworks Community Connection here too FWIW:
If the expression refers to attributes on a flow file, you will need to pass a reference to the flow file into evaluateAttributeExpressions:
FlowFile flowFile = session.get();
jsonObject.put("hostname", context.getProperty(ADD_ATTRIBUTE).evaluateAttributeExpressions(flowFile).getValue());
If the property contains an attribute name (rather than an Expression containing an attribute name) and you want the value from the flow file:
jsonObject.put("hostname", flowFile.getAttribute(context.getProperty(ADD_ATTRIBUTE).getValue()));
If the attribute value itself contains Expression Language and you want to evaluate it, take a look at the following class:
org.apache.nifi.attribute.expression.language.Query

Regarding testing...
Assuming you are evaluating the expression language against an incoming FlowFile (evaluateAttributeExpressions(flowFile)) then you can do the following:
runner.setProperty(ADD_ATTRIBUTE, "${my.attribute}");
Then create an attribute Map that has my.attribute in it:
final Map<String,String> attributes = new HashMap<>();
attributes.put("my.attribute", myAttribute);
Then enqueue some content with the attributes:
runner.enqueue(fileIn, attributes);
runner.run();
An example from the code base:
https://github.com/apache/nifi/blob/1e56de9521e4bc0752b419ffc7d62e096db1c389/nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/test/java/org/apache/nifi/processors/solr/TestPutSolrContentStream.java#L243

Related

How to read YAML configuration sections

I'm creating a Yaml configuration reader. I'm having some problems to get the values
from sections. I have already created a way to read not sectioned-keys but when a section like
Test:
Yes:
True: a
False: b
No:
True:
Default: c
False: d
is on the yml file, I want to add the path of the values to a HashMap. In the case above,
the path in the HashMap would be
Test.Yes.True = a
Test.Yes.False = b
Test.No.True.Default = c
Test.No.False = d
notice that Yes and No are sections inside Test. True and False values are inside the section Yes that is in Test etc.
I have no idea how read like the block above.
First of all, you should show what you have tried and where you failed. That helps us to provide very specific answer to your question.
For now, Try converting yaml to corresponding Java POJO. You can do this directly or in steps. You can do it by yourself or can use online converters.
Firstly, you can convert from yaml file to Json. https://www.json2yaml.com/
then with the json, you can create Java POJO's. http://pojo.sodhanalibrary.com/ OR http://www.jsonschema2pojo.org/
The converted json to java POJO will automatically create proper mapping format for you and then you can easily use them.
for your most nested "Default" value case, you can take HashMap with name "HashMap<String, String> trueHashMap = new HashMap<String, String>()" if you wish, otherwise a new POJO with "True.java" will also work".
public class True
{
private String false;
private String true;
// getters setters
}
And then, You need to do some research to map them and pick the configuration.
NOTE:
You can do it by multiple online converters. I have provided only the popular ones.

Passing a variable to #Value annotation for reading specific property from properties file

In my Spring boot application, a specific value needs to be read from the properties file depending on a string value returned from another function. My properties file is as follows:
A=value_a
B=value_b
A function returns either A or B, and stores it in a String variable called stringValue . I am looking for doing something along the lines of the following:
#Value(stringValue)
String propertyValue
However, I get the following message on my IDE:
Attribute value must be constant
I have tried to convert stringValue to a static final variable, but to no avail.
I do know that I can pass a specific key to be read from the properties file, such as the following:
#Value("${A}")
String valueOfA
My question is whether I can pass a variable to the #Value annotation?
#Value annotations are resolved during startup, when Spring context is built. Since stringValue would not be available at this time, you can't use it for injection purposes.
An exception to this scenario would be if the bean with #Value annotation is prototype-scoped, in which case a new instance of it would be created any time it's requested. Still, stringValue would need to be available to the Spring context in order to be used at injection point.
Without seeing more code, it's not possible to give you any more detailed answer.
You can autowire Environment in your application and use it to read from the properties file as it is supposed to be at runtime.( This is assuming all files where your properties are have been added to environment).
I will be posting my answer with assumptions as you have not updated your question, with all information I requested.
So your code would be-
lets call your class where your making a call to a function called getValue to get value of stringValue- Example. Now lets assume you are making a call to the function getValue in a method in the class Example called doSomething().
class Example{
#Autowire
private Enviornment environment.
private String propertyValue
public void doSomething(){
String value=getValue()// this is where u know whether
its A or B.
propertyValue=environment.getProperty(value);
// do whatever u want know
}
Thanks for the help guys! I read the values from the properties file into a org.springframework.core.io.Resource object, and then used that to retrieve the specific value that I required. The following was how I structured my solution code:
#Value("classpath:config.properties")
private Resource propertiesfile;
I then declared a java.util.Properties object and read the values from the Resource object into it.
Properties properties = new Properties();
try{
properties.load(propertiesfile.getInputStream());
}catch(IOException e){
logger.error("Parsing error while reading properties file",e.toString());
}
Finally, based on the value in my stringValue variable, I read the corresponding value from the properties file
String propertyValue = properties.getProperty(stringValue);
if your variable is another environment variable you can try in this format.
#Value("${${stringvalue}}")
where stringvalue is the environment variable.

How to get property attributes for EntityType using Olingo

I'm building a java service that accesses OData information. I'm using olingo to parse all my returned json. If I pull the metadata I can see that my properties have custom attributes under all of my EntityTypes. I'm trying to access these attributes. I can currently access all the properties but all I can seem to get is the key, value pairs.
for(Entry<String, Object> prop : entry.getProperties().entrySet())
Steeping through in Eclipse I can see that the EntityType object has parsed these attributes and is storing them in a hashmap. Any idea how to get at this using Olingo? Last resort would be to side process the metadata xml again but I really really don't want to do that.
I've also seen in debugger that the call
entitySet.getEntityType().getProperty("createdBy")
contains a SimpleProperty property variable that contains all my attribute annotations. Again can't figure out how to get access (both hacking and googleing). sigh :\
After talking with the guys on the olingo user group I have a solution. The code snippet
entitySet.getEntityType().getProperty("createdBy")
Returns an EdmTyped object. This can be cast to a EdmProperty in which the annotations and attributes are available for lookup.
EdmProperty prop = (EdmProperty)entitySet.getEntityType().getProperty("createdBy")
for(EdmAnnotationAttribute attr : property.getAnnotations().getAnnotationAttributes())
{
attr contains your info
}
I didn't try this earlier as when I looked in the debugger after casting to EdmProperty annotations variable was null. Lesson learned again and again, never assume. :)
Annotations are loaded at the time of the call to prop.getAnnotations()

Dynamically add fields to an object using annotations (Java/Groovy)

I'm trying to use Java annotations to be able to add specific fields to an object.
The need is the following : I have a class that processes configuration files where keys are associated with values with the form key=value.
The problem is that I want to be able to let the user defining himself required fields which, if not present, throws exception.
The easiest solution is to pass these fields to the constructor in a String[] but, I also want the user to be able to use these required fields as it were properties of the class, so he's able to write in the code something like :
#RequiredFields(
field1,
field2,
field3)
MyClass myObject = new MyClass(String filePath);
String value = myObject.field1;
and the field field1 is also a completion proposal ?
I'm actually developping in Groovy, so if not possible in standard Java, would it be in Groovy ?
Thanks !
What about a factory method which does the validation? Something like:
MyClass my = MyClassFactory.create("arg1", "arg2")
Or, with maps on Groovy:
def my = MyClassFactory.create arg1: "foo", arg2: "bar"
And the factory itself checks the properties file.
If you really want the annotation, maybe the Type Annotations on JDK 8 are an option.
On Groovy, you can try a local AST, which seems like an overengineered solution to me, or GContracts which is programming by contract.
You could combine a factory with GContracts to #Ensure the resulting object contains all fields according to the properties file.

Create new child in Dom

I have created test class that created DOM object ,currently I created some attributes
Hard coded ,for example I have create element name structure and for the structure we have attributes,I have created different class that handle the attribute with constructor .
These is the code from the main method
Properties properties = new Properties(document);
Element Property = properties.getProperty();
Properties.setProperty(Property, "product_id","10", "Pro ID");
Type.appendChild(Property);
Properties properties1 = new Properties(document);
Element Property1 = properties1.getProperty();
Properties.setProperty(Property1, "curr","5", "Curr Code");
Type.appendChild(Property1);
The code in the constructor is
public Properties(Document document) {
Property = document.createElement(PROPERTY);
}
As you can see for create new property I have created element property and property1 etc hard coded,which is problematic since what will happen
If I will have table that with list of properties with there data ,how should I handle it?
I am not sure if the constructor is the right solution and my question is how to do that better ?
Thanks!!!
It depends on what you're trying to do.
If you want to create a DOM so that you can test your classes that build DOM's, then simply hard-coding calls will work just fine. You just have to make sure you use enough combinations of calls to thoroughly test you API.
If, on the other hand, you need to create a DOM so that you can then proceed to test your API's that require some sort of DOM input, you might want to consider simply creating those DOM's in the form of XML documents and then using the org.w3c.dom API's to create the DOM from the XML.

Categories