Use pattern in getAttribute in NiFi - java

How can I use a pattern in getAttribute of a FlowFile?
I'm going to write a processor that receives flowfiles from ListenTCP and ListenUDP processors. ListenTCP has tcp.sender property and ListenUDP has udp.sender property. How can I get the sender property of a FlowFile?
The current solution is:
String sender = flowfile.getAttribute("tcp.sender");
if(sender!=null && !sender.isEmpty()) {
// do something
}
else {
sender = flowfile.getAttribute("udp.sender");
if(sender!=null && !sender.isEmpty()) {
//do something
}
}
How can I avoid using if? I need something like this:
String sender = flowfile.getAttribute("*.sender");

There currently isn't a way to get an attribute based on a pattern. If there was, it would return a list of multiple attribute values, and you will still have to go through the list and find the one you are interested in.
You could make your custom processor require an attribute like "network.sender" and after ListenTCP and ListenUDP, have an UpdateAttribute processor for each of them that renames "tcp.sender" to "network.sender" and "udp.sender" to "network.sender".

Related

Get Original Field Name on GraphQL

I'm using https://github.com/leangen/graphql-spqr with spring-boot java application. I can reach to alias name easily but how can I reach to original fieldName?
class Food {
#GraphQLQuery(name = "aliasNameX", description = "A food's name")
private String originalName;
...
}
....
#GraphQLQuery(name = "foods") // READ ALL
#Override
public List<Food> getFoods(#GraphQLEnvironment ResolutionEnvironment env) {
DataFetchingFieldSelectionSet selectionSet = env.dataFetchingEnvironment.getSelectionSet();
List<SelectedField> fields = selectionSet.getFields();
for (SelectedField f: fields)
{
System.out.println(f.getName());
}
return foodRepository.findAll();
}
When I run this code, Output looks like with alias fields: "aliasNameX", ..., but I need original name like "originalName". Is there a way to do it?
Solved, according to:
https://github.com/leangen/graphql-spqr/issues/381
Posting my original answer here as well.
You want the underlying field names, but from a level above. Still possible, but ugly :(
for (SelectedField selectedField : env.dataFetchingEnvironment.getSelectionSet().getImmediateFields()) {
Optional<Operation> operation = Directives.getMappedOperation(selectedField.getFieldDefinition());
String javaName = operation.map(op -> ((Member) op.getTypedElement().getElement()).getName()).orElse(null);
}
Be very careful though. If there's more than one Java element exposed per GraphQL field, getTypedElement().getElement() will explode. So to be sure you'd have to call getTypedElement().getElements() (plural) instead and decide what to do. ClassUtils#getPropertyMembers might also be useful, or the ClassUtils.findXXX family of methods.
You'd basically have to do this:
List<AnnotatedElement> elements = getTypedElement().getElements();
//Look for a field and use its name
Optional<String> field = Utils.extractInstances(elements, Field.class).findFirst().map(Field::getName);
//Look for a getter and find its associated field name
Optional<String> getter = Utils.extractInstances(elements, Method.class).findFirst().map(ClassUtils::getFieldNameFromGetter);
This API might have to change in future, as SDL-based tools are proliferating, so complex directives like the ones SPQR is using are causing problems...

Retry strategy/framework for doing retry on result for failed insertions

I'm making a service call with some input objects to a database. Few of the objects succeed and rest of them fail with a reason. I want to have an exponential retry strategy to trigger the same call with the failed objects.
Is there any framework i can use with a defined retry strategy in java
Example call
Output output = mDbService.persist(List<Objects> objects)
The output of the service call looks like
class Output {
int successCount;
List<FailedObject> failedObject;
}
// Each failedObject has a structure
class FailedObject {
Object object;
int errorCode; // Some are retryable and other are not.
}
I don’t know, a framework seems overkill, but maybe I’m misunderstanding your question.
Would something like this do what you want?
Output output = mDbService.persist(objects);
while(output.failedObject.size() > 0){
List<Object> retryList = new ArrayList<>();
for(FailedObject failed:output.failedObject){
if(shouldRetry(failed.errorCode)){
retryList.add(failed.object);
}
}
output = mDbService.persist(retryList);
}
You will naturally also have to provide an implementation that checks the error code and performs the logic to determine if the associated object should be retried or not.
private boolean shouldRetry(int errorCode){
// your logic here
}

Java Jersey REST Request Parameter Sanitation

I'm trying to make sure my Jersey request parameters are sanitized.
When processing a Jersey GET request, do I need to filter non String types?
For example, if the parameter submitted is an integer are both option 1 (getIntData) and option 2 (getStringData) hacker safe? What about a JSON PUT request, is my ESAPI implementation enough, or do I need to validate each data parameter after it is mapped? Could it be validated before it is mapped?
Jersey Rest Example Class:
public class RestExample {
//Option 1 Submit data as an Integer
//Jersey throws an internal server error if the type is not Integer
//Is that a valid way to validate the data?
//Integer Data, not filtered
#Path("/data/int/{data}/")
#GET
#Produces(MediaType.TEXT_HTML)
public Response getIntData(#PathParam("data") Integer data){
return Response.ok("You entered:" + data).build();
}
//Option 2 Submit data as a String, then validate it and cast it to an Integer
//String Data, filtered
#Path("/data/string/{data}/")
#GET
#Produces(MediaType.TEXT_HTML)
public Response getStringData(#PathParam("data") String data) {
data = ESAPI.encoder().canonicalize(data);
if (ESAPI.validator().isValidInteger("data", data, 0, 999999, false))
{
int intData = Integer.parseInt(data);
return Response.ok("You entered:" + intData).build();
}
return Response.status(404).entity("404 Not Found").build();
}
//JSON data, HTML encoded
#Path("/post/{requestid}")
#POST
#Consumes({MediaType.APPLICATION_FORM_URLENCODED, MediaType.APPLICATION_JSON})
#Produces(MediaType.TEXT_HTML)
public Response postData(String json) {
json = ESAPI.encoder().canonicalize(json);
json = ESAPI.encoder().encodeForHTML(json);
//Is there a way to iterate through each JSON KeyValue and filter here?
ObjectMapper mapper = new ObjectMapper();
DataMap dm = new DataMap();
try {
dm = mapper.readValue(json, DataMap.class);
} catch (Exception e) {
e.printStackTrace();
}
//Do we need to validate each DataMap object value and is there a dynamic way to do it?
if (ESAPI.validator().isValidInput("strData", dm.strData, "HTTPParameterValue", 25, false, true))
{
//Is Integer validation needed or will the thrown exception be good enough?
return Response.ok("You entered:" + dm.strData + " and " + dm.intData).build();
}
return Response.status(404).entity("404 Not Found").build();
}
}
Data Map Class:
public class DataMap {
public DataMap(){}
String strData;
Integer intData;
}
The short answer is yes, though by "filter" I interpret it as "validate," because no amount of "filtering" will EVER provide you with SAFE data. You can still run into integer overflows in Java, and while those may not have immediate security concerns, they could still put parts of your application in an unplanned for state, and hacking is all about perturbing the system in ways you can control.
You packed waaaaay too many questions into one "question," but here we go:
First off, the lines
json = ESAPI.encoder().canonicalize(json);
json = ESAPI.encoder().encodeForHTML(json);
Aren't doing what you think they're doing. If your JSON is coming in as a raw String right here, these two calls are going to be applying mass rules across the entire string, when you really need to handle these with more surgical precision, which you seem to at least be subconsciously aware of in the next question.
//Is there a way to iterate through each JSON KeyValue and filter
here?
Partial duplicate of this question.
While you're in the loop discussed here, you can perform any data transformations you want, but what you should really be considering is using the JSONObject class referenced in that first link. Then you'll have JSON parsed into an object where you'll have better access to JSON key/value pairs.
//Do we need to validate each DataMap object value and is there a
dynamic way to do it?
Yes, we validate everything that comes from a user. All users are assumed to be trained hackers, and smarter than you. However if you handled filtering before you do your data mapping transformation, you don't need to do it a second time. Doing it dynamically?
Something like:
JSONObject json = new JSONObject(s);
Iterator iterator = json.keys();
while( iterator.hasNext() ){
String data = iterator.next();
//filter and or business logic
}
^^That syntax is skipping typechecks but it should get you where you need to go.
/Is Integer validation needed or will the thrown exception be good
enough?
I don't see where you're throwing an exception with these lines of code:
if (ESAPI.validator().isValidInput("strData", dm.strData, "HTTPParameterValue", 25, false, true))
{
//Is Integer validation needed or will the thrown exception be good enough?
return Response.ok("You entered:" + dm.strData + " and " + dm.intData).build();
}
Firstly, in java we have autoboxing which means this:
int foo = 555555;
String bar = "";
//the code
foo + bar;
Will be cast to a string in any instance. The compiler will promote the int to an Integer and then silently call the Integer.toString() method. Also, in your Response.ok( String ); call, THIS is where you're going to want to encodeForHTML or whatever the output context may be. Encoding methods are ALWAYS For outputting data to user, whereas canonicalize you want to call when receiving data. Finally, in this segment of code we also have an error where you're assuming that you're dealing with an HTTPParameter. NOT at this point in the code. You'll validate http Parameters in instances where you're calling request.getParameter("id"): where id isn't a large blob of data like an entire JSON response or an entire XML response. At this point you should be validating for things like "SafeString"
Usually there are parsing libraries in Java that can at least get you to the level of Java objects, but on the validation side you're always going to be running through every item and punting whatever might be malicious.
As a final note, while coding, keep these principles in mind your code will be cleaner and your thought process much more focused:
user input is NEVER safe. (Yes, even if you've run it through an XSS filter.)
Use validate and canonicalize methods whenever RECEIVING data, and encode methods whenever transferring data to a different context, where context is defined as "Html field. Http attribute. Javascript input, etc...)
Instead of using the method isValidInput() I'd suggest using getValidInput() because it will call canonicalize for you, making you have to provide one less call.
Encode ANY time your data is going to be passed to another dynamic language, like SQL, groovy, Perl, or javascript.

Using camel to aggregate messages of same header

I have multiple clients that send files to a server. For one set of data there are two files that contain information about that data, each with the same name. When a file is received, the server sends a message out to my queue containing the file path, file name, ID of the client, and the "type" of file it is (all have same file extension but there are two "types," call them A and B).
The two files for one set of data have the same file name. As soon as the server has received both of the files I need to start a program that combines the two. Currently I have something that looks like this:
from("jms:queue.name").aggregate(header("CamelFileName")).completionSize(2).to("exec://FILEPATH?args=");
Where I am stuck is the header("CamelFileName"), and more specifically how the aggregator works.
With the completionSize set to 2 does it just suck up all the messages and store them in some data structure until a second message that matches the first comes through? Also, does the header() expect a specific value? I have multiple clients so I was thinking of having the client ID and the file name in the header, but then again I don't know if I have to give a specific value. I also don't know if I can use a regex or not.
Any ideas or tips would be super helpful.
Thanks
EDIT:
Here is some code I have now. Based on my description of the problem here and in comments on selected answer does it seem accurate (besides close brackets that I didn't copy over)?
public static void main(String args[]) throws Exception{
CamelContext c = new DefaultCamelContext();
c.addComponent("activemq", activeMQComponent("vm://localhost?broker.persistent=false"));
//ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("vm://localhost?broker.persistent=false");
//c.addComponent("jms", JmsComponent.jmsComponentAutoAcknowledge(connectionFactory));
c.addRoutes(new RouteBuilder() {
public void configure() {
from("activemq:queue:analytics.camelqueue").aggregate(new MyAggregationStrategy()).header("subject").completionSize(2).to("activemq:queue:analytics.success");
}
});
c.start();
while (true) {
System.out.println("Waiting on messages to come through for camel");
Thread.sleep(2 * 1000);
}
//c.stop();
}
private static class MyAggregationStrategy implements AggregationStrategy {
public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
if (oldExchange == null)
return newExchange;
// and here is where combo stuff goes
String oldBody = oldExchange.getIn().getBody(String.class);
String newBody = newExchange.getIn().getBody(String.class);
boolean oldSet = oldBody.contains("set");
boolean newSet = newBody.contains("set");
boolean oldFlow = oldBody.contains("flow");
boolean newFlow = newBody.contains("flow");
if ( (oldSet && newFlow) || (oldFlow && newSet) ) {
//they match so return new exchange with info so extractor can be started with exec
String combined = oldBody + "\n" + newBody + "\n";
newExchange.getIn().setBody(combined);
return newExchange;
}
else {
// no match so do something....
return null;
}
}
}
you must supply an AggregationStrategy to define how you want to combine Exchanges...
if you are only interested in the fileName and receiving exactly 2 Exchanges, then you can just use the UseLatestAggregationStrategy to just pass the newest Exchange through once 2 have been 'aggregated'...
that said, it sounds like you need to retain both Exchanges (one for each clientId) so you can pass that info on to the 'exec' step...if so, you can just combine the Exchanges into a GroupedExchange holder using the built-in aggregation strategy enabled via the groupExchanges option...or specificy a custom AggregationStrategy to combine them however you'd like. just need to keep in mind that your 'exec' step needs to handle whatever aggregated structure you decide to use...
see these unit tests for examples:
https://svn.apache.org/repos/asf/camel/trunk/camel-core/src/test/java/org/apache/camel/processor/aggregator/AggregatorTest.java
https://svn.apache.org/repos/asf/camel/trunk/camel-core/src/test/java/org/apache/camel/processor/aggregator/AggregateGroupedExchangeTest.java

Deadbolt - Play Framework - How to check a #RestrictedResource with parameters in a controller?

With Deadbolt's module we can check the restrictedResource with a ressource name and parameters in the view.
For example in my view, I have it, and it works well:
#{deadbolt.restrictedResource resourceKeys:['Domain'] , resourceParameters:['domainid':domain.id]}
<li>${domain.title}</li>
#{/deadbolt.restrictedResource}
But in my controller, I just can check the ressource name but I don't find a way to check it in my RestrictedResourcesHandler passing the domainid with.
I am looking for a solution to do something like that:
#RestrictedResource(name = {"Domain"}, params = {domainid})
public static void showDomain(String domainid)
{
}
Thanks in advance
It's not possible to have dynamic information in an annotation, but you can use params to define the name of an incoming value in the request. However, this information isn't passed into the handler at the moment because it expects a map. While you can pass in a map of parameters from the restrictedResource tag, you can't do this from an annotation so an empty map is passed into the handler.
Your best approach here is to pull a well-known parameter name from the request object. I need to have a rethink about the best way to do this without breaking backwards compatibility.
Steve (author of Deadbolt)
I've found a way the solved the problem, not the best I think, but it is the Steve Chaloner's solution (Deadbolt's creator), and it works.
For example, if your Controller's method argument is named "id", and you want to check this id inside your checkAccess method :
// Controller's method :
#RestrictedResource(name = {"Domain"})
public static void showDomain(String id){}
Just check at the beginning of your checkAccess method the Map "resourceParameters" is empty, and use the request object to get the parameters:
public AccessResult checkAccess(List<String> resourceNames,
Map<String, String> resourceParameters)
{
Map<String, String> hashm = new HashMap<String,String>();
if(resourceParameters != null && !resourceParameters.isEmpty()){
hashm = resourceParameters;
}else if(Http.Request.current().routeArgs!= null && !Http.Request.current().routeArgs.isEmpty()){
hashm = Http.Request.current().routeArgs;
}
}
Then just have to foreach your hashmap inside your checkAccess method to get your Controller's method argument and check the Access as you wish.
for (Map.Entry<String,String> mp : hashm.entrySet())
{
// Get the id argument
if(mp.getKey().equals("id"))
{
// Do something with the value..
mp.getValue()
}
}

Categories