I'm creating a Yaml configuration reader. I'm having some problems to get the values
from sections. I have already created a way to read not sectioned-keys but when a section like
Test:
Yes:
True: a
False: b
No:
True:
Default: c
False: d
is on the yml file, I want to add the path of the values to a HashMap. In the case above,
the path in the HashMap would be
Test.Yes.True = a
Test.Yes.False = b
Test.No.True.Default = c
Test.No.False = d
notice that Yes and No are sections inside Test. True and False values are inside the section Yes that is in Test etc.
I have no idea how read like the block above.
First of all, you should show what you have tried and where you failed. That helps us to provide very specific answer to your question.
For now, Try converting yaml to corresponding Java POJO. You can do this directly or in steps. You can do it by yourself or can use online converters.
Firstly, you can convert from yaml file to Json. https://www.json2yaml.com/
then with the json, you can create Java POJO's. http://pojo.sodhanalibrary.com/ OR http://www.jsonschema2pojo.org/
The converted json to java POJO will automatically create proper mapping format for you and then you can easily use them.
for your most nested "Default" value case, you can take HashMap with name "HashMap<String, String> trueHashMap = new HashMap<String, String>()" if you wish, otherwise a new POJO with "True.java" will also work".
public class True
{
private String false;
private String true;
// getters setters
}
And then, You need to do some research to map them and pick the configuration.
NOTE:
You can do it by multiple online converters. I have provided only the popular ones.
Related
I am trying to create my own validator for validating a List<String> read from a YAML configuration file. What I want to do, is validate it against a Enum class that I have created which looks like the following:
public enum BundleName {
DK("dk.bundle"),
UK("uk.bundle"),
US("us.bundle"),
DK_DEBUG("dk.bundle.debug"),
UK_DEBUG("uk.bundle.debug"),
US_DEBUG("us.bundle.debug"),
;
private String bundleName;
BundleName(String bundleName) {
this.bundleName = bundleName;
}
public String getBundleName() {
return this.bundleName
}
}
My YAML file has the following defined:
bundleNames:
- ${DK}
- ${UK}
- ${US}
- ${DK_DEBUG}
- ${UK_DEBUG}
- ${US_DEBUG}
Now either some of these environment variables might be set or all of them might be set or just one is set. Different environment different combo. Anyway what I want is to be able to validate these environment variables against the bundleName in enum BundleName.class. Now Hibernate have a nice example, and also all other examples I find on the net is, where validation is done against a specific simple enum. I do not want to check on the enum name but the enum value, and I simply cannot figure out how. All what I have tried so fare is some combinations of what I found in some other posts like this and this and many more.
In my ApiConfiguration.java I would like to end up with something like the following on my list read from YAML:
#NotNull
#BundleNameValidation(acceptedValues = BundleName.class)
private List<String> bundleNames;
If that is possible somehow.
This is the first time I am working with YAML files, so the first think I looked at was to find any library that could help me to parse the file.
I have found two libraries, YamlBean and SnakeYAML. I am not sure which one that I am going to use.
Here is an example of the file that I am trying to parse:
users:
user1:
groups:
- Premium
user2:
groups:
- Mod
user3:
groups:
- default
groups:
Mod:
permissions:
test: true
inheritance:
- Premium
default:
permissions:
test.test: true
inheritance:
- Mod
Admin:
permissions:
test.test.test: true
inheritance:
- Mod
The file will change dynamical so I don't know how many users or groups the file would contain.
The information I would like to fetch from this is the user name and the group like this:
user1 Premium
user2 Mod
user3 default
And from the groups only the group names, like this:
Mod
default
Admin
Anyone could get me started here? And what is the best library to use for this?
YamlBean or SnakeYAML?
I guess, I need to save the information in something that I easily could iterate over.
You could also use Jacksons YAML module.
In order to use that, you'll need a few classes. The model classes which will carry the content of your file and the a class that takes care of reading the YAML file.
The root model class could look like this:
public class MyYamlFile {
#JsonProperty
private List<User> users;
#JsonProperty
private List<Group> groups;
// getter methods ommitted
}
The User(*) class:
public class User {
#JsonProperty
private List<String> name;
#JsonProperty
private List<GroupType> groups;
// getter methods ommitted
}
The GroupType could be an Enum containing all possible group types:
public enum GroupType {
Premium, Mod, Default
}
Don't forget that the enum entries are case sensitive. So "premium" won't work.
You can build all your model classes that way. Every sub entry should get an own model class.
Now to the part where you can read that YAML file:
public MyYamlFile readYaml(final File file) {
final ObjectMapper mapper = new ObjectMapper(new YAMLFactory()); // jackson databind
return mapper.readValue(file, MyYamlFile.class);
}
As you can see, this part is really neat, because you don't need much. The file instance contains your YAML file. You can create one like this:
File file = new File("path/to/my/yaml/usersAndGroups.yaml");
Instead of File the readValue method also supports InputStream, java.io.Reader, String (with the whole content), java.net.URL and byte array.
You should find something that suits you.
(*) You should consider changing the structure of your YAML file, because I don't think it is possible to use dynamic keys with Jackson (maybe someone knows more about that):
users:
- name: user1
groups:
- Premium
- name: user2
groups:
- Mod
- name: user3
groups:
- Default
groups:
....
I ended up using SnakeYaml and made some split strings to solve my issue.
Loaded the yaml file to Object and then into a Map, then split the result from the Map into String[] and then in a for loop I read out the name from the String[]. I did the same with groups.
I know that there is better solutions out there but this is good enough for this project.
Thanks all for the replies.
Found this helpful link that will parse the input without touching java code, if ever need to change the config
https://stackabuse.com/reading-and-writing-yaml-files-in-java-with-snakeyaml
InputStream inputStream = new FileInputStream(new File("src/main/resources/customer.yaml"));
Yaml yaml = new Yaml();
Map<String, Object> data = yaml.load(inputStream);
System.out.println(data);
YamlBean is included to DMelt Java numeric computational environment (http://jwork.org/dmelt/). You can create Yaml files using jhplot.io.HFileYAML class which creates a key-value map and save as yaml file.
I'm trying to make a custom processor in Apache NiFi that can add an attribute/string to the JSON object in the flowfile content. At the moment it works when I just use a string but it's not working when I use NiFi's expression language although I have it supported in my code.
The expression language is 100% correct as it works in another processor and I've also tried different attributes to make sure it's not the attribute.
The property:
public static final PropertyDescriptor ADD_ATTRIBUTE = new PropertyDescriptor
.Builder().name("Add Attribute")
.description("Example Property")
.required(true)
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.expressionLanguageSupported(true)
.build();
Later in my code when I want to get the value and put in the JSON object I use:
jsonObject.put("hostname", context.getProperty(ADD_ATTRIBUTE).evaluateAttributeExpressions().getValue());
I also made a Unit Test and it works when I assign a text value to the testrunner.setProperty. However I don't know how I can assign an attribute to the testrunner or how I can use expression language in my test.
Thanks in advance for any suggestions or a solution!
I'll put my answer from Hortonworks Community Connection here too FWIW:
If the expression refers to attributes on a flow file, you will need to pass a reference to the flow file into evaluateAttributeExpressions:
FlowFile flowFile = session.get();
jsonObject.put("hostname", context.getProperty(ADD_ATTRIBUTE).evaluateAttributeExpressions(flowFile).getValue());
If the property contains an attribute name (rather than an Expression containing an attribute name) and you want the value from the flow file:
jsonObject.put("hostname", flowFile.getAttribute(context.getProperty(ADD_ATTRIBUTE).getValue()));
If the attribute value itself contains Expression Language and you want to evaluate it, take a look at the following class:
org.apache.nifi.attribute.expression.language.Query
Regarding testing...
Assuming you are evaluating the expression language against an incoming FlowFile (evaluateAttributeExpressions(flowFile)) then you can do the following:
runner.setProperty(ADD_ATTRIBUTE, "${my.attribute}");
Then create an attribute Map that has my.attribute in it:
final Map<String,String> attributes = new HashMap<>();
attributes.put("my.attribute", myAttribute);
Then enqueue some content with the attributes:
runner.enqueue(fileIn, attributes);
runner.run();
An example from the code base:
https://github.com/apache/nifi/blob/1e56de9521e4bc0752b419ffc7d62e096db1c389/nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/test/java/org/apache/nifi/processors/solr/TestPutSolrContentStream.java#L243
I have a java application which saves the fields of an object to a file and load them later again.
I used so far a modified version of java.util.properties.Properties file (the properties were written in the order they were put in the properties object). I defined for each field a property-key and use the following function to set the property in the properties object
properties.setProperty(PROPERTY_KEY_FIELD_X, field_x);
and used the following function if I want to read back from the properties file
field_x = properties.getProperty(PROPERTY_KEY_FIELD_X, ""));
where I can add a default value which will be choosed, when there is no property with the specified key. As I maybe add some fields in the future in the class from which the fields were saved in the file, I used this option with the default value to set a field to some defined value for the case I load a properties file which was written with an older version of this class.
So far it works but with some obvious disadvantage, for example I can't use the same property key more than once, I have to add some kind of indices like 1_PROPERTY_KEY, 2_PROPERTY_KEY. Thats why I came across JSON.
I'm not very familiar with JSON and all the things we can achieve with it. But I know I could use it to save the fields of an object to the JSON notation and read it back for example with Gson
//Object to JSON
Gson gson = new Gson();
String json = gson.toJson(object);
//JSON to Object
object = gson.fromJson(json);
But I assume this will fail, if the class is a newer version (i.e. some new fields) when try to read back from the JSON. How can I treat this? Do I have to take care of every seperate field similar to the the current implementation with properties? For example like this (with com.google.gson), assuming jsonString is the string read from the file:
//try/catch block omitted
JSONParser parser = new JSONParser();
JSONElement json = parser.parse(jsonString);
field_x = json.getAsString("field_x");
and take care if something failed to set default values to the fields?
Or is there a completely different approach without JSON or is JSON in general suitable for my use-case?
I'm not an expert of JSON too. But I used it sometimes with Jackson and I think Gson is very similar. If you want to use versioned objects just assure in your object constructor that fields are given with a default value (outside correct values range). Jackson will match only those json fields where it finds a correspondent filed in th Java object. Otherwise the default value is kept.
With GSON, you get to define your model as an object. As an example:
public class Configuration
{
private String x;
private String y;
}
Later, you may adjust Configuration to have an additional field, z:
public class Configuration
{
private String x;
private String y;
private String z;
}
When you use GSON to deserialise your JSON file that does not contain a value for z, it will be left null (or whatever the Java default value for that type is).
Since you have an object, you can define getters that substitute a default value if one is not specified:
public String getZ()
{
if (this.z == null)
{
return "the default z value";
}
return this.z;
}
If you're not goind to send data across network, I think you don't need Json but use native Java classes (ie properties). It's reduntant object creation
Typically I could copy values between two java beans, which have identical property names, using BeanUtils with java reflection e.g. PropertyUtils.setProperty(....)
In protobuf Message, we use the message builder class to set the value. This works but I would rather use reflection to automatically copy properties from the bean to the message as both have identical property names and type.
When I invoke the PropertyUtils.setProperty on the builder object ( got from message.newBuilder()), I get this message.
java.lang.NoSuchMethodException: Property 'testProp' has no setter method in class 'class teststuff.TestBeanProtos$TestBeanMessage$Builder'
How do I automatically copy values from java bean to protobuf message object (and vice-versa) using java reflection?
I hate to answer my question but I cant believe that I am the only one who ran into this problem. Documenting solution here in case other people are also getting started with protobuf and java. Using reflection saves wrting dozens of getter and setters.
Ok , I managed to get it to work using some of example test code shipping with protobuf. This is a very simple use case; typically a message would be a lot more complex. This code does not handle nested messages or repeated messages.
public static void setMessageBuilder(com.google.protobuf.GeneratedMessage.Builder message,Descriptors.Descriptor descriptor,Object srcObject) throws Exception {
String cname = srcObject.getClass().getName();
/*BeanMapper.getSimpleProperties -- this is a warpper method that gets the list of property names*/
List<String> simpleProps = BeanMapper.getSimpleProperties(srcObject.getClass());
Map map = new HashMap();
for (String pName : simpleProps) {
System.out.println(" processing property "+ pName);
Object value= PropertyUtils.getProperty(srcObject, pName);
if(value==null) continue;
Descriptors.FieldDescriptor fd=descriptor.findFieldByName(pName) ;
System.out.println(" property "+ pName+" , found fd :"+ (fd==null ? "nul":"ok"));
message.setField(fd, value);
System.out.println(" property "+ pName+" set ok,");
}
return ;
}
I may be off, but would protostuff help? It has nice extended support for working with other data formats, types. And even if it didn't have direct conversion support, if you go to/from JSON there are many choices for good data binding.
You can go throw all properties getClass().getFields() and make copy using reflection. It will be smt like:
for(Field f : to.getClass().getFields()){
f.set(to, from.getClass().getField(f.getName()).get(from));
}
+ probably you might be use field.setAccessible(true) invocation.
I don't know the size of your project but you may want to try Dozer, a mapper that recursively copies data from one object to another of the same type or between different complex types. Supports implicit and explicit mapping as well. I used it in a big project and worked very well. It could be as simple as
Mapper mapper = new DozerBeanMapper();
DestinationObject destObject = mapper.map(sourceObject, DestinationObject.class);
I've got the same issue, the solution is a little bit tricky.
Please use
MethodUtils.invokeMethod
instead.
where the method name is "setXXX".