We Keep Coding

Apache Flink Process xml and write them to database

i have the following use case.
Xml files are written to a kafka topic which i want to consume and process via flink.
The xml attributes have to be renamed to match the database table columns. These renames have to be flexible and maintainable from outside the flink job.
At the end the attributes have to be written to the database.
Each xml document repesent a database record.
As a second step all some attributes of all xml documents from the last x minutes have to be aggregated.
As i know so far flink is capable of all the mentioned steps but i am lacking of an idea how to implement it corretly.
Currently i have implemented the kafka source, retrieve the xml document and parse it via custom MapFunction. There i create a POJO and store each attribute name and value in a HashMap.
public class Data{
private Map<String,String> attributes = HashMap<>();
}
HashMap containing:
Key: path.to.attribute.one Value: Value of attribute one
Now i would like to use the Broadcasting State to change the original attribute names to the database column names.
At this stage i stuck as i have my POJO data with the attributes inside the HashMap but i don't know how to connect it with the mapping via Broadcasting.
Another way would be to flatMap the xml document attributes in single records. This leaves me with two problems:
How to assure that attributes from one document don't get mixed with them from another document within the stream
How to merge all the attributes of one document back to insert them as one record into the database
For the second stage i am aware of the Window function even if i don't have understood it in every detail but i guess it would fit my requirement. The question on this stage would be if i can use more than one sink in one job while one would be a stream of the raw data and one of the aggregated.
Can someone help with a hint?
Cheers
UPDATE
Here is what i got so far - i simplified the code the XmlData POJO is representing my parsed xml document.
public class StreamingJob {
static Logger LOG = LoggerFactory.getLogger(StreamingJob.class);
public static void main(String[] args) throws Exception {
// set up the streaming execution environment
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
XmlData xmlData1 = new XmlData();
xmlData1.addAttribute("path.to.attribute.eventName","Start");
xmlData1.addAttribute("second.path.to.attribute.eventTimestamp","2020-11-18T18:00:00.000");
xmlData1.addAttribute("third.path.to.attribute.eventSource","Source1");
xmlData1.addAttribute("path.to.attribute.additionalAttribute","Lorem");
XmlData xmlData2 = new XmlData();
xmlData2.addAttribute("path.to.attribute.eventName","Start");
xmlData2.addAttribute("second.path.to.attribute.eventTimestamp","2020-11-18T18:00:01.000");
xmlData2.addAttribute("third.path.to.attribute.eventSource","Source2");
xmlData2.addAttribute("path.to.attribute.additionalAttribute","First");
XmlData xmlData3 = new XmlData();
xmlData3.addAttribute("path.to.attribute.eventName","Start");
xmlData3.addAttribute("second.path.to.attribute.eventTimestamp","2020-11-18T18:00:01.000");
xmlData3.addAttribute("third.path.to.attribute.eventSource","Source1");
xmlData3.addAttribute("path.to.attribute.additionalAttribute","Day");
Mapping mapping1 = new Mapping();
mapping1.addMapping("path.to.attribute.eventName","EVENT_NAME");
mapping1.addMapping("second.path.to.attribute.eventTimestamp","EVENT_TIMESTAMP");
DataStream<Mapping> mappingDataStream = env.fromElements(mapping1);
MapStateDescriptor<String, Mapping> mappingStateDescriptor = new MapStateDescriptor<String, Mapping>(
"MappingBroadcastState",
BasicTypeInfo.STRING_TYPE_INFO,
TypeInformation.of(new TypeHint<Mapping>() {}));
BroadcastStream<Mapping> mappingBroadcastStream = mappingDataStream.broadcast(mappingStateDescriptor);
DataStream<XmlData> dataDataStream = env.fromElements(xmlData1, xmlData2, xmlData3);
//Convert the xml with all attributes to a stream of attribute names and values
DataStream<Tuple2<String, String>> recordDataStream = dataDataStream
.flatMap(new CustomFlatMapFunction());
//Map the attributes with the mapping information
DataStream<Tuple2<String,String>> outputDataStream = recordDataStream
.connect(mappingBroadcastStream)
.process();
env.execute("Process xml data and write it to database");
}
static class XmlData{
private Map<String,String> attributes = new HashMap<>();
public XmlData(){
}
public String toString(){
return this.attributes.toString();
}
public Map<String,String> getColumns(){
return this.attributes;
}
public void addAttribute(String key, String value){
this.attributes.put(key,value);
}
public String getAttributeValue(String attributeName){
return attributes.get(attributeName);
}
}
static class Mapping{
//First string is the attribute path and name
//Second string is the database column name
Map<String,String> mappingTuple = new HashMap<>();
public Mapping(){}
public void addMapping(String attributeNameWithPath, String databaseColumnName){
this.mappingTuple.put(attributeNameWithPath,databaseColumnName);
}
public Map<String, String> getMappingTuple() {
return mappingTuple;
}
public void setMappingTuple(Map<String, String> mappingTuple) {
this.mappingTuple = mappingTuple;
}
}
static class CustomFlatMapFunction implements FlatMapFunction<XmlData, Tuple2<String,String>> {
#Override
public void flatMap(XmlData xmlData, Collector<Tuple2< String,String>> collector) throws Exception {
for(Map.Entry<String,String> entrySet : xmlData.getColumns().entrySet()){
collector.collect(new Tuple2<>(entrySet.getKey(), entrySet.getValue()));
}
}
}
static class CustomBroadcastingFunction extends BroadcastProcessFunction {
#Override
public void processElement(Object o, ReadOnlyContext readOnlyContext, Collector collector) throws Exception {
}
#Override
public void processBroadcastElement(Object o, Context context, Collector collector) throws Exception {
}
}
}

Here's some example code of how to do this using a BroadcastStream. There's a subtle issue where the attribute remapping data might show up after one of the records. Normally you'd use a timer with state to hold onto any records that are missing remapping data, but in your case it's unclear whether a missing remapping is a "need to wait longer" or "no mapping exists". In any case, this should get you started...
private static MapStateDescriptor<String, String> REMAPPING_STATE = new MapStateDescriptor<>("remappings", String.class, String.class);
#Test
public void testUnkeyedStreamWithBroadcastStream() throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment(2);
List<Tuple2<String, String>> attributeRemapping = new ArrayList<>();
attributeRemapping.add(new Tuple2<>("one", "1"));
attributeRemapping.add(new Tuple2<>("two", "2"));
attributeRemapping.add(new Tuple2<>("three", "3"));
attributeRemapping.add(new Tuple2<>("four", "4"));
attributeRemapping.add(new Tuple2<>("five", "5"));
attributeRemapping.add(new Tuple2<>("six", "6"));
BroadcastStream<Tuple2<String, String>> attributes = env.fromCollection(attributeRemapping)
.broadcast(REMAPPING_STATE);
List<Map<String, Integer>> xmlData = new ArrayList<>();
xmlData.add(makePOJO("one", 10));
xmlData.add(makePOJO("two", 20));
xmlData.add(makePOJO("three", 30));
xmlData.add(makePOJO("four", 40));
xmlData.add(makePOJO("five", 50));
DataStream<Map<String, Integer>> records = env.fromCollection(xmlData);
records.connect(attributes)
.process(new MyRemappingFunction())
.print();
env.execute();
}
private Map<String, Integer> makePOJO(String key, int value) {
Map<String, Integer> result = new HashMap<>();
result.put(key, value);
return result;
}
#SuppressWarnings("serial")
private static class MyRemappingFunction extends BroadcastProcessFunction<Map<String, Integer>, Tuple2<String, String>, Map<String, Integer>> {
#Override
public void processBroadcastElement(Tuple2<String, String> in, Context ctx, Collector<Map<String, Integer>> out) throws Exception {
ctx.getBroadcastState(REMAPPING_STATE).put(in.f0, in.f1);
}
#Override
public void processElement(Map<String, Integer> in, ReadOnlyContext ctx, Collector<Map<String, Integer>> out) throws Exception {
final ReadOnlyBroadcastState<String, String> state = ctx.getBroadcastState(REMAPPING_STATE);
Map<String, Integer> result = new HashMap<>();
for (String key : in.keySet()) {
if (state.contains(key)) {
result.put(state.get(key), in.get(key));
} else {
result.put(key, in.get(key));
}
}
out.collect(result);
}
}

JSP to hashmap? [duplicate]

Can a Spring form command be a Map? I made my command a Map by extending HashMap and referenced the properties using the ['property'] notation but it didn't work.
Command:
public class MyCommand extends HashMap<String, Object> {
}
HTML form:
Name: <form:input path="['name']" />
Results in the error:
org.springframework.beans.NotReadablePropertyException: Invalid property '[name]' of bean class [com.me.MyCommand]: Bean property '[name]' is not readable or has an invalid getter method: Does the return type of the getter match the parameter type of the setter?
Is this not allowed or do I have incorrect syntax?

Springn MVC commands need to use JavaBeans naming conventins (ie getXXX() and setXXX()) so no you can't use a map for that.
One alternative is to have a bean with a single Map property ie:
public class MyCommand {
private final Map<String, Object> properties = new HashMap<String, Object>();
public Map<String, Object> getProperties() { return properties; }
// setter optional
}
Then you can do something like this (not 100% sure on the syntax but it is possible):
Name: <form:input path="properties['name']" />

Combining the answer of cletus and dbell I could actually make it work and would like to share the solution with you (including binding of values when submitting the form, a flaw reported for cletus solution)
You cannot use directly a map as command, however have another domain object that needs to wrap a lazy map
public class SettingsInformation {
private Map<String, SettingsValue> settingsMap= MapUtils.lazyMap(new HashMap<String, SettingsValue>(),FactoryUtils.instantiateFactory(SettingsValue.class));
public Map<String, SettingsValue> getSettingsMap() {
return settingsMap;
}
public void setSettingsMap(Map<String, SettingsValue > settingsMap) {
this.settingsMap = settingsMap;
}
}
SettingsValue is a class that actually wraps the value.
public class SettingsValue {
private String value;
public SettingsValue(String value) {
this.value = value;
}
public SettingsValue() {
}
public String getValue() {
return value;
}
public void setValue(String propertyValue) {
this.value = propertyValue;
}
The controller method providing the model looks like this:
#RequestMapping(value="/settings", method=RequestMethod.GET)
public ModelAndView showSettings() {
ModelAndView modelAndView = new ModelAndView("settings");
SettingsDTO settingsDTO = settingsService.getSettings();
Map<String, String> settings = settingsDTO.getSettings();
SettingsInformation settingsInformation = new SettingsInformation();
for (Entry<String, String> settingsEntry : settings.entrySet()) {
SettingsValue settingsValue = new SettingsValue(settingsEntry.getValue());
settingsInformation.getSettingsMap().put(settingsEntry.getKey(), settingsValue);
}
modelAndView.addObject("settings", settingsInformation);
return modelAndView;
}
Your form should look like this
<form:form action="${actionUrl}" commandName="settings">
<form:input path="settingsMap['exampleKey'].value"/>
<input type="submit" value="<fmt:message key="settings.save"/>"/>
</form:form>
The controller method handling form submission works as usual
#RequestMapping(value="/settings", method=RequestMethod.POST)
public ModelAndView updateSettings(#ModelAttribute(value="settings") SettingsInformation settings) {
[...]
}
I verified the SettingsInformation bean is actually filled with the values in the form.
Thanks for helping me with this one; If you have any questions feel free to ask.

Ok i have a solution that works for me i use MapUtils.lazyMap
//My Root domain
public class StandardDomain {
private Map<String, AnotherDomainObj> templateMap= MapUtils.lazyMap(new HashMap<String, AnotherDomainObj>(),FactoryUtils.instantiateFactory(AnotherDomainObj.class));
public Map<String, AnotherDomainObj> getTemplateContentMap() {
return templateMap;
}
public void setTemplateContentMap(Map<String, AnotherDomainObj > templateMap) {
templateMap = templateMap;
}
}
//my second domain
public class AnotherDomainObj {
String propertyValue="";
public String getPropertyValue() {
return propertyValue;
}
public void setPropertyValue(String propertyValue) {
this.propertyValue = propertyValue;
}
}
//In my JSP
<input type="text" value="testthis" name="templateMap['keyName'].propertyValue"/>

Yes it can but... You need to reference it as <form:input path="name[index].key|value"/>
e.g.
<form:input path="name[0].value"/>

struts multi-dimensional map-backed action form [duplicate]

I'm trying to have a dynamic form and, depending on an attribute type, I would like to display a different input style (textfield, radio buttons, dropdown, checklist, ...).
In order to have the dynamic form, I've set up the ActionForm with a Map.
Map<String, Object> values;
public void setValue(String key, Object value);
public Object getValue(String key);
My problem comes when I try to set up a checklist or multibox. The ActionForm only passes one value, although I would have expected that the String[] would be mapped to the Object argument.
Any idea about how can I solve this?
EDIT: in the JSP:
<input type=checkbox name="value(foo)" />

I looked into this issue and found out what was happening. The problem is not with Struts but with BeanUtils (which Struts uses for populating the form with the request parameters).
I managed to duplicate this by extracting a (test only) snippet of code from the framework:
public class MyForm {
// assume this is your Struts ActionForm
public void setValue(String key, Object val) {
System.out.println(key + "=" + val);
}
}
public class Test {
public static void main(String[] args)
throws IllegalAccessException, InvocationTargetException {
MyForm s = new MyForm();
Map<String, Object> properties = new HashMap<String, Object>();
// Your request should be like yourActionUrl?value(foo)=1&value(foo)=2&value(foo)=3
// and Struts calls bean utils with something like:
properties.put("value(foo)", new String[] {"1", "2", "3"});
BeanUtils.populate(s, properties);
}
}
When your run this you get printed one value only (just as you desrbibed):
foo=1
The thing is that BeanUtils considers this a mapped property and treats it as such, going for a scalar value for the key. Since your value is an array it just uses the first element:
...
} else if (value instanceof String[]) {
newValue = getConvertUtils().convert(((String[]) value)[0], type);
...
What you can do is modify your JSP and ActionForm to treat the list of values separately. Example modified:
public class MyForm {
private Map<String, Object> map = new HashMap<String, Object>();
public void setValue(String key, Object val) {
map.put(key, val);
}
public void setPlainValue(String[] values) {
// this is correctly called; now delegate to what you really wanted
setValue("foo", values);
}
}
public class Test {
public static void main(String[] args)
throws IllegalAccessException, InvocationTargetException {
MyForm s = new MyForm();
Map<String, Object> properties = new HashMap<String, Object>();
// Notice the change to your URL..
// yourActionUrl?plainValue=1&plainValue=2&plainValue=3
properties.put("plainValue", new String[] {"1", "2", "3"});
BeanUtils.populate(s, properties);
}
}
The above means that you use
<input type="..." name="value(foo)" ... />
for all the single elements in your JSP, while for your checkboxes (extending it to multivalue elements in general) you use
<input type="checkbox" name="plainValue" ... />
and you delegate to the map once in your ActionForm.

I'm getting "Value is not a valid option" for SelectManyCheckBox

I have this code in my JSP page:
<h:selectManyCheckbox id="chb" value="#{MyBean.selectedCheckBoxes}" layout="pageDirection">
<f:selectItems value="#{MyBean.checkBoxItems}"/>
</h:selectManyCheckbox>
And in my MyBean:
public class MyBean {
public MyBean() {
for (Elem section : sections) {
checkBoxItems.put(section.getName(), section.getObjectID());
}
}
private String[] selectedCheckBoxes;
private Map<String, Object> checkBoxItems = new LinkedHashMap<String, Object>();
public String save() {
//save is not being executed....
return FORWARD;
}
public Map<String, Object> getCheckBoxItems() {
return checkBoxItems;
}
public void setCheckBoxItems(Map<String, Object> checkBoxItems) {
this.checkBoxItems = checkBoxItems;
}
public String[] getSelectedCheckBoxes() {
return selectedCheckBoxes;
}
public void setSelectedCheckBoxes(String[] selectedCheckBoxes) {
this.selectedCheckBoxes = selectedCheckBoxes;
}
}
When I click save it is giving the below message in <t:message for="chb"/>
"chb": Value is not a valid option.
Even though I did not add the required attribute for h:selectManyCheckbox, it is trying to validate or doing something else...
I've changed checkBoxItems variable type(with getter/setters) to List<SelectItem>, but it is not working as well.
What can be the reason, how can I solve it?
PS: I'm using JSF 1.1

You will get this error when the equals() test on a selected item has not returned true for any of the available items. So, when roughly the following happens under JSF's covers:
boolean valid = false;
for (Object availableItem : availableItems) {
if (selectedItem.equals(availableItem)) {
valid = true;
break;
}
}
if (!valid) {
// Validation error: Value is not valid!
}
That can in your particular case only mean that section.getObjectID() does not return a String which is what your selectedCheckboxes is declared to, but a different type or a custom type where equals() is not implemented or broken.
Update as per your comment, the getObjectID() returns Integer. It's thus been treated as String because selectedCheckBoxes is declared as String[]. You should change the following
private String[] selectedCheckBoxes;
private Map<String, Object> checkBoxItems = new LinkedHashMap<String, Object>();
to
private Integer[] selectedCheckBoxes;
private Map<String, Integer> checkBoxItems = new LinkedHashMap<String, Integer>();
and maybe (not sure, can't tell from top of head now) also explicitly supply a converter:
<h:selectManyCheckbox ... converter="javax.faces.Integer">

i didnt find any problem in th code, i thought there is the problem the list u passed to oneManyCheckBox.
hardcode some values in list in getter than check
public Map<String, Object> getCheckBoxItems() {
checkBoxItems.clear();
checkBoxItems.put("aaaa", "aaaa");
checkBoxItems.put("bbbb", "bbbb");
checkBoxItems.put("cccc", "cccc");
checkBoxItems.put("dddd", "dddd");
checkBoxItems.put("eeee", "eeee");
return checkBoxItems;
}

Need help with HashMaps?

I have a following scenario:
public class MapTest {
String name = "guru";
/**
* #param args
*/
public static void main(String[] args) {
MapTest mapTest = new MapTest();
Map map = new HashMap();
map.put("name", mapTest.name);
System.out.println(mapTest.name);
map.put("name", "raj");
System.out.println(mapTest.name);
}
}
output is:
guru
guru
is there any way that I can get the output as
guru
raj
ie. I want the HashMap map and the member variable name to in sync.
Thanks.

You can't do that. That's not the way Java works. When you write:
map.put("name", mapTest.name);
that's putting the current value of mapTest.name into the map. After the argument has been evaluated, it's completely independent of the original expression.
If you need to do something like this, you would have some sort of mutable wrapper class - you'd put a reference to the wrapper into the map, and then you can change the value within the wrapper, and it doesn't matter how you get to the wrapper, you'll still see the change.
Sample code:
import java.util.*;
class StringWrapper {
private String value;
StringWrapper(String value) {
this.value = value;
}
public String getValue() {
return value;
}
public void setValue(String value) {
this.value = value;
}
#Override
public String toString() {
return value;
}
}
public class Test {
public static void main(String[] args) throws Exception {
Map<String, StringWrapper> map = new HashMap<String, StringWrapper>();
StringWrapper wrapper = new StringWrapper("Original");
map.put("foo", wrapper);
System.out.println(map.get("foo"));
wrapper.setValue("Changed");
System.out.println(map.get("foo"));
}
}

You appear to be confused about how maps work. For a better idea, try printing out the map.
map.put("name", mapTest.name);
System.out.println(map);
map.put("name", "raj");
System.out.println(map);
You will get:
{"name"="guru"}
{"name"="raj"}
Note that mapTest.name == "guru" always as you never modify it.

Java maps just don't support anything like this. When you do
map.put("name", mapTest.name);
You put a reference to the object referenced by mapTest.name in the map. When you do
map.put("name", "raj");
You put a reference to the new String object in the map. The reference to mapTest.name isn't in the map anymore.
Out of curiosity, why don't you want to just use map.get("name")?

If you want something to be performed dynamically, you should use a function/method.
public class MapTest {
private final Map<String, String> map = new HashMap<String, String>();
public String name() {
return map.get("name");
}
public static void main(String[] args) {
MapTest mapTest = new MapTest();
mapTest.map.put("name", "guru");
System.out.println(mapTest.name());
mapTest.map.put("name", "raj");
System.out.println(mapTest.name());
}
}
prints
guru
raj