We Keep Coding

Apache Flink Process xml and write them to database

i have the following use case.
Xml files are written to a kafka topic which i want to consume and process via flink.
The xml attributes have to be renamed to match the database table columns. These renames have to be flexible and maintainable from outside the flink job.
At the end the attributes have to be written to the database.
Each xml document repesent a database record.
As a second step all some attributes of all xml documents from the last x minutes have to be aggregated.
As i know so far flink is capable of all the mentioned steps but i am lacking of an idea how to implement it corretly.
Currently i have implemented the kafka source, retrieve the xml document and parse it via custom MapFunction. There i create a POJO and store each attribute name and value in a HashMap.
public class Data{
private Map<String,String> attributes = HashMap<>();
}
HashMap containing:
Key: path.to.attribute.one Value: Value of attribute one
Now i would like to use the Broadcasting State to change the original attribute names to the database column names.
At this stage i stuck as i have my POJO data with the attributes inside the HashMap but i don't know how to connect it with the mapping via Broadcasting.
Another way would be to flatMap the xml document attributes in single records. This leaves me with two problems:
How to assure that attributes from one document don't get mixed with them from another document within the stream
How to merge all the attributes of one document back to insert them as one record into the database
For the second stage i am aware of the Window function even if i don't have understood it in every detail but i guess it would fit my requirement. The question on this stage would be if i can use more than one sink in one job while one would be a stream of the raw data and one of the aggregated.
Can someone help with a hint?
Cheers
UPDATE
Here is what i got so far - i simplified the code the XmlData POJO is representing my parsed xml document.
public class StreamingJob {
static Logger LOG = LoggerFactory.getLogger(StreamingJob.class);
public static void main(String[] args) throws Exception {
// set up the streaming execution environment
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
XmlData xmlData1 = new XmlData();
xmlData1.addAttribute("path.to.attribute.eventName","Start");
xmlData1.addAttribute("second.path.to.attribute.eventTimestamp","2020-11-18T18:00:00.000");
xmlData1.addAttribute("third.path.to.attribute.eventSource","Source1");
xmlData1.addAttribute("path.to.attribute.additionalAttribute","Lorem");
XmlData xmlData2 = new XmlData();
xmlData2.addAttribute("path.to.attribute.eventName","Start");
xmlData2.addAttribute("second.path.to.attribute.eventTimestamp","2020-11-18T18:00:01.000");
xmlData2.addAttribute("third.path.to.attribute.eventSource","Source2");
xmlData2.addAttribute("path.to.attribute.additionalAttribute","First");
XmlData xmlData3 = new XmlData();
xmlData3.addAttribute("path.to.attribute.eventName","Start");
xmlData3.addAttribute("second.path.to.attribute.eventTimestamp","2020-11-18T18:00:01.000");
xmlData3.addAttribute("third.path.to.attribute.eventSource","Source1");
xmlData3.addAttribute("path.to.attribute.additionalAttribute","Day");
Mapping mapping1 = new Mapping();
mapping1.addMapping("path.to.attribute.eventName","EVENT_NAME");
mapping1.addMapping("second.path.to.attribute.eventTimestamp","EVENT_TIMESTAMP");
DataStream<Mapping> mappingDataStream = env.fromElements(mapping1);
MapStateDescriptor<String, Mapping> mappingStateDescriptor = new MapStateDescriptor<String, Mapping>(
"MappingBroadcastState",
BasicTypeInfo.STRING_TYPE_INFO,
TypeInformation.of(new TypeHint<Mapping>() {}));
BroadcastStream<Mapping> mappingBroadcastStream = mappingDataStream.broadcast(mappingStateDescriptor);
DataStream<XmlData> dataDataStream = env.fromElements(xmlData1, xmlData2, xmlData3);
//Convert the xml with all attributes to a stream of attribute names and values
DataStream<Tuple2<String, String>> recordDataStream = dataDataStream
.flatMap(new CustomFlatMapFunction());
//Map the attributes with the mapping information
DataStream<Tuple2<String,String>> outputDataStream = recordDataStream
.connect(mappingBroadcastStream)
.process();
env.execute("Process xml data and write it to database");
}
static class XmlData{
private Map<String,String> attributes = new HashMap<>();
public XmlData(){
}
public String toString(){
return this.attributes.toString();
}
public Map<String,String> getColumns(){
return this.attributes;
}
public void addAttribute(String key, String value){
this.attributes.put(key,value);
}
public String getAttributeValue(String attributeName){
return attributes.get(attributeName);
}
}
static class Mapping{
//First string is the attribute path and name
//Second string is the database column name
Map<String,String> mappingTuple = new HashMap<>();
public Mapping(){}
public void addMapping(String attributeNameWithPath, String databaseColumnName){
this.mappingTuple.put(attributeNameWithPath,databaseColumnName);
}
public Map<String, String> getMappingTuple() {
return mappingTuple;
}
public void setMappingTuple(Map<String, String> mappingTuple) {
this.mappingTuple = mappingTuple;
}
}
static class CustomFlatMapFunction implements FlatMapFunction<XmlData, Tuple2<String,String>> {
#Override
public void flatMap(XmlData xmlData, Collector<Tuple2< String,String>> collector) throws Exception {
for(Map.Entry<String,String> entrySet : xmlData.getColumns().entrySet()){
collector.collect(new Tuple2<>(entrySet.getKey(), entrySet.getValue()));
}
}
}
static class CustomBroadcastingFunction extends BroadcastProcessFunction {
#Override
public void processElement(Object o, ReadOnlyContext readOnlyContext, Collector collector) throws Exception {
}
#Override
public void processBroadcastElement(Object o, Context context, Collector collector) throws Exception {
}
}
}

Here's some example code of how to do this using a BroadcastStream. There's a subtle issue where the attribute remapping data might show up after one of the records. Normally you'd use a timer with state to hold onto any records that are missing remapping data, but in your case it's unclear whether a missing remapping is a "need to wait longer" or "no mapping exists". In any case, this should get you started...
private static MapStateDescriptor<String, String> REMAPPING_STATE = new MapStateDescriptor<>("remappings", String.class, String.class);
#Test
public void testUnkeyedStreamWithBroadcastStream() throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment(2);
List<Tuple2<String, String>> attributeRemapping = new ArrayList<>();
attributeRemapping.add(new Tuple2<>("one", "1"));
attributeRemapping.add(new Tuple2<>("two", "2"));
attributeRemapping.add(new Tuple2<>("three", "3"));
attributeRemapping.add(new Tuple2<>("four", "4"));
attributeRemapping.add(new Tuple2<>("five", "5"));
attributeRemapping.add(new Tuple2<>("six", "6"));
BroadcastStream<Tuple2<String, String>> attributes = env.fromCollection(attributeRemapping)
.broadcast(REMAPPING_STATE);
List<Map<String, Integer>> xmlData = new ArrayList<>();
xmlData.add(makePOJO("one", 10));
xmlData.add(makePOJO("two", 20));
xmlData.add(makePOJO("three", 30));
xmlData.add(makePOJO("four", 40));
xmlData.add(makePOJO("five", 50));
DataStream<Map<String, Integer>> records = env.fromCollection(xmlData);
records.connect(attributes)
.process(new MyRemappingFunction())
.print();
env.execute();
}
private Map<String, Integer> makePOJO(String key, int value) {
Map<String, Integer> result = new HashMap<>();
result.put(key, value);
return result;
}
#SuppressWarnings("serial")
private static class MyRemappingFunction extends BroadcastProcessFunction<Map<String, Integer>, Tuple2<String, String>, Map<String, Integer>> {
#Override
public void processBroadcastElement(Tuple2<String, String> in, Context ctx, Collector<Map<String, Integer>> out) throws Exception {
ctx.getBroadcastState(REMAPPING_STATE).put(in.f0, in.f1);
}
#Override
public void processElement(Map<String, Integer> in, ReadOnlyContext ctx, Collector<Map<String, Integer>> out) throws Exception {
final ReadOnlyBroadcastState<String, String> state = ctx.getBroadcastState(REMAPPING_STATE);
Map<String, Integer> result = new HashMap<>();
for (String key : in.keySet()) {
if (state.contains(key)) {
result.put(state.get(key), in.get(key));
} else {
result.put(key, in.get(key));
}
}
out.collect(result);
}
}

map() with JSON array

Not Looking for Jackson Solution.
List of Imports:
import java.io.File;
import java.io.FileReader;
import java.util.ArrayList;
import org.json.simple.JSONArray;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import java.util.List;
import java.util.stream.Collectors;
JSON data:
{
"Name": "ABC.com",
"Developer": "Vishwa ratna",
"Project List": [
"Compnay: National",
"Compnay: Enterprise",
"Compnay: Alamo"
]
}
My program:
#SuppressWarnings("unchecked")
public static void main(String[] args)
{
File text = new File("R:/TestJSON.txt");
JSONParser parser = new JSONParser();
try {
Object obj = parser.parse(new FileReader("R:\\Desktop\\TestJSON.txt"));
JSONObject jsonObject = (JSONObject) obj;
List<String> list = new ArrayList<>();
String name = (String) jsonObject.get("Name");
System.out.println(name);
String author = (String) jsonObject.get("Developer");
System.out.println(author);
JSONArray companyList = (JSONArray) jsonObject.get("Project List");
companyList.stream().map(e->{if(e.equals("Compnay: National")); return e;}).collect(Collectors.toList());
list.forEach(System.out::println);
}
catch (Exception e)
{}
}
}
Everything is executing correctly except
companyList.stream().map(e->{if(e.equals("Compnay: National")); return e;}).collect(Collectors.toList());
list.forEach(System.out::println);
I can get the desired result another way around but i want to know why my map is not returning anything?? should it not return Compnay: National.
Result i am getting:
ABC.com
Vishwa ratna
I am expecting:
ABC.com
Vishwa ratna
Compnay: National
Edit: Even after using filter() as suggested by some people here i am unable to get the desired result.

One you have to use filter instead of map.
Second you have to assign the result of collect to list
Your code should be :
list = (List<String>) companyList.stream()
.filter(e-> e.equals("Compnay: National"))
.map(Object::toString)
.collect(Collectors.toList());
Another solution maybe is to after the filter collect toArray, thenyou convert the array to stream use map with Object::toString then collect to list, this can work also
list = Arrays.stream(
companyList.stream()
.filter(e -> e.equals("Compnay: National"))
.toArray())
.map(Object::toString)
.collect(Collectors.toList());
Why filter and not map
map alwayse return a result, so if you make a condition inside map you have to use else part, it works in your case because you have a typo, your condition not make any sinse, but why you use if inside a map, and you have filter where you can avoid conditions?
Stream<T> filter(Predicate<? super T> predicate)
Like you see filter took a predicate and return a new stream not a boolean like you said in your comment

Another solution using Jackson library
I'm not sure what library you are using to parse your Json file, but, it seems not the perfect way, I would suggest another solution using jackson library, so you have :
1- add to pom.xml the dependency :
<!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-core -->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.9.8</version>
</dependency>
2- create a Job which is similar to your Json file like so :
class MyObject {
private String name;
private String developer;
private List<String> projectList;
public String getName() {
return name;
}
#JsonSetter("Name")
public void setName(String name) {
this.name = name;
}
public String getDeveloper() {
return developer;
}
#JsonSetter("Developer")
public void setDeveloper(String developer) {
this.developer = developer;
}
public List<String> getProjectList() {
return projectList;
}
#JsonSetter("Project List")
public void setProjectList(List<String> projectList) {
this.projectList = projectList;
}
}
3- your code to parse
import com.fasterxml.jackson.annotation.JsonSetter;
import com.fasterxml.jackson.databind.MapperFeature;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.File;
import java.io.IOException;
import java.util.List;
import java.util.stream.Collectors;
public class Mcve {
public static void main(String[] args) throws IOException {
String path = "R:\\Desktop\\TestJSON.txt";
ObjectMapper mapper = new ObjectMapper();
mapper.configure(MapperFeature.ACCEPT_CASE_INSENSITIVE_PROPERTIES, true);
MyObject obj = mapper.readValue(new File(path), MyObject.class);
String name = obj.getName();
System.out.println(name);
String author = obj.getDeveloper();
System.out.println(author);
List<String> list = obj.getProjectList()
.stream()
.filter(e -> e.equals("Compnay: National"))
.collect(Collectors.toList());
list.forEach(System.out::println);
}
}
Outputs
ABC.com
Vishwa ratna
Compnay: National
as you see, working with Objects is more easier and more helpful.

Convert a JSON object to a Map object

I have a JSON object which look as the following:
[{"var1":"value1","var2":"value2"},{"var2":"value22","var3":[["0","1","2"],["3","4","5"],["6","7","8"]]}]
(Note: var2 appears twice in the example and the complex form of the value of var3.)
The desired output should be a map object like:
key value
var1 value1
var2 value2,value22
var3 [["0","1","2"],["3","4","5"],["6","7","8"]]
What I would like is to convert this to a map object with the first elements (var1, var2, var3) as keys and the corresponding values as the values in the map. In case with the identical keys (e.g.: var2) the two values which belong to this key shoul be concatenated, but separated, e.g., by a comma.
Can someone help me with this?

You don't need an adapter to parse a json. you just need to tell ObjectMapper exactly what type to parse into. you also need a bit of post processing since you want some special processing regarding duplicate keys
you get Jackson from GIT: https://github.com/FasterXML/jackson
here is a complete solution for you:
import java.util.*;
import com.fasterxml.jackson.databind.*;
import com.fasterxml.jackson.databind.type.TypeFactory;
public class Test
{
public static void main(String[] args)
{
String input = "[{\"var1\":\"value1\",\"var2\":\"value2\"},{\"var2\":\"value22\",\"var3\":[[\"0\",\"1\",\"2\"],[\"3\",\"4\",\"5\"],[\"6\",\"7\",\"8\"]]}]" ;
Map<String, String> result = new HashMap<>(); // final result, with duplicate keys handles and everything
try {
// ObjectMapper is Jackson json parser
ObjectMapper om = new ObjectMapper();
// we need to tell ObjectMapper what type to parse into
// in this case: list of maps where key is string and value is some cimplex Object
TypeFactory tf = om.getTypeFactory();
JavaType mapType = tf.constructMapType(HashMap.class, String.class, Object.class);
JavaType listType = tf.constructCollectionType(List.class, mapType);
#SuppressWarnings("unchecked")
// finally we parse the input into the data struct
List<Map<String, Object>> list = (List<Map<String, Object>>)om.readValue(input, listType);
// post procesing: populate result, taking care of duplicates
for (Map<String, Object> listItem : list) {
for (Map.Entry<String, Object> mapItem : listItem.entrySet()) {
String key = mapItem.getKey();
String value = mapItem.getValue().toString();
if (result.containsKey(key)) value = result.get(key) + "," + value;
result.put(key, value);
}
}
// result sohuld hold expected outut now
System.out.println(result);
} catch (Exception e) {
e.printStackTrace();
}
}
}
output:
{var3=[[0, 1, 2], [3, 4, 5], [6, 7, 8]], var2=value2,value22, var1=value1}

You can use Jackson to convert to and from JSON to Map. Use the following code and instantiate the JSonAdapter class, use the method marshal(String) to convert the json string to map and unmarshall(Map) for vice versa.
import java.io.IOException;
import java.util.Map;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
public class JsonAdapter {
private static final ObjectMapper MAPPER = new ObjectMapper();
public String unmarshal(final Map<?, ?> jsonList) throws Exception {
return MAPPER.writeValueAsString(jsonList);
}
public Map<?, ?> marshal(final String jsonString) throws Exception {
try {
return MAPPER.readValue(jsonString, new TypeReference<Map<?, ?>>() {
});
} catch (final IOException e) {
e.printStackTrace();
}
return null;
}
}

How to input the objects to a ArrayList <HashMap<String,Object>>

I have 1 object(Goods) have 2 attributes: String and boolean. How to input object Goods to ArrayList<HashMap<String, Object>> ? Because I want input ArrayList<HashMap<String, Object>> to SimpleAdapter
public class Goods {
private String goodsName;
private boolean isCheck = false;
public String getGoodsName() {
return goodsName;
}
public void setGoodsName(String goodsName) {
this.goodsName = goodsName;
}
public boolean isCheck() {
return isCheck;
}
public void setCheck(boolean isCheck) {
this.isCheck = isCheck;
}
}

package ngo;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
public class Main {
public static void main(String[] args) {
Goods g = new Goods();
g.setGoodsName("foo");
g.setCheck(true);
Map<String, Goods> map = new HashMap<String, Goods>();
map.put(g.getGoodsName(), g);
List<Map<String, Goods>> list = new ArrayList<Map<String, Goods>>();
list.add(map);
System.out.println(list.get(0).get("foo").isCheck());
}
}
Displays true
This should be an acceptable, if simple, structure for the data parameter of SimpleAdapter's constructor. A more exhaustive example of it's use can be found here

The following is the basic example, not runnable in itself.
ArrayList<HashMap<String, Goods>> listOfMappedGoods = new ArrayList<HashMap<String, Goods>>();
HashMap<String, Goods> goodsList = new HashMap<String, Goods>();
Goods g = new Goods();
g.setGoodsName("foo");
g.setCheck(true);
goodsList.add(g.getGoodsName(), g);
listOfMappedGoods.add(goodsList);
The point to note, is that just like every new Goods object needs to be created using new Goods(), each new goodsList needs to be created also using new HashMap<String, Goods>().
If you use something like goodsList.clear(), then you are still referring to the original map that was first added to the listOfMappedGoods, so you will clear that instead.

Is there any way to convert a Map to a JSON representation using Jackson without writing to a file?

I'm trying to use Jackson to convert a HashMap to a JSON representation.
However, all the ways I've seen involve writing to a file and then reading it back, which seems really inefficient. I was wondering if there was anyway to do it directly?
Here's an example of an instance where I'd like to do it
public static Party readOneParty(String partyName) {
Party localParty = new Party();
if(connection==null) {
connection = new DBConnection();
} try {
String query = "SELECT * FROM PureServlet WHERE PARTY_NAME=?";
ps = con.prepareStatement(query);
ps.setString(1, partyName);
resultSet = ps.executeQuery();
meta = resultSet.getMetaData();
String columnName, value;
resultSet.next();
for(int j=1;j<=meta.getColumnCount();j++) { // necessary to start at j=1 because of MySQL index starting at 1
columnName = meta.getColumnLabel(j);
value = resultSet.getString(columnName);
localParty.getPartyInfo().put(columnName, value); // this is the hashmap within the party that keeps track of the individual values. The column Name = label, value is the value
}
}
}
public class Party {
HashMap <String,String> partyInfo = new HashMap<String,String>();
public HashMap<String,String> getPartyInfo() throws Exception {
return partyInfo;
}
}
The output would look something like this
"partyInfo": {
"PARTY_NAME": "VSN",
"PARTY_ID": "92716518",
"PARTY_NUMBER": "92716518"
}
So far every example I've come across of using ObjectMapper involves writing to a file and then reading it back.
Is there a Jackson version of Java's HashMap or Map that'll work in a similar way to what I have implemented?

Pass your Map to ObjectMapper.writeValueAsString(Object value)
It's more efficient than using StringWriter, according to the docs:
Method that can be used to serialize any Java value as a String. Functionally equivalent to calling writeValue(Writer,Object) with StringWriter and constructing String, but more efficient.
Example
import org.codehaus.jackson.map.ObjectMapper;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
public class Example {
public static void main(String[] args) throws IOException {
Map<String,String> map = new HashMap<>();
map.put("key1","value1");
map.put("key2","value2");
String mapAsJson = new ObjectMapper().writeValueAsString(map);
System.out.println(mapAsJson);
}
}

You can use a StringWriter.
package test;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.io.StringWriter;
import java.util.HashMap;
import java.util.Map;
public class StringWriterExample {
private static ObjectMapper objectMapper = new ObjectMapper();
public static void main(String[] args) throws IOException {
Map<String,String> map = new HashMap<>();
map.put("key1","value1");
map.put("key2","value2");
StringWriter stringWriter = new StringWriter();
objectMapper.writeValue(stringWriter, map);
System.out.println(stringWriter.toString());
}
}
produces
{"key2":"value2","key1":"value1"}