We Keep Coding

Flattening a heavily nested JSON in Java - Time Complexity

{
"id": "12345678",
"data": {
"address": {
"street": "Address 1",
"locality": "test loc",
"region": "USA"
},
"country_of_residence": "USA",
"date_of_birth": {
"month": 2,
"year": 1988
},
"links": {
"self": "https://testurl"
},
"name": "John Doe",
"nationality": "XY",
"other": [
{
"key1": "value1",
"key2": "value2
},
{
"key1": "value1",
"key2": "value2"
}
],
"notified_on": "2016-04-06"
}
}
I am trying to read data from a GraphQL API that returns paginated JSON response. I need to write this into a CSV. I have been exploring Spring Batch for implementation where I would read JSON data in the ItemReader and flatten each JSON entry (in ItemProcessor) and then write this flattened data into a CSV (in ItemWriter). While I could use something like Jackson for flattening the JSON, I am concerned about possible performance implications if the JSON data is heavily nested.
expected output:
id, data.address.street, data.address.locality, data.address.region, data.country_of_residence, data.date_of_birth.month, data.date_of_birth.year, data.links.self, data.name, data.nationality, data.other (using jsonPath), data.notified_on
I need to do process more than a million records. While I believe flattening the CSV would be a linear operation O(n), I was still wondering if there could be other caveats if the JSON structure gets severely nested.

How to select fields in different levels of a jsonfile with jsonPath?

I want to convert jsonobjcts into csv files. Wy (working) attempt so far is to load the json file as a JSONObject (from the googlecode.josn-simple library), then converting them with jsonPath into a string array which is then used to build the csv rows. However I am facing a problem with jsonPath. From the given example json...
{
"issues": [
{
"key": "abc",
"fields": {
"issuetype": {
"name": "Bug",
"id": "1",
"subtask": false
},
"priority": {
"name": "Major",
"id": "3"
},
"created": "2020-5-11",
"status": {
"name": "OPEN"
}
}
},
{
"key": "def",
"fields": {
"issuetype": {
"name": "Info",
"id": "5",
"subtask": false
},
"priority": {
"name": "Minor",
"id": "2"
},
"created": "2020-5-8",
"status": {
"name": "DONE"
}
}
}
]}
I want to select the following:
[
"abc",
"Bug",
"Major",
"2020-5-11",
"OPEN",
"def",
"Info",
"Minor",
"2020-5-8",
"DONE"
]
The csv should look like that:
abc,Bug,Major,2020-5-11,OPEN
def,Info,Minor,2020-5-8,DONE
I tried $.issues.[*].[key,fields] and I get
"abc",
{
"issuetype": {
"name": "Bug",
"id": "1",
"subtask": false
},
"priority": {
"name": "Major",
"id": "3"
},
"created": "2020-5-11",
"status": {
"name": "OPEN"
}
},
"def",
{
"issuetype": {
"name": "Info",
"id": "5",
"subtask": false
},
"priority": {
"name": "Minor",
"id": "2"
},
"created": "2020-5-8",
"status": {
"name": "DONE"
}
}
]
But when I want to select e.g. only "created" $.issues.[*].[key,fields.[created]
[
"2020-5-11",
"2020-5-8"
]
This is the result.
But I just do not get how to select "key" and e.g. "name" in the field issuetype.
How do I do that with jsonPath or is there a better way to filter a jsonfile and then convert it into a csv?

I recommend what I believe is a better way - which is to create a set of Java classes which represent the structure of your JSON data. When you read the JSON into these classes, you can manipulate the data using standard Java.
I also recommend a different JSON parser - in this case Jackson, but there are others. Why? Mainly, familiarity - see later on for more notes on that.
Starting with the end result: Assuming I have a class called Container which contains all the issues listed in the JSON file, I can then populate it with the following:
//import com.fasterxml.jackson.databind.ObjectMapper;
String jsonString = "{...}" // your JSON data as a string, for this demo.
ObjectMapper objectMapper = new ObjectMapper();
Container container = objectMapper.readValue(jsonString, Container.class);
Now I can print out all the issues in the CSV format you want as follows:
container.getIssues().forEach((issue) -> {
printCsvRow(issue);
});
Here, the printCsvRow() method looks like this:
private void printCsvRow(Issue issue) {
String key = issue.getKey();
Fields fields = issue.getFields();
String type = fields.getIssuetype().getName();
String priority = fields.getPriority().getName();
String created = fields.getCreated();
String status = fields.getStatus().getName();
System.out.println(String.join(",", key, type, priority, created, status));
}
In reality, I would use a CSV library to ensure records are formatted correctly - the above is just for illustration, to show how the JSON data can be accessed.
The following is printed:
abc,Bug,Major,2020-5-11,OPEN
def,Info,Minor,2020-5-8,DONE
And to filter only OPEN records, I can do something like this:
container.getIssues()
.stream()
.filter(issue -> issue.getFields().getStatus().getName().equals("OPEN"))
.forEach((issue) -> {
printCsvRow(issue);
});
The following is printed:
abc,Bug,Major,2020-5-11,OPEN
To enable Jackson, I use Maven with the following dependency:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.10.3</version>
</dependency>
In case you don't use Maven, this gives me 3 JARs: jackson-databind, jackson-annotations, and jackson-core.
To create the nested Java classes I need (to mirror the structure of the JSON), I use a tool which generates them for me using your sample JSON.
In my case, I used this tool, but there are others.
I chose "Container" as the name of the root Java class; a source type of JSON; and selected Jackson 2.x annotations. I also requested getters and setters.
I added the generated classes (Fields, Issue, Issuetype, Priority, Status, and Container) to my project.
WARNING: The completeness of these Java classes is only as good as the sample JSON. But you can, of course, enhance these classes to more accurately reflect the actual JSON you need to handle.
The Jackson ObjectMapper takes care of loading the JSON into the class structure.
I chose to use Jackson instead of JsonPath, simply because of familiarity. JsonPath appears to have very similar object mapping capabilities - but I have never used those features of JsonPath.
Final note: You can use xpath style predicates in JsonPath to access individual data items and groups of items - as you describe in your question. But (in my experience) it is almost always worth the extra effort to create Java classes, if you want to process all your data in more flexible ways - especially if that involves transforming the JSON input into different output structures.

How to get value from json object with java obect mapper

I want to get the value at the field first inside name.
How i can access in this field using HashMap in java
{ "payload":{
"name": {
"first": "jean",
"last": "bob,
},
"address": {
"code": "75",
"city": "paris",
"country": "France"
},
}}

Use one of the available Java libraries for handling JSON. E.g. Gson from Guava API. They are pretty straing fw.

Jackson converting dynamic json to map

I have a problem where some structure of the json is fixed while some part is dynamic. The end output has to be an object of type
Map<String,List<Map<String,String>>>
I am pasting a sample json code for which the jackson work -
{
"contentlets": [
{
"template": "8f8fab8e-0955-49e1-a2ed-ff45e3296aa8",
"modDate": "2017-01-06 13:13:20.0",
"cachettl": "0",
"title": "New Early Warnings",
"subscribeToListIi": "am#zz.com",
"inode": "15bd497-1d8e-4bc7-b0f4-c799ed89fdc9",
"privacySetting": "public",
"__DOTNAME__": "New gTLD Early Warnings",
"activityStatus": "Completed",
"host": "10b6f94a-7671-4e08-9f4b-27bca80702e7",
"languageId": 1,
"createNotification": false,
"folder": "951ff45c-e844-40d4-904f-92b0d2cd0c3c",
"sortOrder": 0,
"modUser": "dotcms.org.2897"
}
]
}
ObjectMapper mapper = new ObjectMapper();
Map<String,List<Map<String,String>>> myMap=mapper.readValue(responseStr.getBytes(), new TypeReference<HashMap<String,List<Map<String,String>>>>() {});
The above code is working fine but when the json changes to (basically a metadata tag is added) it is not able to convert to map.
{
"contentlets": [
{
"template": "8f8fab8e-0955-49e1-a2ed-ff45e3296aa8",
"modDate": "2017-01-06 13:13:20.0",
"cachettl": "0",
"title": "New gTLD Early Warnings",
"subscribeToListIi": "aml#bb.com",
"inode": "15bd4057-1d8e-4bc7-b0f4-c799ed89fdc9",
"metadata": {
"author": "jack",
"location": "LA"
},
"privacySetting": "public",
"__DOTNAME__": "New gTLD Early Warnings",
"activityStatus": "Completed",
"host": "10b6f94a-7671-4e08-9f4b-27bca80702e7",
"languageId": 1,
"createNotification": false,
"folder": "951ff45c-e844-40d4-904f-92b0d2cd0c3c",
"sortOrder": 0,
"modUser": "dotcms.org.2897"
}
]
}

This is expected since the type of the value of metadata is not a String. If you change the type of the map accordingly then it works:
Map<String,List<Map<String,Object>>> myMap = mapper.readValue(reader, new TypeReference<HashMap<String,List<Map<String,Object>>>>() {});
Of course you are left with the problem that values in the map are not of the same type. so you need to ask yourself what is the desired data structure you want and how you further process it. However, one cannot deserialize a json structure into a simple String.

single file Json Parsing to multiple java objects

I have a Json file which i want to parse and store it to different objects based on a particular value in the json using java?
I have a file of this type :
{
"TEST_ID": "INV_CRE_184",
"TEST_DESCRIPTION": "This validate Invoice Creation by mocking dependency",
"TEST_PLAN": "LINK",
"TEST_VARIABLE": [{
"retailcountryCode": "DE"
}],
"TEST_CASE": [{
"OUTPUT_PLACEHOLDER": "V1",
"ACTION": "Generate",
"PARAMETERS": [
"GenerateVendor",
"VENDOR",
"${retailcountryCode}"
]
}, {
"OUTPUT_PLACEHOLDER": "P1",
"ACTION": "Generate",
"PARAMETERS": [
"GeneratePayee",
"${V1}",
"${ofaOrg}"
]
},
I have two objects V1 and P1 and according to the placeholder values they must be parsed respectively