We Keep Coding

Java Spring - How to get one specific key value from a very nested json without mapping to a specific pojo?

Small java and spring question regarding how to get a specific key value from a very nested json, without having to map back to java pojos please.
I am consuming an api, where the json response is gigantic. The raw response does not fit in a screen.
The response is also very nested. Meaning, it has fields inside fields inside fields... etc
I have no access, no way to modify this API.
Nonetheless, it is a very interesting API, and in this very gigantic payload, very nested, there is always exactly one "the-key-i-need": "the-value-i-need",
Furthermore, there is no way to know in advanced how nested (which layer, which child) and no way to know where will "the-key-i-need": "the-value-i-need", be.
Also, there is no way to map the response back to any POJO, it changes always, the only information, "the-key-i-need": "the-value-i-need", exists, and it is always there.
I am well aware of GSON or fasterxml libraries, that can help map the json string back to existing pojos.
However, in my case, such will not help, since the existing pojo does not exists. the response is always different, the structure, the level of nesting is always different.
My question is, instead of trying to map back to pojos that will always change, and very nested, is there a simpler way, to just use some kind of regex, or something else, to extract the key value, "the-key-i-need": "the-value-i-need", and only this please?
I already tried mapping to all kinds of pojos, and unfortunately the response structure is too dynamic.
Thank you

Since you're using Spring, you very likely already have Jackson FasterXML in you classpath.
Normally, Spring uses the Databind module, which relies on the Streaming module, aka the Core module.
In this case, you want to use Streaming directly, so get the JSON text as a String, and start the parser.
static String getFieldValue(String json, String field) throws JsonParseException, IOException {
JsonFactory factory = new JsonFactory();
try (JsonParser parser = factory.createParser(json)) {
for (JsonToken token; (token = parser.nextToken()) != null; ) {
if (token == JsonToken.FIELD_NAME && parser.getCurrentName().equals(field)) {
token = parser.nextToken();
// check token here if object or array should throw exception instead of returning null
return parser.getValueAsString();
}
}
}
return null; // or throw exception if "not found" shouldn't return null
}
Test
String json = "{ \"A\": { \"B\": [ 5, { \"C\": \"D\" }, true ], \"E\": null, \"F\": 42, \"G\": false }}";
System.out.println("C: " + getFieldValue(json, "C")); // "D"
System.out.println("E: " + getFieldValue(json, "E")); // null (null-value)
System.out.println("F: " + getFieldValue(json, "F")); // "42"
System.out.println("G: " + getFieldValue(json, "G")); // "false"
System.out.println("H: " + getFieldValue(json, "H")); // null (not found)
System.out.println("B: " + getFieldValue(json, "B")); // null (not a value)

JsonPath
If using an external library is an option, then JsonPath might help.
Maven
<dependency>
<groupId>com.jayway.jsonpath</groupId>
<artifactId>json-path</artifactId>
<version>2.5.0</version>
</dependency>
StackOverflow Tag for JsonPath
Example usage as shared in the README
{
"store": {
"book": [
{
"category": "reference",
"author": "Nigel Rees",
"title": "Sayings of the Century"
},
{
"category": "fiction",
"author": "Evelyn Waugh",
"title": "Sword of Honour",
"price": 12.99
}
],
"bicycle": {
"color": "red",
"price": 19.95
}
},
"expensive": 10
}
To access all the author names
List<String> authors = JsonPath.read(json, "$..author");
System.out.println(authors);
Output
["Nigel Rees","Evelyn Waugh"]
To access all the prices (across both book and bicycle)
List<Double> prices = JsonPath.read(json, "$..price");
System.out.println(prices);
Output
[12.99, 19.95]
Note
Some missing keys can cause remapping extracted data across two fields difficult
Say fetching category and price from the above example will make it difficult to summarize a category to price mapping
List<String> categories = JsonPath.read(json, "$..category");
System.out.println(categories);
Output
["reference", "fiction"]
Based on the above example, price and category does not have correct 1-1 mapping

Not sure why it has to say not valid Json after replacing the values of paymentKey and Session key

I am trying to read a json file and after traversing to attrib`s called "paymentKey" and "Session key" and changing their values through JSONObject , the post operation failing.
When i checked the out json after performing above changes it seems that structure is bit unordered , changed and even got to learn that json is not an valid one.
This is bit annoying and not sure how to keep the json format in tag after replacing the attrib`s values.
Below is the Json used
{
"idempotentId": "133215472229",
"customerId": "12345",
"brandId": "ANCHOR",
"sellingChannel": "WEBOA",
"items": [
{
"lineItemId": 123,
"productId": "ANCHOR-WEBOA-640213214",
"price": 1.19,
"quantity": 1,
"modifierGroups": [],
"childItems": [],
"note": " Drink without snacks"
}
],
"fulfillment": {
"email": "12#gmail.com",
"phoneNumber": "+912222621",
"fulfillmentType": "PickUp",
"asap": true,
"pickupFirstName": "Kiran",
"pickupLastName": "Kumar",
"locationId": "33211111"
},
"payment": {
"paymentKey": "12222-444-555-2222-44444121e",
"sessionKey": "02f3waAjHJnVCTstOIu0jcSZfm_1HnGum1lZdsu6iDlLxxjO1FYsG9DHz9130ZzMMkjYY9j5w.7V8CijbmiPSo5ESDsq5hsQ.RpYSS5wkgoSSOMjktEyDTHZh1IPq0wNayp--DE3HE53uUgTEehCvHjSsUP5q8U2ZN1kZXbsufwm_mRCV8hLCrmWVTchhVUTJtmEpyYy142DtSp1ikXOVzGN5i9z_oP5e79QvgmU7_n1C5DeARFRagQClT87vUFBUfleSbLaRyH5v3wkU7ji9URUetcq1iAfS5-cNt6-uJaulFJc2y6uNdn0OtjIe74Hp5G7Gx54VYggduoqx5X1rsCssobfUSJUDLt_vVpz5BvhQM88EaysMAB6EcQHoOnZd_YWrz4IDAAZSwSBUFQAkypVmHo5pbvp64cTDrZE73EYkEwJLGf0dRmedMFe2HiU3DiCr97K3I3KuufxYM_eMRIcn739dntxTq4QePtFdqYGWBzXWQutvvqxWQPbNi7PG_-aauEOzlwJiXG94C8t7NGu0SjB8xHf11Z3orf5Ni4-fRKugY8VJNBl39hnb4-d-g47ut7iuiFDkDHJzlSgt9LFq__CxShG_.YkL2w7QEU85VHjpOj5urieCr4-G"
},
"subTotal": 100.19,
"tax": 4.19
}
Below is the snippet of the code
import org.json.JSONObject;
import org.json.JSONArray;
public JSONObject constructCreateOrderPayload( String freedomPayPaymentKey,String orderInit_SessionKey, String payloadFile) {
String filepath = System.getProperty("user.dir")+"/src/test/resources/JsonFiles/"+payloadFile;
try {
String jsonContents = new String((Files.readAllBytes(Paths.get(filepath))));
JSONObject jsonObject = new JSONObject(jsonContents);
JSONObject payment_obj = (JSONObject) jsonObject.get("payment");
payment_obj.put("paymentKey", freedomPayPaymentKey);
payment_obj.put("sessionKey",orderInit_SessionKey);
System.out.println("-------------------------------------------------------------------------------------------------------------------------------------------------------------------------");
System.out.println( " After Changes in JSON OBJECT : ");
System.out.println(jsonObject.toString());
System.out.println("");
System.out.println("-------------------------------------------------------------------------------------------------------------------------------------------------------------------------");
payload = jsonObject; // when i print the json boject the format is displaced hence when validated it says invalid json
} catch (IOException e) {
System.out.println("No file found in the path ");
e.printStackTrace();
}
return payload;
}
When I validated the Json after changes it shows as invalid with errors as shown in below snapshot
I tried a lot but no success, can somebody please look in to issue and advise me where I am going wrong or provide an solution this issue.

JSON in unordered, When you print jsonObject before making the changes you will know the order of the JSON is changed, I have used the Jackson Databind libraries and below is a working code, Change it accordingly
String filepath = "C:\\Users\\wilfred\\Desktop\\Input.json";
try {
String jsonContents = new String((Files.readAllBytes(Paths.get(filepath))));
ObjectMapper mapper = new ObjectMapper();
JsonNode expected = mapper.readTree(jsonContents);
System.out.println("Before converting : " + expected.toString());
JsonNode payment_obj = (expected.get("payment"));
((ObjectNode) payment_obj).put("paymentKey", "Trial1");
((ObjectNode) payment_obj).put("sessionKey", "Trial2");
System.out.println("After converting : " + expected.toString());
} catch (IOException e) {
System.out.println("No file found in the path ");
e.printStackTrace();
}
}

My approach was correct. The only mistake was i had not supply/pass on the correct values to few of the JSon attributes and that resulted in error response.
Rectified as per requirements and was able to get results correctly, hence closing this.

Why do I always get JSON Exception?

This Is my first time with parsing JSON data. I am using the Google knowledge graph api. I got the api working and I can get the JSON result. This is Google 's sample return data for a sample query which I'm using now for testing.
{
"#context": {
"#vocab": "http://schema.org/",
"goog": "http://schema.googleapis.com/",
"resultScore": "goog:resultScore",
"detailedDescription": "goog:detailedDescription",
"EntitySearchResult": "goog:EntitySearchResult",
"kg": "http://g.co/kg"
},
"#type": "ItemList",
"itemListElement": [
{
"#type": "EntitySearchResult",
"result": {
"#id": "kg:/m/0dl567",
"name": "Taylor Swift",
"#type": [
"Thing",
"Person"
],
"description": "Singer-songwriter",
"image": {
"contentUrl": "https://t1.gstatic.com/images?q=tbn:ANd9GcQmVDAhjhWnN2OWys2ZMO3PGAhupp5tN2LwF_BJmiHgi19hf8Ku",
"url": "https://en.wikipedia.org/wiki/Taylor_Swift",
"license": "http://creativecommons.org/licenses/by-sa/2.0"
},
"detailedDescription": {
"articleBody": "Taylor Alison Swift is an American singer-songwriter and actress. Raised in Wyomissing, Pennsylvania, she moved to Nashville, Tennessee, at the age of 14 to pursue a career in country music. ",
"url": "http://en.wikipedia.org/wiki/Taylor_Swift",
"license": "https://en.wikipedia.org/wiki/Wikipedia:Text_of_Creative_Commons_Attribution-ShareAlike_3.0_Unported_License"
},
"url": "http://taylorswift.com/"
},
"resultScore": 896.576599
}
]
}
So I want to parse it so that I can get the name, description, detailed description. This is my code but I always seem to get the exception. Any ideas why?
try {
JSONObject object=new JSONObject(gggg);
JSONArray itemListElement = object.getJSONArray("itemListElement");
for(int i=0; i < itemListElement.length();i++){
JSONObject c = itemListElement.getJSONObject(i);
JSONObject results = c.getJSONObject("result");
String name = results.getString("name").toString();
String description = results.getString("description").toString();
String detailedDescription = results.getString("articleBody").toString();
gggg = "Name: "+name+"\n Description: "+description+"\n "+detailedDescription;
}
responseView.append(gggg);
} catch (JSONException e) {
Toast.makeText(MainActivity.this,gggg,Toast.LENGTH_LONG).show();
}
Also the string gggg contains the JSON data. I don't know why but I am always getting the exception. Please tell me what is the error in my code and how to repair it.
Thanks.

"Name: Taylor Swift Description: Singer-songwriter Taylor Alison
Swift is an American singer-songwriter and actress. Raised in
Wyomissing, Pennsylvania, she moved to Nashville, Tennessee, at the
age of 14 to pursue a career in country music. "
The problem is your String detailedDescription line.
You need to get the detailedDescription object before you retrieve the articleBody.
for(int i=0; i < itemListElement.length();i++){
JSONObject c = itemListElement.getJSONObject(i);
JSONObject results = c.getJSONObject("result");
String name = results.getString("name");
String description = results.getString("description");
JSONObject detailedDescription = results.getJSONObject("detailedDescription");
String articleBody = detailedDescription.getString("articleBody");
String x = "Name: "+name+"\n Description: "+description+"\n "+articleBody;
}
Also your .toString() method calls are redundant as you are calling .getString() on the JSON object.

With in the android json library it has a method called has element, which returns true or false. After successfully checking then access the element. The expection be caused by tring to access an element that isn't there.
Might be worth printing out after each time you create a new object to ensure that the objects are being created. It will also piont to where the expection is happening.

Update nested field in an index of ElasticSearch with Java API

I am using Java API for CRUD operation on elasticsearch.
I have an typewith a nested field and I want to update this field.
Here is my mapping for the type:
"enduser": {
"properties": {
"location": {
"type": "nested",
"properties":{
"point":{"type":"geo_point"}
}
}
}
}
Of course my enduser type will have other parameters.
Now I want to add this document in my nested field:
"location":{
"name": "London",
"point": "44.5, 5.2"
}
I was searching in documentation on how to update nested document but I couldn't find anything. For example I have in a string the previous JSON obect (let's call this string json). I tried the following code but seems to not working:
params.put("location", json);
client.prepareUpdate(index, ElasticSearchConstants.TYPE_END_USER,id).setScript("ctx._source.location = location").setScriptParams(params).execute().actionGet();
I have got a parsing error from elasticsearch. Anyone knows what I am doing wrong ?

You don't need the script, just update it.
UpdateRequestBuilder br = client.prepareUpdate("index", "enduser", "1");
br.setDoc("{\"location\":{ \"name\": \"london\", \"point\": \"44.5,5.2\" }}".getBytes());
br.execute();

I tried to recreate your situation and i solved it by using an other way the .setScript method.
Your updating request now would looks like :
client.prepareUpdate(index, ElasticSearchConstants.TYPE_END_USER,id).setScript("ctx._source.location =" + json).execute().actionGet()
Hope it will help you.

I am not sure which ES version you were using, but the below solution worked perfectly for me on 2.2.0. I had to store information about named entities for news articles. I guess if you wish to have multiple locations in your case, it would also suit you.
This is the nested object I wanted to update:
"entities" : [
{
"disambiguated" : {
"entitySubTypes" : [],
"disambiguatedName" : "NameX"
},
"frequency" : 1,
"entityType" : "Organization",
"quotations" : ["...", "..."],
"name" : "entityX"
},
{
"disambiguated" : {
"entitySubType" : ["a", "b" ],
"disambiguatedName" : "NameQ"
},
"frequency" : 5,
"entityType" : "secondTypeTest",
"quotations" : [ "...", "..."],
"name" : "entityY"
}
],
and this is the code:
UpdateRequest updateRequest = new UpdateRequest();
updateRequest.index(indexName);
updateRequest.type(mappingName);
updateRequest.id(url); // docID is a url
XContentBuilder jb = XContentFactory.jsonBuilder();
jb.startObject(); // article
jb.startArray("entities"); // multiple entities
for ( /*each namedEntity*/) {
jb.startObject() // entity
.field("name", name)
.field("frequency",n)
.field("entityType", entityType)
.startObject("disambiguated") // disambiguation
.field("disambiguatedName", disambiguatedNameStr)
.field("entitySubTypes", entitySubTypeArray) // multi value field
.endObject() // disambiguation
.field("quotations", quotationsArray) // multi value field
.endObject(); // entity
}
jb.endArray(); // array of nested objects
b.endObject(); // article
updateRequest.doc(jb);
Blblblblblblbl's answer couldn't work for me atm, because scripts are not enabled in our server. I didn't try Bask's answer yet - Alcanzar's gave me a hard time, because I supposedly couldn't formulate the json string correctly that setDoc receives. I was constantly getting errors that either I am using objects instead of fields or vice versa. I also tried wrapping the json string with doc{} as indicated here, but I didn't manage to make it work. As you mentioned it is difficult to understand how to formulate a curl statement at ES's java API.

A simple way to update the arraylist and object value using Java API.
UpdateResponse update = client.prepareUpdate("indexname","type",""+id)
.addScriptParam("param1", arrayvalue)
.addScriptParam("param2", objectvalue)
.setScript("ctx._source.field1=param1;ctx._source.field2=param2").execute()
.actionGet();
arrayvalue-[
{
"text": "stackoverflow",
"datetime": "2010-07-27T05:41:52.763Z",
"obj1": {
"id": 1,
"email": "sa#gmail.com",
"name": "bass"
},
"id": 1,
}
object value -
"obj1": {
"id": 1,
"email": "sa#gmail.com",
"name": "bass"
}

Converting Nested Json files to CSV in java

{
"Employee": [
{
"empMID": "mock:1",
"comments": [],
"col1": "something",
"contact": [{"address":"2400 waterview", "freetext":true}
],
"gender": "male"
},
{
"empMID": "mock:2",
"comments": [],
"col1": "something",
"contact": [{"address":"2200 waterview", "freetext":true}
],
"gender": "female"
}
],
"cola": false,
"colb": false
}
This is how my Json file looks .I m required to convert this json to a csv .(I m trying to convert a multi-dimesional data to 2d).I m using gson for my purpose.I cannot use gson.fromgson() function to object map with a template because it should be generic .
I know we can use CDL to convert jsonarray to csv format but It wont work in my case .
my csv format looks like
Employee*
empMID,comment.$,contact.address,contact.freetext,gender
mock:1,,2400 waterview,TRUE,male
mock:123,,2200 waterview,TRUE,female
colA#
TRUE
colB#
FALSE
I tried using google-GSON api to convert to this format .But I m not able to convert to this format .I have used * to represent its a json array and # to represent its a primitive type and contact.address to represent nested array inside another json array .I having problem relating this nested structure .I m able to traverse everything recursively like a column. Thanks in advance
public static void main(String[] args) throws IOException{
BufferedReader reader=null;
StringBuilder content=null;
String result=null;
reader = new BufferedReader(new FileReader("temp.json"));
String line = null;
content= new StringBuilder();
while ((line = reader.readLine()) != null) {
content.append(line);
}
reader.close();
result= content.toString();
JsonElement jelement = new JsonParser().parse(result);
printJsonRecursive(jelement);
}
public static void printJsonRecursive(JsonElement jelement){
if(jelement.isJsonPrimitive()){
System.out.println(jelement.getAsString());
return;
}
if(jelement.isJsonArray()){
JsonArray jarray= jelement.getAsJsonArray();
for(int i=0;i<jarray.size();i++){
JsonElement element= jarray.get(i);
printJsonRecursive(element);
}
return;
}
JsonObject jobject= jelement.getAsJsonObject();
Set<Entry<String, JsonElement>> set= jobject.entrySet();
for (Entry<String, JsonElement> s : set) {
printJsonRecursive(s.getValue());
}
}
}

You can achieve this thru reflection if you have a object mapped to the json.
use gson/jackson to convert json to java object
append fields using reflection by iterating the class and get any field you interested in.
append value with reflection by getting value from the target object.
More detail look at my blog post below:
vcfvct.wordpress.com/2015/06/30/converting-nested-json-files-to-csv-in-java-with-reflection/

You are not printing the key. This should fix it.
for (Entry<String, JsonElement> s : set) {
System.out.println(s.getKey()); //Added
printJsonRecursive(s.getValue());
}
You can take care of \ns from here.
EDIT
If you want to print the keys just once for repeating json objects, create a Java bean to hold the data and populate it during your recursion. Once the bean is complete, add a method there to print all the data in the format you want (printing keys only once and so on).

You can use the library json2flat for converting your JSON to CSV.
This library doesn't require any POJO's. It simply takes your JSON as string and returns a 2D representation of it in the format of List<Object[]>.
For example for the JSON:
{
"Employee": [
{
"empMID": "mock:1",
"comments": [],
"col1": "something",
"contact": [{"address":"2400 waterview", "freetext":true}
],
"gender": "male"
},
{
"empMID": "mock:2",
"comments": [],
"col1": "something",
"contact": [{"address":"2200 waterview", "freetext":true}
],
"gender": "female"
}
],
"cola": false,
"colb": false
}
It gives an output:
/cola,/colb,/Employee/empMID,/Employee/col1,/Employee/gender,/Employee/contact/address,/Employee/contact/freetext
,,"mock:1","something",,"2400 waterview",true
,,"mock:2","something",,"2200 waterview",true
false,false,,,,,

/**
* Get separated comlumns used a separator (comma, semi column, tab).
*
* #param headers The CSV headers
* #param map Map of key-value pairs contains the header and the value
*
* #return a string composed of columns separated by a specific separator.
*/
private static String getSeperatedColumns(Set<String> headers, Map<String, String> map, String separator) {
List<String> items = new ArrayList<String>();
for (String header : headers) {
String value = map.get(header) == null ? "" : map.get(header).replaceAll("[\\,\\;\\r\\n\\t\\s]+", " ");
items.add(value);
}
return StringUtils.join(items.toArray(), separator);
}