We Keep Coding

Comparing JSON with arrays

I am trying to compare 2 JSON files, they have arrays with duplicated values.
My first JSON Object has an array like this:
"categories": [
"May",
"Apr",
"Mar"
]
My second JSON object has an array like this:
"categories": [
"May",
"May",
"Apr",
"Apr",
"Mar",
"Mar"
]
I am comparing the JSON using flat maps that can be found in this link comparing JSONs using guava
Here is part of my code:
private String smartJSONsCompare(JSONObject leftJson, JSONObject rightJson) {
String result = "</br>";
Gson gson = new Gson();
Type type = new TypeToken<Map<String, Object>>(){}.getType();
Map<String, Object> leftMap = gson.fromJson(leftJson.toString(), type);
Map<String, Object> rightMap = gson.fromJson(rightJson.toString(), type);
Map<String, Object> leftFlatMap = FlatMapUtil.flatten(leftMap);
Map<String, Object> rightFlatMap = FlatMapUtil.flatten(rightMap);
MapDifference<String, Object> difference = Maps.difference(leftFlatMap, rightFlatMap);
StringBuilder SB = new StringBuilder("</br>");
SB.append("Entries only on LEFT: </br>");
difference.entriesOnlyOnLeft().forEach((key, value) -> SB.append(key + ": " + value + "</br>"));
SB.append("Entries only on RIGHT: </br>");
difference.entriesOnlyOnRight().forEach((key, value) -> SB.append(key + ": " + value + "</br>"));
SB.append("Entries full difference : </br>");
difference.entriesDiffering().forEach((key, value) -> SB.append(key + ": " + value + "</br>"));
result = SB.toString();
return result;
}
I wish to be able to present the difference in a more "smart" way. In other words: showing all the objects / arrays in the JONSs that don't match. What is missing or what was added to the compared JSON.
For the "categories" array my code returns a message that their is a mismatch, but doesn't state the difference in an elegant way.
What can I do?

I have change a bit in your solution to get the wanted result.
I would do my difference check in List, therefore I will create method to change JSON to list of strings based on your code:
private static List<String> jsonToList(String json){
List<String> list = new ArrayList<>();
Gson gson = new Gson();
Type type = new TypeToken<Map<String, Object>>(){}.getType();
Map<String, Object> jsonMap = gson.fromJson(json, type);
Map<String, Object> flatten = FlatMapUtil.flatten(jsonMap);
flatten.forEach((k, v) -> list.add(v.toString()));
return list;
}
Update
When I answered the question I did things a bit fast, the jsonToList was based on your code. As it is right now it is over complicated to what you are asking for. I have therefore made much lighter version using the following method in stead:
private static List<String> jsonToList(String json) {
JSONObject response = new JSONObject(json);
List<String> list = new ArrayList<>();
JSONArray jsonArray = response.getJSONArray("categories");
if (jsonArray != null) {
for (int i = 0; i < jsonArray.length(); i++) {
list.add(jsonArray.get(i).toString());
}
}
return list;
}
That said, now you have two choices and it is up to you to find out which one fits best to your needs and take it from here.
End of Update
for this example I have made 3 test examples
String main = "{\"categories\":[\"May\",\"Apr\",\"Mar\"]}";
String json1 = "{\"categories\":[\"May\",\"May\",\"Apr\",\"Apr\",\"Mar\",\"Mar\"]}";
String json2 = "{\"categories\":[\"May\",\"Apr\",\"Apr\",\"Mar\",\"Mar\",\"Mar\"]}";
String json3 = "{\"categories\":[\"May\",\"Apr\",\"Mar\",\"Mar\"]}";
in my second step I will create a
List<String> mainList = jsonToList(main);
List<String> list1 = jsonToList(json1);
so far so good. Now I make a method to take the extra difference of the 2 list, that mean as you requested in your comments, we take only all values that are duplicated more than once and return them in list. In this method I used hashmap only count duplicates and than take the all that is repeated more than 1 time:
private static List<String> diffList(List<String> mainList, List<String> secondList){
List<String> list = new ArrayList<String>();
Map<String, Integer> wordCount = new HashMap<>();
for(String word: secondList) {
if(mainList.contains(word)) {
Integer count = wordCount.get(word);
wordCount.put(word, (count == null) ? 1 : count + 1);
if(wordCount.get(word) > 1){
list.add(word);
}
}
}
return list;
}
Finally I would test all cases, for instance for list1:
List<String> diff1 = diffList(mainList, list1);
for (String s : diff1) {
System.out.println(s);
}
The output will be
May
Apr
Mar
for list2
Apr
Mar
Mar
And for list3
Mar
Now I will separate view method from the your method and create some thing like, just to make my code more clear and easy to work with:
private static String viewResult(List<String> list1, List<String> list2, List<String> duplicate){
String result;
StringBuilder SB = new StringBuilder("</br>");
SB.append("Entries only on LEFT: </br>");
list1.forEach(e -> SB.append(e + "</br>"));
SB.append("Entries only on RIGHT: </br>");
list2.forEach(e -> SB.append(e + "</br>"));
SB.append("Entries full difference : </br>");
duplicate.forEach(e -> SB.append(e + "</br>"));
result = SB.toString();
return result;
}
So if we put all this code together I will be some thing like this, and the following code is to demonstrate how things works, but from here you can take it to the next level in your code:
public static void main(String[] args) {
String main = "{\"categories\":[\"May\",\"Apr\",\"Mar\"]}";
String json1 = "{\"categories\":[\"May\",\"May\",\"Apr\",\"Apr\",\"Mar\",\"Mar\"]}";
String json2 = "{\"categories\":[\"May\",\"Apr\",\"Apr\",\"Mar\",\"Mar\",\"Mar\"]}";
String json3 = "{\"categories\":[\"May\",\"Apr\",\"Mar\",\"Mar\"]}";
List<String> mainList = jsonToList(main);
List<String> list1 = jsonToList(json1);
List<String> diff1 = diffList(mainList, list1);
for (String s : diff1) {
System.out.println(s);
}
String view = viewResult(mainList, list1, diff1);
}
private static List<String> jsonToList(String json){
List<String> list = new ArrayList<String>();
Gson gson = new Gson();
Type type = new TypeToken<Map<String, Object>>(){}.getType();
Map<String, Object> jsonMap = gson.fromJson(json, type);
Map<String, Object> flatten = FlatMapUtil.flatten(jsonMap);
flatten.forEach((k, v) -> list.add(v.toString()));
return list;
}
private static List<String> diffList(List<String> mainList, List<String> secondList){
List<String> list = new ArrayList<String>();
Map<String, Integer> wordCount = new HashMap<>();
for(String word: secondList) {
if(mainList.contains(word)) {
Integer count = wordCount.get(word);
wordCount.put(word, (count == null) ? 1 : count + 1);
if(wordCount.get(word) > 1){
list.add(word);
}
}
}
return list;
}
private static String viewResult(List<String> list1, List<String> list2, List<String> duplicate){
String result;
StringBuilder SB = new StringBuilder("</br>");
SB.append("Entries only on LEFT: </br>");
list1.forEach(e -> SB.append(e + "</br>"));
SB.append("Entries only on RIGHT: </br>");
list2.forEach(e -> SB.append(e + "</br>"));
SB.append("Entries full difference : </br>");
duplicate.forEach(e -> SB.append(e + "</br>"));
result = SB.toString();
return result;
}

If you want something more generic with a good diff you could utilize AssertJ here.
Its usually used for Testing, but the diff looks really good and you can also use it in normal code.
Example:
Expecting:
<["Mai", "Apr", "Mar"]>
to contain exactly in any order:
<["May", "Apr", "Mar", "Mar"]>
elements not found:
<["May", "Mar"]>
and elements not expected:
<["Mai"]>
Can be created by:
[...]
import org.assertj.core.api.Assertions;
public class JsonTest {
final static String arr = " [\n"+
" \"Mai\",\n"+
" \"Apr\",\n"+
" \"Mar\"\n"+
" ]";
final static String arr2 = " [\n"+
" \"May\",\n"+
" \"Apr\",\n"+
" \"Mar\",\n"+
" \"Mar\"\n"+
" ]";
public static void main(String[] args){
System.out.println(smartJSONsCompare(arr,arr2));
}
private static String smartJSONsCompare(String leftJson, String rightJson) {
Gson gson = new Gson();
Type type = new TypeToken<List<String>>(){}.getType();
List<String> left = gson.fromJson(leftJson, type);
List<String> right = gson.fromJson(rightJson, type);
try{
Assertions.assertThat(left).containsExactlyInAnyOrderElementsOf(right);
}catch(AssertionError ae){
return ae.getMessage();
}
return "Matched";
}
}
I added the dependencies in gradle with:
dependencies {
compile("org.assertj:assertj-core:3.11.1")
}

If you want to create a patch between your two JSON Objects have a look at json-patch.
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.github.fge.jsonpatch.diff.JsonDiff;
import java.io.IOException;
public class JsonPatchTest {
public static void main(String[] args) throws IOException {
String jsonFirst = "{\"categories\":[\"May\",\"Apr\",\"Mar\"]}";
String jsonSecond = "{\"categories\":[\"May\",\"May\",\"Apr\",\"Apr\",\"Mar\",\"Mar\"]}";
ObjectMapper mapper = new ObjectMapper();
JsonNode jsonNodeFirst = mapper.readTree(jsonFirst);
JsonNode jsonNodeSecond = mapper.readTree(jsonSecond);
JsonNode patchNode = JsonDiff.asJson(jsonNodeFirst, jsonNodeSecond);
System.out.println(mapper.writerWithDefaultPrettyPrinter().writeValueAsString(patchNode));
}
}
Would produce the following output for your scenario:
[ {
"op" : "replace",
"path" : "/categories/1",
"value" : "May"
}, {
"op" : "replace",
"path" : "/categories/2",
"value" : "Apr"
}, {
"op" : "add",
"path" : "/categories/-",
"value" : "Apr"
}, {
"op" : "add",
"path" : "/categories/-",
"value" : "Mar"
}, {
"op" : "add",
"path" : "/categories/-",
"value" : "Mar"
} ]

I believe you should handle json arrays on your own in order to present their difference in a more "smart" way. Here is a library which contains CollectionUtils class with disjunction method.
MapDifference<String, Object> difference = Maps.difference(leftMap, rightMap);
difference.entriesDiffering().forEach((key, value) -> {
Object left = value.leftValue();
Object right = value.rightValue();
if (left instanceof Iterable && right instanceof Iterable) {
Collection<?> diff = CollectionUtils.disjunction((Iterable<?>) right, (Iterable<?>) left);
System.out.println(key + " -> " + diff);
}
...
});

This code work for me (2 years ago) on production.
public class App {
private final Gson GSON = new GsonBuilder().create();
public boolean isDifference(final String path, Map<String, Object> oldData, Map<String, Object> newData) {
MapDifference<String, Object> difference = Maps.difference(oldData, newData);
difference.entriesOnlyOnLeft().forEach((key, value) -> {
publishChange(Action.REMOVE, path, key, value);
});
difference.entriesOnlyOnRight().forEach((key, value) -> {
publishChange(Action.ADD, path, key, value);
});
difference.entriesDiffering().forEach((key, value) -> {
if (value.rightValue() instanceof Map && value.leftValue() instanceof Map) {
if (!path.isEmpty()) {
key = path.concat("-").concat(key);
}
isDifference(key, (Map) value.leftValue(), (Map) value.rightValue());
} else {
publishChange(Action.MODIFY, path, key, value);
}
});
return !difference.areEqual();
}
public void publishChange(Action action, String path, String key, Object value) {
if (value instanceof MapDifference.ValueDifference) {
value = ((MapDifference.ValueDifference) value).rightValue();
}
JsonElement jsonValue = GSON.toJsonTree(value);
String event = createEvent(action, path, key, jsonValue);
System.out.println("Differrence: " + event);
}
public String createEvent(Action action, String paths, String key, JsonElement value) {
JsonObject root = new JsonObject();
JsonArray arrPaths = new JsonArray();
for (String path : paths.split("-")) {
arrPaths.add(path);
}
root.addProperty("action", action.toString());
root.add("paths", arrPaths);
JsonObject data = new JsonObject();
data.addProperty("key", key);
data.add("value", value);
root.add("data", data);
return root.toString();
}
public static enum Action {
ADD, REMOVE, MODIFY
}}
Test/ Example:
public class AppTest {
#Test
public void testAppHasAGreeting() {
App classUnderTest = new App();
Gson gson = new GsonBuilder()
.setPrettyPrinting()
.create();
// JsonOld: {"a":1,"b":1,"c":true,"array":[1,2,3],"object":{"arrayKey":["a","b","c","d"]}}
String jsonOld = "{\"a\":1,\"b\":1,\"c\":true,\"array\":[1,2,3],\"object\":{\"arrayKey\":[\"a\",\"b\",\"c\",\"d\"]}}";
// JsonNew: {"a":2,"b":1,"array":[1,2,3,2],"another":{"d":false,"e":["a","b","c"]},"object":{"booleanKey":true,"arrayKey":["a","b","c"]}}
String jsonNew = "{\"a\":2,\"b\":1,\"array\":[1,2,3,2],\"another\":{\"d\":false,\"e\":[\"a\",\"b\",\"c\"]},\"object\":{\"booleanKey\":true,\"arrayKey\":[\"a\",\"b\",\"c\"]}}";
Type mapType = new TypeToken<Map<String, Object>>() {
}.getType();
Map<String, Object> jsonOldAsMap = gson.fromJson(jsonOld, mapType);
Map<String, Object> jsonNewAsMap = gson.fromJson(jsonNew, mapType);
System.out.println("Old Json: " + gson.toJson(jsonOldAsMap));
System.out.println("New Json: " + gson.toJson(jsonNewAsMap));
System.out.println("========== Result ==========");
// When
boolean diff = classUnderTest.isDifference("", jsonOldAsMap, jsonNewAsMap);
// Then
assertTrue(diff);
}}
The result will print like this:
Differrence: {"action":"REMOVE","paths":[""],"data":{"key":"c","value":true}}
Differrence: {"action":"ADD","paths":[""],"data":{"key":"another","value":{"d":false,"e":["a","b","c"]}}}
Differrence: {"action":"MODIFY","paths":[""],"data":{"key":"a","value":2.0}}
Differrence: {"action":"MODIFY","paths":[""],"data":{"key":"array","value":[1.0,2.0,3.0,2.0]}}
Differrence: {"action":"ADD","paths":["object"],"data":{"key":"booleanKey","value":true}}
Differrence: {"action":"MODIFY","paths":["object"],"data":{"key":"arrayKey","value":["a","b","c"]}}
The code available here: https://github.com/liemle3893/compare-json

Java and XSS: How to html escape a JSON string to protect against XSS?

In Java, we've got some code that takes a complex java object and serializes it to json. It then writes that json directly to the markup of a page, in a script tag, assigning it to a variable.
// Get object as JSON using Jackson
ObjectWriter jsonWriter = new ObjectMapper().writer().withDefaultPrettyPrinter();
String json = jsonWriter.writeValueAsString(complexObject);
// Write JSON out to page, and assign it to a javascript variable.
Writer out = environment.getOut();
out.write("var data = " + json);
The complex object can have end user content in it, which could open us up to XSS attacks.
How can I get a json version of the complex java object that has each json attribute HTML escaped, to protect against XSS injection?
I've read the OWASP XSS Guide and the best I've come up with so far is this, which HTML escapes the entire JSON string, then undoes the quotes, so it can be assigned to a variable in javascript. I'm sure there are better ways to do this, but this seems to work. Any suggestions?
private String objectToHtmlEscapedJson(Object value) {
try {
String result = jsonWriter.writeValueAsString(value);
result = StringEscapeUtils.escapeHtml(result);
result = result.replace(""", "\"");
return result;
} catch (JsonProcessingException e) {
return "null";
}
}

A possible approach could be to iterate over the object entries and individually escape each key and value once the node is constructed by your chosen library.
Following my comment above, I've implemented a simple recursive solution using both Jackson (from your question) and GSON, a different library where objects are slightly easier to construct and the code is more readable. The escaping mechanism used is the OWASP Java Encoder:
Jackson
private static JsonNode clean(JsonNode node) {
if(node.isValueNode()) { // Base case - we have a Number, Boolean or String
if(JsonNodeType.STRING == node.getNodeType()) {
// Escape all String values
return JsonNodeFactory.instance.textNode(Encode.forHtml(node.asText()));
} else {
return node;
}
} else { // Recursive case - iterate over JSON object entries
ObjectNode clean = JsonNodeFactory.instance.objectNode();
for (Iterator<Map.Entry<String, JsonNode>> it = node.fields(); it.hasNext(); ) {
Map.Entry<String, JsonNode> entry = it.next();
// Encode the key right away and encode the value recursively
clean.set(Encode.forHtml(entry.getKey()), clean(entry.getValue()));
}
return clean;
}
}
GSON
private static JsonElement clean(JsonElement elem) {
if (elem.isJsonPrimitive()) { // Base case - we have a Number, Boolean or String
JsonPrimitive primitive = elem.getAsJsonPrimitive();
if(primitive.isString()) {
// Escape all String values
return new JsonPrimitive(Encode.forHtml(primitive.getAsString()));
} else {
return primitive;
}
} else if (elem.isJsonArray()) { // We have an array - GSON requires handling this separately
JsonArray cleanArray = new JsonArray();
for(JsonElement arrayElement: elem.getAsJsonArray()) {
cleanArray.add(clean(arrayElement));
}
return cleanArray;
} else { // Recursive case - iterate over JSON object entries
JsonObject obj = elem.getAsJsonObject();
JsonObject clean = new JsonObject();
for(Map.Entry<String, JsonElement> entry : obj.entrySet()) {
// Encode the key right away and encode the value recursively
clean.add(Encode.forHtml(entry.getKey()), clean(entry.getValue()));
}
return clean;
}
}
Sample input (both libraries):
{
"nested": {
"<html>": "<script>(function(){alert('xss1')})();</script>"
},
"xss": "<script>(function(){alert('xss2')})();</script>"
}
Sample output (both libraries):
{
"nested": {
"<html>": "<script>(function(){alert('xss1')})();</script>"
},
"xss": "<script>(function(){alert('xss2')})();</script>"
}

I think Paul Benn's answer is the best approach overall, but if you don't want to iterate over the json nodes, you could consider using Encode.forHtmlContent, which doesn't escape quotes. I feel this is probably safe as I can't think of how introducing an additional quote into a quoted string could cause an exploit. I'll leave it to the reader to review the docs and decide for themselves!
ivy.xml
<dependency org="org.owasp.encoder" name="encoder" rev="1.2.1"/>
and some code to do the html encoding
private String objectToJson(Object value)
{
String result;
try
{
result = jsonWriter.writeValueAsString(value);
return Encode.forHtmlContent(result);
}
catch (JsonProcessingException e)
{
return "null";
}
}

Updating Paul Benn's answer of the Gson version to include json value being an array
private static JsonElement clean(JsonElement elem) {
if(elem.isJsonPrimitive()) { // Base case - we have a Number, Boolean or String
JsonPrimitive primitive = elem.getAsJsonPrimitive();
if(primitive.isString()) {
// Escape all String values
return new JsonPrimitive(Encode.forHtml(primitive.getAsString()));
} else {
return primitive;
}
} else if( elem.isJsonArray() ) { // If the object is an array "cars": ["toyota", "nissan", "bmw"]
JsonArray jsonA = elem.getAsJsonArray();
JsonArray cleanedNewArray = new JsonArray();
for(JsonElement jsonAE: jsonA) {
cleanedNewArray.add(clean(jsonAE));
}
return cleanedNewArray;
} else { // Recursive case - iterate over JSON object entries
JsonObject obj = elem.getAsJsonObject();
JsonObject clean = new JsonObject();
for(Map.Entry<String, JsonElement> entry : obj.entrySet()) {
// Encode the key right away and encode the value recursively
clean.add(Encode.forHtml(entry.getKey()), clean(entry.getValue()));
}
return clean;
}
}

Adding a version of JKRo using Jackson with Esapi.
private JsonNode clean(JsonNode node, ObjectMapper mapper) {
if(node.isValueNode()) { // Base case - we have a Number, Boolean or String
if(JsonNodeType.STRING == node.getNodeType()) {
// Escape all String values
return JsonNodeFactory.instance.textNode(ESAPI.encoder().encodeForHTML(node.asText()));
} else {
return node;
}
} else if(node.isArray()) { // If the object is an array "cars": ["toyota", "nissan", "bmw"]
ArrayNode cleanedNewArray = mapper.createArrayNode();
for (final JsonNode objNode : node) {
cleanedNewArray.add(clean(objNode, mapper));
}
return cleanedNewArray;
} else { // Recursive case - iterate over JSON object entries
ObjectNode clean = JsonNodeFactory.instance.objectNode();
for (Iterator<Map.Entry<String, JsonNode>> it = node.fields(); it.hasNext(); ) {
Map.Entry<String, JsonNode> entry = it.next();
// Encode the key right away and encode the value recursively
clean.set(ESAPI.encoder().encodeForHTML(entry.getKey()), clean(entry.getValue(), mapper));
}
return clean;
}
}
Request Body:
{
"param1": "<input class='btn btn-default' value='0' placeholder='Ingrese sus datos'></input>",
"param3": [
{
"nombre" : "<input class='btn btn-default' value='0' placeholder='Ingrese sus datos'></input>",
"apellido": "<script>alert('Hola mundex');</script>"
},
{
"param4": {
"nombre" : "<input class='btn btn-default' value='0' placeholder='Ingrese sus datos'></input>",
"apellido": "<script>alert('Hola mundex');</script>"
}
}],
"param2": "alert('Hola')"
}
Response Body:
{
"param1": "<input class='btn btn-default' value='0' placeholder='Ingrese sus datos'></input>",
"param3": [
{
"nombre": "<input class='btn btn-default' value='0' placeholder='Ingrese sus datos'></input>",
"apellido": "<script>alert('Hola mundex');</script>"
},
{
"param4": {
"nombre": "<input class='btn btn-default' value='0' placeholder='Ingrese sus datos'></input>",
"apellido": "<script>alert('Hola mundex');</script>"
}
}
],
"param2": "alert('Hola')"
}

How to parse a JSON string using ggson to get field values

I have a sample JSON as below. I need to get the individual fields like ASIdentifer and ExternalIdentifer. I have stored this JSON data in a string.
Using GoogleJson as the module(ggson)
JSON data:
{
"DeviceCommon": {
"ASIdentifier": "123",
"DatadeliveyMechanism": "notify",
"MobileOriginatorCallbackReference": {
"url": "http://application.example.com/inbound/notifications/modatanotification/"
},
"AccessiblityCallbackReference": {
"url": "http://application.example.com/inbound/notifications/accessibilitystatusnotification"
}
},
"DeviceList": [{
"ExternalIdentifer": "123456#mydomain.com",
"msisdn": "123456",
"senderName": "Device1",
"MobileOriginatorCallbackReference": {
"notifyURL": "http://application.example.com/inbound/notifications/modatanotification/"
},
"ConfigurationResultCallbackReference": {
"notifyURL": "http://application.example.com/inbound/notifications/configurationResult"
},
"ASreferenceID": "AS000001",
"NIDDduration": "1d"
}]
}
I created the POJO classes and parsed the data using below code
data = new Gson().fromJson(new FileReader("/home/raj/apache-tomcat-8.0.3/webapps/file.json"), Data.class);
System.out.println(data);
Output:
Data{
deviceCommon=DeviceCommon{
asIdentifier='123'
datadeliveyMechanism='notify'
mobileOriginatorCallbackReference=http://application.example.com/inbound/notifications/modatanotification/
accessiblityCallbackReference=http://application.example.com/inbound/notifications/accessibilitystatusnotification
}
deviceList=[DeviceListEntry{
externalIdentifer='123456#mydomain.com'
msisdn='123456'
senderName='Device1'
mobileOriginatorCallbackReference=http://application.example.com/inbound/notifications/modatanotification/
configurationResultCallbackReference=http://application.example.com/inbound/notifications/configurationResult
asReferenceID='AS000001'
nidDduration='1d'
}]
}
String jsonInString = gson.toJson(data);
System.out.println("String is"+ jsonInString);
Output:
String is{"DeviceCommon":{"ASIdentifier":"123","DatadeliveyMechanism":"notify","MobileOriginatorCallbackReference":{"url":"http://application.example.com/inbound/notifications/modatanotification/"},"AccessiblityCallbackReference":{"url":"http://application.example.com/inbound/notifications/accessibilitystatusnotification"}},"DeviceList":[{"ExternalIdentifer":"123456#mydomain.com","msisdn":"123456","senderName":"Device1","MobileOriginatorCallbackReference":{"notifyURL":"http://application.example.com/inbound/notifications/modatanotification/"},"ConfigurationResultCallbackReference":{"notifyURL":"http://application.example.com/inbound/notifications/configurationResult"},"ASreferenceID":"AS000001","NIDDduration":"1d"}]}
I need to parse this JSON string to get individual fields like ExternalIdentifier and ASIdentifier.
I tried something like this but it is not working.
JsonObject jobj = new Gson().fromJson(jsonInString, JsonObject.class);
String result = jobj.get("ASIdentifier").toString();
System.out.println("value is"+ result);
Note: ExternalIdentifier is within the array, so I need to loop through the array to find it.
Can you please tell me what I'm doing wrong?

Possible solution:
String result = jobj.get("DeviceCommon").getAsJsonObject().get("ASIdentifier").getAsString();
System.out.println("ASIdentifier: "+ result);
JsonArray jsonArray = jobj.get("DeviceList").getAsJsonArray();
for (JsonElement device : jsonArray ) {
result = device.getAsJsonObject().get("ExternalIdentifer").getAsString();
System.out.println("ExternalIdentifer: "+ result);
}
Output:
ASIdentifier: 123
ExternalIdentifer: 123456#mydomain.com

public static void printJson(JsonElement jsonElement,String key) {
// Check whether jsonElement is JsonObject or not
if (jsonElement.isJsonObject()) {
Set<Entry<String, JsonElement>> ens = ((JsonObject) jsonElement).entrySet();
if (ens != null) {
// Iterate JSON Elements with Key values
for (Entry<String, JsonElement> en : ens) {
// System.out.println("##key is"+en.getKey() + " : ");
printJson(en.getValue(), en.getKey());
// System.out.println(en.getValue().getAsString());
// System.out.println(jsonElement.getAsString());
}
}
}
// Check whether jsonElement is Primitive or not
else if (jsonElement.isJsonPrimitive()) {
// print value as String
System.out.println("###key is"+key);
System.out.println("### value is"+jsonElement.getAsString());
}
else if (jsonElement.isJsonArray()) {
JsonArray jarr = jsonElement.getAsJsonArray();
// Iterate JSON Array to JSON Elements
System.out.println("\n###Array size is"+ jarr.size());
for (JsonElement je : jarr) {
printJson(je,key);
}
}
}

How do I get a list of all JSON paths to values from a JSON String?

My goal is to read a JSON file and understand the location of all the values, so that when I encounter that same JSON, I can easily read all the values. I am looking for a way to return a list containing all of the paths to each data value, in Jayway JsonPath format.
Example JSON:
{
"shopper": {
"Id": "4973860941232342",
"Context": {
"CollapseOrderItems": false,
"IsTest": false
}
},
"SelfIdentifiersData": {
"SelfIdentifierData": [
{
"SelfIdentifierType": {
"SelfIdentifierType": "111"
}
},
{
"SelfIdentifierType": {
"SelfIdentifierType": "2222"
}
}
]
}
}
Ideally I would like to take that JSON as a String and do something like this:
String json = "{'shopper': {'Id': '4973860941232342', 'Context': {'CollapseOrderItems': false, 'IsTest': false } }, 'SelfIdentifiersData': {'SelfIdentifierData': [{'SelfIdentifierType': {'SelfIdentifierType': '111'} }, {'SelfIdentifierType': {'SelfIdentifierType': '2222'} } ] } }";
Configuration conf = Configuration.defaultConfiguration();
List<String> jsonPaths = JsonPath.using(conf).parse(json).read("$");
for (String path : jsonPaths) {
System.out.println(path);
}
This code would print this, which is the location of all values in the JSON:
$.shopper.Id
$.shopper.Context.CollapseOrderItems
$.shopper.Context.IsTest
$.SelfIdentifiersData[0].SelfIdentifierData.SelfIdentifierType.SelfIdentifierType
$.SelfIdentifiersData[1].SelfIdentifierData.SelfIdentifierType.SelfIdentifierType
Then ideally, I would be able to take that list and parse the same JSON object to get each value present.
//after list is created
Object document = Configuration.defaultConfiguration().jsonProvider().parse(json);
for (String path : jsonPaths) {
Object value = JsonPath.read(document, path);
//do something
}
I am aware that I could get a Map that is a representation of the JSON file, but I am not sure that provides the same ease of access to retrieve all the values. If there is a easy way to do with JSONPath, that would be great, otherwise any other approaches are welcome.

I came up with a solution, sharing in case anyone else is looking for the same thing:
public class JsonParser {
private List<String> pathList;
private String json;
public JsonParser(String json) {
this.json = json;
this.pathList = new ArrayList<String>();
setJsonPaths(json);
}
public List<String> getPathList() {
return this.pathList;
}
private void setJsonPaths(String json) {
this.pathList = new ArrayList<String>();
JSONObject object = new JSONObject(json);
String jsonPath = "$";
if(json != JSONObject.NULL) {
readObject(object, jsonPath);
}
}
private void readObject(JSONObject object, String jsonPath) {
Iterator<String> keysItr = object.keys();
String parentPath = jsonPath;
while(keysItr.hasNext()) {
String key = keysItr.next();
Object value = object.get(key);
jsonPath = parentPath + "." + key;
if(value instanceof JSONArray) {
readArray((JSONArray) value, jsonPath);
}
else if(value instanceof JSONObject) {
readObject((JSONObject) value, jsonPath);
} else { // is a value
this.pathList.add(jsonPath);
}
}
}
private void readArray(JSONArray array, String jsonPath) {
String parentPath = jsonPath;
for(int i = 0; i < array.length(); i++) {
Object value = array.get(i);
jsonPath = parentPath + "[" + i + "]";
if(value instanceof JSONArray) {
readArray((JSONArray) value, jsonPath);
} else if(value instanceof JSONObject) {
readObject((JSONObject) value, jsonPath);
} else { // is a value
this.pathList.add(jsonPath);
}
}
}
}

Refer to this utility : https://github.com/wnameless/json-flattener
Perfect answer to your requirement. Provides Flattened map and Flattened strings for complex json strings.
I am not the author of this but have used it successfully for my usecase.

Convert JSON object with duplicate keys to JSON array

I have a JSON string that I get from a database which contains repeated keys. I want to remove the repeated keys by combining their values into an array.
For example
Input
{
"a":"b",
"c":"d",
"c":"e",
"f":"g"
}
Output
{
"a":"b",
"c":["d","e"],
"f":"g"
}
The actual data is a large file that may be nested. I will not know ahead of time what or how many pairs there are.
I need to use Java for this. org.json throws an exception because of the repeated keys, gson can parse the string but each repeated key overwrites the last one. I need to keep all the data.
If possible, I'd like to do this without editing any library code

As of today the org.json library version 20170516 provides accumulate() method that stores the duplicate key entries into JSONArray
JSONObject jsonObject = new JSONObject();
jsonObject.accumulate("a", "b");
jsonObject.accumulate("c", "d");
jsonObject.accumulate("c", "e");
jsonObject.accumulate("f", "g");
System.out.println(jsonObject);
Output:
{
"a":"b",
"c":["d","e"],
"f":"g"
}

I want to remove the repeated keys by combining their values into an array.
Think other than JSON parsing library. It's very simple Java Program using String.split() method that convert Json String into Map<String, List<String>> without using any library.
Sample code:
String jsonString = ...
// remove enclosing braces and double quotes
jsonString = jsonString.substring(2, jsonString.length() - 2);
Map<String, List<String>> map = new HashMap<String, List<String>>();
for (String values : jsonString.split("\",\"")) {
String[] keyValue = values.split("\":\"");
String key = keyValue[0];
String value = keyValue[1];
if (!map.containsKey(key)) {
map.put(key, new ArrayList<String>());
}
map.get(key).add(value);
}
output:
{
"f": ["g"],
"c": ["d","e"],
"a": ["b"]
}

In order to accomplish what you want, you need to create some sort of custom class since JSON cannot technically have 2 values at one key. Below is an example:
public class SomeClass {
Map<String, List<Object>> values = new HashMap<String, List<Object>>();
public void add(String key, Object o) {
List<Object> value = new ArrayList<Object>();
if (values.containsKey(key)) {
value = values.get(key);
}
value.add(o);
values.put(key, value);
}
public JSONObject toJson() throws JSONException {
JSONObject json = new JSONObject();
JSONArray tempArray = null;
for (Entry<String, List<Object>> en : values.entrySet()) {
tempArray = new JSONArray();
for (Object o : en.getValue()) {
tempArray.add(o);
}
json.put(en.getKey(), tempArray);
}
return json;
}
}
You can then retrieve the values from the database, call the .add(String key, Object o) function with the column name from the database, and the value (as the Object param). Then call .toJson() when you are finished.

Thanks to Mike Elofson and Braj for helping me in the right direction. I only wanted to have the keys with multiple values become arrays so I had to modify the code a bit. Eventually I want it to work for nested JSON as well, as it currently assumes it is flat. However, the following code works for what I need it for at the moment.
public static String repeatedKeysToArrays(String jsonIn) throws JSONException
{
//This assumes that the json is flat
String jsonString = jsonIn.substring(2, jsonIn.length() - 2);
JSONObject obj = new JSONObject();
for (String values : jsonString.split("\",\"")) {
String[] keyValue = values.split("\":\"");
String key = keyValue[0];
String value = "";
if (keyValue.length>1) value = keyValue[1];
if (!obj.has(key)) {
obj.put(key, value);
} else {
Object Oold = obj.get(key);
ArrayList<String> newlist = new ArrayList<String>();
//Try to cast as JSONArray. Otherwise, assume it is a String
if (Oold.getClass().equals(JSONArray.class)) {
JSONArray old = (JSONArray)Oold;
//Build replacement value
for (int i=0; i<old.length(); i++) {
newlist.add( old.getString(i) );
}
}
else if (Oold.getClass().equals(String.class)) newlist = new ArrayList<String>(Arrays.asList(new String[] {(String)Oold}));
newlist.add(value);
JSONArray newarr = new JSONArray( newlist );
obj.put(key,newarr);
}
}
return obj.toString();
}