I have a problem with my ES queries where they fail because of having array indexes in their query strings. This happens because my following approach.
I flatten the JSON requests that I get with the following method.
private void flattenJsonRequestToMap(String currentPath, JsonNode jsonNode, Map<String, Object> map) {
if (jsonNode == null || jsonNode.isNull()) {
map.remove(currentPath);
} else if (jsonNode.isObject()) {
ObjectNode objectNode = (ObjectNode) jsonNode;
Iterator<Map.Entry<String, JsonNode>> iter = objectNode.fields();
String pathPrefix = currentPath.isEmpty() ? "" : currentPath + ".";
while (iter.hasNext()) {
Map.Entry<String, JsonNode> entry = iter.next();
flattenJsonRequestToMap(pathPrefix + entry.getKey(), entry.getValue(), map);
}
} else if (jsonNode.isArray()) {
ArrayNode arrayNode = (ArrayNode) jsonNode;
for (int i = 0; i < arrayNode.size(); i++) {
flattenJsonRequestToMap(currentPath + "[" + i + "]", arrayNode.get(i), map);
}
} else if (jsonNode.isValueNode()) {
ValueNode valueNode = (ValueNode) jsonNode;
map.put(currentPath, valueNode.asText());
} else {
LOGGER.error("JSONNNode unexpected field found during the flattening of JSON request" + jsonNode.asText());
}
}
When the Json requests have lists in them, my flattened map looks like below.
myUserGuid -> user_testuser34_ibzwlm
numberOfOpenings -> 1
managerUserGuids[0] -> test-userYspgF1_S3P6s
accessCategories[0] -> RESTRICTED
employeeUserGuid -> user_user33_m1minh
Now I construct ES Query with the following method using the above map.
public SearchResponse searchForExactDocument(final String indexName, final Map<String, Object> queryMap)
throws IOException {
BoolQueryBuilder queryBuilder = QueryBuilders.boolQuery();
queryMap.forEach((name, value) -> {
queryBuilder.must(QueryBuilders.matchPhraseQuery(name, value));
LOGGER.info("QueryMap key: {} and value: {} ", name, value);
});
return this.executeSearch(indexName, queryBuilder);
}
As you can already see, it ends up executing the query below, with the array indexes in them. My mapping structure is as follows.
{
name=job,
type=_doc,
mappingData={
properties={
myUserGuid ={
type=text,
fields={
keyword={
ignore_above=256,
type=keyword
}
}
},
numberOfOpenings ={
type=long
},
numOfUsage={
type=long
},
accessCategories ={
type=text,
fields={
keyword={
ignore_above=256,
type=keyword
}
}
},
managerUserGuids ={
type=text,
fields={
keyword={
ignore_above=256,
type=keyword
}
}
},
employeeUserGuid ={
type=text,
fields={
keyword={
ignore_above=256,
type=keyword
}
}
}
}
}
Because of the appended array index next to the name, the queries don't return any search results. How can I navigate this issue? One option I see is removing the array index using flattening the map, however I need to be able to construct a POJO object which has list for those fields in concern, using the flattened map. Would appreciate any advice/suggestions. Thanks a lot in advance.
Lists in ES are processed like just having few values for one field so if you have "accessCategories": ["foo", "bar"] this doc will match both "accessCategories": "foo" and "accessCategories": "bar" though there is no way to make a query which would match only one ("foo" but not "bar") with this data schema.
If you need to address specific items, you can unwrap list into separate fields accessCategories_0, accessCategories_1, etc. though there is a limit in Elasticsearch for total number of fields in one index.
I want to flatten a nested hashmap. For eg,
Map<String, Object> map = new HashMap();
Map<String, Object> map2 = new HashMap();
Map<String, Object> map3 = new HashMap();
map3.put("key3", 123);
map2.put("key2", map3);
map2.put("key4", "test");
map.put("key1", map2);
map.put("key6", "test2");
This will have structure something similar to this:
{
"key1" : {
"key2" : {"key3" : 123 },
"key4" : "test"
},
"key6" : "test2"
}
This should be flattened to a hashmap with structure similar to this:
{
"key1.key2.key3" : 123,
"key1.key4" : "test",
"key6" : "test2"
}
I tried solving this recursively in Java, but not able to produce the right form. Any help is appreciated.
Can be acheived using Stream#flatMap and recursion.
flatMapper() will be recursively called and it will return a Stream of string builder with child keys appended.
public class NestedHashMap {
private static Stream<StringBuilder> flatMapper(Map.Entry<String, Object> entrySet) {
Object value = entrySet.getValue();
if (value instanceof Map) {
return ((Map<String, Object>) value).entrySet()
.stream()
.flatMap(NestedHashMap::flatMapper)
.map(s -> s.insert(0, entrySet.getKey() + "."));
}
return Stream.of(new StringBuilder(entrySet.toString()));
}
public static void main(String[] args) {
// ... Code in the question
System.out.println(map.entrySet()
.stream()
.flatMap(NestedHashMap::flatMapper)
.collect(Collectors.joining(",\n\t", "{\n\t", "\t\n}")));
}
}
Output:
{
key1.key2.key3=123,
key1.key4=test,
key6=test2
}
I am trying to compare 2 JSON files, they have arrays with duplicated values.
My first JSON Object has an array like this:
"categories": [
"May",
"Apr",
"Mar"
]
My second JSON object has an array like this:
"categories": [
"May",
"May",
"Apr",
"Apr",
"Mar",
"Mar"
]
I am comparing the JSON using flat maps that can be found in this link comparing JSONs using guava
Here is part of my code:
private String smartJSONsCompare(JSONObject leftJson, JSONObject rightJson) {
String result = "</br>";
Gson gson = new Gson();
Type type = new TypeToken<Map<String, Object>>(){}.getType();
Map<String, Object> leftMap = gson.fromJson(leftJson.toString(), type);
Map<String, Object> rightMap = gson.fromJson(rightJson.toString(), type);
Map<String, Object> leftFlatMap = FlatMapUtil.flatten(leftMap);
Map<String, Object> rightFlatMap = FlatMapUtil.flatten(rightMap);
MapDifference<String, Object> difference = Maps.difference(leftFlatMap, rightFlatMap);
StringBuilder SB = new StringBuilder("</br>");
SB.append("Entries only on LEFT: </br>");
difference.entriesOnlyOnLeft().forEach((key, value) -> SB.append(key + ": " + value + "</br>"));
SB.append("Entries only on RIGHT: </br>");
difference.entriesOnlyOnRight().forEach((key, value) -> SB.append(key + ": " + value + "</br>"));
SB.append("Entries full difference : </br>");
difference.entriesDiffering().forEach((key, value) -> SB.append(key + ": " + value + "</br>"));
result = SB.toString();
return result;
}
I wish to be able to present the difference in a more "smart" way. In other words: showing all the objects / arrays in the JONSs that don't match. What is missing or what was added to the compared JSON.
For the "categories" array my code returns a message that their is a mismatch, but doesn't state the difference in an elegant way.
What can I do?
I have change a bit in your solution to get the wanted result.
I would do my difference check in List, therefore I will create method to change JSON to list of strings based on your code:
private static List<String> jsonToList(String json){
List<String> list = new ArrayList<>();
Gson gson = new Gson();
Type type = new TypeToken<Map<String, Object>>(){}.getType();
Map<String, Object> jsonMap = gson.fromJson(json, type);
Map<String, Object> flatten = FlatMapUtil.flatten(jsonMap);
flatten.forEach((k, v) -> list.add(v.toString()));
return list;
}
Update
When I answered the question I did things a bit fast, the jsonToList was based on your code. As it is right now it is over complicated to what you are asking for. I have therefore made much lighter version using the following method in stead:
private static List<String> jsonToList(String json) {
JSONObject response = new JSONObject(json);
List<String> list = new ArrayList<>();
JSONArray jsonArray = response.getJSONArray("categories");
if (jsonArray != null) {
for (int i = 0; i < jsonArray.length(); i++) {
list.add(jsonArray.get(i).toString());
}
}
return list;
}
That said, now you have two choices and it is up to you to find out which one fits best to your needs and take it from here.
End of Update
for this example I have made 3 test examples
String main = "{\"categories\":[\"May\",\"Apr\",\"Mar\"]}";
String json1 = "{\"categories\":[\"May\",\"May\",\"Apr\",\"Apr\",\"Mar\",\"Mar\"]}";
String json2 = "{\"categories\":[\"May\",\"Apr\",\"Apr\",\"Mar\",\"Mar\",\"Mar\"]}";
String json3 = "{\"categories\":[\"May\",\"Apr\",\"Mar\",\"Mar\"]}";
in my second step I will create a
List<String> mainList = jsonToList(main);
List<String> list1 = jsonToList(json1);
so far so good. Now I make a method to take the extra difference of the 2 list, that mean as you requested in your comments, we take only all values that are duplicated more than once and return them in list. In this method I used hashmap only count duplicates and than take the all that is repeated more than 1 time:
private static List<String> diffList(List<String> mainList, List<String> secondList){
List<String> list = new ArrayList<String>();
Map<String, Integer> wordCount = new HashMap<>();
for(String word: secondList) {
if(mainList.contains(word)) {
Integer count = wordCount.get(word);
wordCount.put(word, (count == null) ? 1 : count + 1);
if(wordCount.get(word) > 1){
list.add(word);
}
}
}
return list;
}
Finally I would test all cases, for instance for list1:
List<String> diff1 = diffList(mainList, list1);
for (String s : diff1) {
System.out.println(s);
}
The output will be
May
Apr
Mar
for list2
Apr
Mar
Mar
And for list3
Mar
Now I will separate view method from the your method and create some thing like, just to make my code more clear and easy to work with:
private static String viewResult(List<String> list1, List<String> list2, List<String> duplicate){
String result;
StringBuilder SB = new StringBuilder("</br>");
SB.append("Entries only on LEFT: </br>");
list1.forEach(e -> SB.append(e + "</br>"));
SB.append("Entries only on RIGHT: </br>");
list2.forEach(e -> SB.append(e + "</br>"));
SB.append("Entries full difference : </br>");
duplicate.forEach(e -> SB.append(e + "</br>"));
result = SB.toString();
return result;
}
So if we put all this code together I will be some thing like this, and the following code is to demonstrate how things works, but from here you can take it to the next level in your code:
public static void main(String[] args) {
String main = "{\"categories\":[\"May\",\"Apr\",\"Mar\"]}";
String json1 = "{\"categories\":[\"May\",\"May\",\"Apr\",\"Apr\",\"Mar\",\"Mar\"]}";
String json2 = "{\"categories\":[\"May\",\"Apr\",\"Apr\",\"Mar\",\"Mar\",\"Mar\"]}";
String json3 = "{\"categories\":[\"May\",\"Apr\",\"Mar\",\"Mar\"]}";
List<String> mainList = jsonToList(main);
List<String> list1 = jsonToList(json1);
List<String> diff1 = diffList(mainList, list1);
for (String s : diff1) {
System.out.println(s);
}
String view = viewResult(mainList, list1, diff1);
}
private static List<String> jsonToList(String json){
List<String> list = new ArrayList<String>();
Gson gson = new Gson();
Type type = new TypeToken<Map<String, Object>>(){}.getType();
Map<String, Object> jsonMap = gson.fromJson(json, type);
Map<String, Object> flatten = FlatMapUtil.flatten(jsonMap);
flatten.forEach((k, v) -> list.add(v.toString()));
return list;
}
private static List<String> diffList(List<String> mainList, List<String> secondList){
List<String> list = new ArrayList<String>();
Map<String, Integer> wordCount = new HashMap<>();
for(String word: secondList) {
if(mainList.contains(word)) {
Integer count = wordCount.get(word);
wordCount.put(word, (count == null) ? 1 : count + 1);
if(wordCount.get(word) > 1){
list.add(word);
}
}
}
return list;
}
private static String viewResult(List<String> list1, List<String> list2, List<String> duplicate){
String result;
StringBuilder SB = new StringBuilder("</br>");
SB.append("Entries only on LEFT: </br>");
list1.forEach(e -> SB.append(e + "</br>"));
SB.append("Entries only on RIGHT: </br>");
list2.forEach(e -> SB.append(e + "</br>"));
SB.append("Entries full difference : </br>");
duplicate.forEach(e -> SB.append(e + "</br>"));
result = SB.toString();
return result;
}
If you want something more generic with a good diff you could utilize AssertJ here.
Its usually used for Testing, but the diff looks really good and you can also use it in normal code.
Example:
Expecting:
<["Mai", "Apr", "Mar"]>
to contain exactly in any order:
<["May", "Apr", "Mar", "Mar"]>
elements not found:
<["May", "Mar"]>
and elements not expected:
<["Mai"]>
Can be created by:
[...]
import org.assertj.core.api.Assertions;
public class JsonTest {
final static String arr = " [\n"+
" \"Mai\",\n"+
" \"Apr\",\n"+
" \"Mar\"\n"+
" ]";
final static String arr2 = " [\n"+
" \"May\",\n"+
" \"Apr\",\n"+
" \"Mar\",\n"+
" \"Mar\"\n"+
" ]";
public static void main(String[] args){
System.out.println(smartJSONsCompare(arr,arr2));
}
private static String smartJSONsCompare(String leftJson, String rightJson) {
Gson gson = new Gson();
Type type = new TypeToken<List<String>>(){}.getType();
List<String> left = gson.fromJson(leftJson, type);
List<String> right = gson.fromJson(rightJson, type);
try{
Assertions.assertThat(left).containsExactlyInAnyOrderElementsOf(right);
}catch(AssertionError ae){
return ae.getMessage();
}
return "Matched";
}
}
I added the dependencies in gradle with:
dependencies {
compile("org.assertj:assertj-core:3.11.1")
}
If you want to create a patch between your two JSON Objects have a look at json-patch.
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.github.fge.jsonpatch.diff.JsonDiff;
import java.io.IOException;
public class JsonPatchTest {
public static void main(String[] args) throws IOException {
String jsonFirst = "{\"categories\":[\"May\",\"Apr\",\"Mar\"]}";
String jsonSecond = "{\"categories\":[\"May\",\"May\",\"Apr\",\"Apr\",\"Mar\",\"Mar\"]}";
ObjectMapper mapper = new ObjectMapper();
JsonNode jsonNodeFirst = mapper.readTree(jsonFirst);
JsonNode jsonNodeSecond = mapper.readTree(jsonSecond);
JsonNode patchNode = JsonDiff.asJson(jsonNodeFirst, jsonNodeSecond);
System.out.println(mapper.writerWithDefaultPrettyPrinter().writeValueAsString(patchNode));
}
}
Would produce the following output for your scenario:
[ {
"op" : "replace",
"path" : "/categories/1",
"value" : "May"
}, {
"op" : "replace",
"path" : "/categories/2",
"value" : "Apr"
}, {
"op" : "add",
"path" : "/categories/-",
"value" : "Apr"
}, {
"op" : "add",
"path" : "/categories/-",
"value" : "Mar"
}, {
"op" : "add",
"path" : "/categories/-",
"value" : "Mar"
} ]
I believe you should handle json arrays on your own in order to present their difference in a more "smart" way. Here is a library which contains CollectionUtils class with disjunction method.
MapDifference<String, Object> difference = Maps.difference(leftMap, rightMap);
difference.entriesDiffering().forEach((key, value) -> {
Object left = value.leftValue();
Object right = value.rightValue();
if (left instanceof Iterable && right instanceof Iterable) {
Collection<?> diff = CollectionUtils.disjunction((Iterable<?>) right, (Iterable<?>) left);
System.out.println(key + " -> " + diff);
}
...
});
This code work for me (2 years ago) on production.
public class App {
private final Gson GSON = new GsonBuilder().create();
public boolean isDifference(final String path, Map<String, Object> oldData, Map<String, Object> newData) {
MapDifference<String, Object> difference = Maps.difference(oldData, newData);
difference.entriesOnlyOnLeft().forEach((key, value) -> {
publishChange(Action.REMOVE, path, key, value);
});
difference.entriesOnlyOnRight().forEach((key, value) -> {
publishChange(Action.ADD, path, key, value);
});
difference.entriesDiffering().forEach((key, value) -> {
if (value.rightValue() instanceof Map && value.leftValue() instanceof Map) {
if (!path.isEmpty()) {
key = path.concat("-").concat(key);
}
isDifference(key, (Map) value.leftValue(), (Map) value.rightValue());
} else {
publishChange(Action.MODIFY, path, key, value);
}
});
return !difference.areEqual();
}
public void publishChange(Action action, String path, String key, Object value) {
if (value instanceof MapDifference.ValueDifference) {
value = ((MapDifference.ValueDifference) value).rightValue();
}
JsonElement jsonValue = GSON.toJsonTree(value);
String event = createEvent(action, path, key, jsonValue);
System.out.println("Differrence: " + event);
}
public String createEvent(Action action, String paths, String key, JsonElement value) {
JsonObject root = new JsonObject();
JsonArray arrPaths = new JsonArray();
for (String path : paths.split("-")) {
arrPaths.add(path);
}
root.addProperty("action", action.toString());
root.add("paths", arrPaths);
JsonObject data = new JsonObject();
data.addProperty("key", key);
data.add("value", value);
root.add("data", data);
return root.toString();
}
public static enum Action {
ADD, REMOVE, MODIFY
}}
Test/ Example:
public class AppTest {
#Test
public void testAppHasAGreeting() {
App classUnderTest = new App();
Gson gson = new GsonBuilder()
.setPrettyPrinting()
.create();
// JsonOld: {"a":1,"b":1,"c":true,"array":[1,2,3],"object":{"arrayKey":["a","b","c","d"]}}
String jsonOld = "{\"a\":1,\"b\":1,\"c\":true,\"array\":[1,2,3],\"object\":{\"arrayKey\":[\"a\",\"b\",\"c\",\"d\"]}}";
// JsonNew: {"a":2,"b":1,"array":[1,2,3,2],"another":{"d":false,"e":["a","b","c"]},"object":{"booleanKey":true,"arrayKey":["a","b","c"]}}
String jsonNew = "{\"a\":2,\"b\":1,\"array\":[1,2,3,2],\"another\":{\"d\":false,\"e\":[\"a\",\"b\",\"c\"]},\"object\":{\"booleanKey\":true,\"arrayKey\":[\"a\",\"b\",\"c\"]}}";
Type mapType = new TypeToken<Map<String, Object>>() {
}.getType();
Map<String, Object> jsonOldAsMap = gson.fromJson(jsonOld, mapType);
Map<String, Object> jsonNewAsMap = gson.fromJson(jsonNew, mapType);
System.out.println("Old Json: " + gson.toJson(jsonOldAsMap));
System.out.println("New Json: " + gson.toJson(jsonNewAsMap));
System.out.println("========== Result ==========");
// When
boolean diff = classUnderTest.isDifference("", jsonOldAsMap, jsonNewAsMap);
// Then
assertTrue(diff);
}}
The result will print like this:
Differrence: {"action":"REMOVE","paths":[""],"data":{"key":"c","value":true}}
Differrence: {"action":"ADD","paths":[""],"data":{"key":"another","value":{"d":false,"e":["a","b","c"]}}}
Differrence: {"action":"MODIFY","paths":[""],"data":{"key":"a","value":2.0}}
Differrence: {"action":"MODIFY","paths":[""],"data":{"key":"array","value":[1.0,2.0,3.0,2.0]}}
Differrence: {"action":"ADD","paths":["object"],"data":{"key":"booleanKey","value":true}}
Differrence: {"action":"MODIFY","paths":["object"],"data":{"key":"arrayKey","value":["a","b","c"]}}
The code available here: https://github.com/liemle3893/compare-json