We Keep Coding

Jackson & Scala: How to get property value from a list of objects by property value?

I'd like to get the requestedInstanceCount from instanceGroupName = slave. How can this be achieved with Jackson?
Below is the job-flow.json:
{
"generalId": "ABC"
"instanceCount": 4,
"instanceGroups": [
{
"instanceGroupId": "CDE",
"instanceGroupName": "master",
"requestedInstanceCount": 1
},
{
"instanceGroupId": "FGH",
"instanceGroupName": "slave",
"requestedInstanceCount": 8
}
]
}
So far this is what I have:
val jobFlowJson: String = new String(Files.readAllBytes(Paths.get("/mnt/var/lib/info/job-flow.json")))
val jsonNode = mapper.readValue(jobFlowJson, classOf[JsonNode])
val instanceCount = jsonNode.get("requestedInstanceCount").asInt
But there are 2 values and the order between master & slave can change at any time. Thanks in advance!

You have to go through the JSON tree step by step:
get the instanceGroups as an array
iterate over the array to find the item you want
extract the value requestedInstanceCount
Something like this (pseudo Scala code):
jsonNode.get("instance groups")
.asArray
.collect {
case item if item.get("instanceGroupName").asString == "..." =>
item.get("requestedInstanceCount")
}
Or define some case class representing the structure and pass on your JSON into the case class. It will be way easier to manipulate if you have no specific reason to not do this.

Concurrent exception in Groovy even when changes are made to another LinkedHashMap

I have created a new LinkedHashMap 'workingStrData' using 'strData' and I still get the error.
I am trying to remove some items from this LinkedHashMap based of another list.
The way strData is structured is
strData = [components[{key1:value1}{key2:value2}...]]
def workingStrData = new LinkedHashMap(strData)
List componentsToRemove = ['item1','item2'...]
int itemsRemoved = 0
workingStrData.components.eachWithIndex {
comp, workingStrIndex ->
println("Index: "+workingStrIndex+" Component: "+comp.number)
def baditem = comp.number in componentsToRemove
if (baditem) {
strData.components.remove(workingStrIndex - itemsRemoved)
itemsRemoved += 1
}
}

You cannot remove an element from a list while iterating with each or with eachWithIndex which by the way is a bad practice. Groovy offers much more elegant and simpler solutions.
As suggested, try retainAll() or as suggested here, try removeAll():
def strData = [
components : [
[number: 'item0'],
[number: 'item1']
]
]
def componentsToRemove = [
'item1','item2'
]
componentsToRemove.each { removeIt ->
strData.components.removeAll { it.number == removeIt }
}
assert strData.components == [[number: 'item0']]

It is not complaining about the strData/workingStrData maps, it is complaining about the List referenced by the value of the "components" key in the map.
You know, the List you are actually iterating, and the List you actually call remove(int index) on.
You don't actually iterate or modify the map, so copying that is meaningless.

Eddiel is right, you can't modify array ( and strData.components is an array ) while you iterate it.
Even if you created a copy of root map def workingStrData = new LinkedHashMap(strData) the content of this map referencing the same data.
it means that workingStrData.components and strData.components referencing the same array.
groovy styled code to modify array:
def strData = [
components:[
[number:'111', name:'aaa'],
[number:'222', name:'bbb'],
[number:'333', name:'ccc'],
[number:'444', name:'ddd'],
]
]
List componentsToRemove = ['222','333']
//--------------------------------------
// if you want to keep original data:
def modData = [
components: strData.components.findAll{c-> c.number in componentsToRemove }
]
println "original : $strData"
println "modified : $modData"
//--------------------------------------
// if you want to modify existing data
strData.components.retainAll{c-> !( c.number in componentsToRemove) }
println "original (modified): $strData"

You can not modify the map unless you use an iterator or you use a synchronized collection. This is by design in Java. Check https://docs.oracle.com/javase/8/docs/api/java/util/Collections.html#synchronizedMap-java.util.Map- to wrap your collection or, remove the elements using the iterator's own remove function.
In java, that would look like this:
Iterator<String> iterator = workingStrData.keySet().iterator();
while (iterator.hasNext()) {
iterator.next();
iterator.remove();
}
// workingStrData is empty at this point.

Print JSON with ordered properties

I have JSON with objects in specific order:
{
"Aaa": {
"Langs": {
"Val": [
"Test"
],
"Pro": [
"Test2"
]
}
},
"Bbb": {
"Langs": {
"Val": [
"Test"
],
"Pro": [
"Test2"
]
}
},
"Ddd": {
"Langs": {
"Val": [
"Test"
],
"Pro": [
]
}
},
}
And I would like to add new object Ccc between Bbb and Ddd. I tried to configure object mapper like this:
ObjectMapper mapper = new ObjectMapper()
.enable(SerializationFeature.INDENT_OUTPUT)
.configure(MapperFeature.SORT_PROPERTIES_ALPHABETICALLY, true)
.configure(SerializationFeature.ORDER_MAP_ENTRIES_BY_KEYS, true);
and then print with this code, but Ccc ends at the end of file.
DefaultPrettyPrinter prettyPrinter = new DefaultPrettyPrinter();
prettyPrinter.indentArraysWith(DefaultIndenter.SYSTEM_LINEFEED_INSTANCE);
//Write whole JSON in FILE
String finalJson = mapper.writer(prettyPrinter).writeValueAsString(rootFlores);
finalJson = finalJson.replaceAll("\\[ ]", "[" + System.lineSeparator() + " ]");
finalJson = finalJson.replaceAll("/", "\\\\/");
Files.write(Paths.get("DictionaryFlores_new.json"), Collections.singleton(finalJson));
Is here a way how to print JSON ordered?

Jackson deserialization/serialization does not sort properties
According to this answer, the Jackson SORT_PROPERTIES_ALPHABETICALLY only applies to POJO properties, not Maps. In JSON there is no difference between a Map and an Object, so you need to set the order in the Map first by using a LinkedHashMap or TreeMap

By definition, the keys of an object are unordered. I guess some libraries could offer an option to control the order of the keys when stringifying, but I wouldn't count on it.
When you need a certain order in json, you need to use an array. Of course, then you'd have to move the keys to a property in the child objects, and then the resulting array could only be indexed by number (not by the key). So then you might have to do additional processing to covert the data structure in the JSON to the data structure you really want to process.

Since you seems ready to use regex to update a JSON, I would suggest a "safer" approach. Don't try to create a pattern that would unsure that you don't update a value somewhere.
Iterate you values, on object at the time. Stringify the object and append the String yourself. That way, you are in charge of the object order. Example :
StringBuilder sb = new StringBuilder("{");
List<JsonPOJO> list = new ArrayList<>();
//populate the list
for(JsonPOJO pojo : list){
sb.append(pojo.stringify()).append(",");
}
sb.setLength(sb.length() - 1); //remove the last commma
sb.append("}");
Here, you are only managing the comma between each JSON object, not create the "complex" part about the JSON. And you are in full control of the order of the value in the String representation, it will only depend on the way you populate the List.
Note: sorry for the "draft" code, I don't really have access to my system here so just write this snippet to give you a basic idea on how to "manage" a JSON without having to recreating an API completely.
Note2: I would note suggest this unless this looks really necessary. As you mention in a comment, you are have only the problem with one key where you already have a JSON with 80000 keys, so I guess this is a "bad luck" scenario asking for last resort solution ;)

Create json object using json.simple

I am using the following libraries to create some JSON object.
import org.json.simple.JSONArray;
import org.json.simple.JSONObject;
The json I am trying to create is this:
{
"function": "create_contact_group",
"parameters": [{
"user_id": "teer",
"comp_id": "97",
"contact_group_name": "Test01",
"invite_user_list": [{
"invite_user_id": "steve"
}]
}]
}
My function looks like this:
public JSONObject createJSONRequest() {
/* Create json object */
JSONObject jsonObject = new JSONObject();
Map<String, String> map = new HashMap<>();
map.put("user_id", "teer");
map.put("comp_id", "97");
map.put("contact_group_name", "Test01");
List<String> mInviteUserList = new ArrayList<>();
mInviteUserList.add("steve");
/* Create the list of invitee */
Map<String, String> inviteList = new HashMap<>();
for(String user : mInviteUserList) {
inviteList.put("invite_user_id", user);
}
/* Add the invitees into the json array */
JSONArray inviteArray = new JSONArray();
inviteArray.add(inviteList);
/* Add the json array to the json object */
jsonObject.put("invite_user_list", inviteArray);
JSONArray parameterlist = new JSONArray();
parameterlist.add(map);
parameterlist.add(jsonObject);
jsonObject.put("parameters", parameterlist);
jsonObject.put("function", "create_contact_group");
Log.d(TAG, "jsonObject: " + jsonObject.toJSONString());
return jsonObject;
}
However, the function crashes when I get to the following line:
Log.d(TAG, "jsonObject: " + jsonObject.toJSONString())
I think it has something to do with this line here:
parameterlist.add(jsonObject);
Stacktrace:
java.lang.StackOverflowError java.lang.AbstractStringBuilder.enlargeBuffer(AbstractStringBuilder.java:95) at java.lang.AbstractStringBuilder.append0(AbstractStringBuilder.java:132)
at java.lang.StringBuffer.append(StringBuffer.java:126) at org.json.simple.JSONValue.escape(JSONValue.java:266) at org.json.simple.JSONObject.toJSONString(JSONObject.java:116)
Many thanks for any suggestions,

There are a couple of issues in your code.
One issue is that the structure you create in the java code will not match the structure that you show above. This I will try to describe a bit below.
The second issue is that you get a stackoverflow exception (which you know but don't know why).
The stackoverflow exception is thrown cause the program runs out of the stack memory assigned by the computer. Why you ask? Well, cause you create a recursive or cyclic JSON object.
This isn't good, but its not that big a deal cause its kinda easy to fix.
So why does the program throw this exception? Well, look at the following snippet:
JSONArray parameterlist = new JSONArray();
parameterlist.add(map);
parameterlist.add(jsonObject);
jsonObject.put("parameters", parameterlist);
jsonObject.put("function", "create_contact_group");
You create a JSONArray and then add the JSONObject created before to the array.
After that you add the same array to the object that is already in the array.
I expect that you see the issue with that!
So, that should not be done.
And how to fix this? Well, I kinda think its better that I describe how you should write the code to get the structure you are actually asking for, so I'll try do that...
What to do...?
A JSON (JavaScript Object Notation) -object is always declared with this type of brackets: {} a JSON array with [], so, the JSON you are trying to generate should be in the following data types:
{ // <= Root object (a JSON-object).
"function": "create_contact_group", // <= Key in the root-object (where the key is a string and the value a string)
"parameters": [ // <= Key in the root-object (where the key is a string and the value is an array.
{ // <= Object inside the array.
"user_id": "teer", // Key (string/string)
"comp_id": "97", // Key (string/string)
"contact_group_name": "Test01", // Key (string/string)
"invite_user_list": [ // Key (string/array)
{ // Object inside the invite_user_list array
"invite_user_id": "steve" // Key (string/string)
}
]
}
]
}
So when creating the JSON-object in java, you will want to create a root object then add all the diff params inside it.
Adding a value to a JSONObject is done with the JSONObject.put(string, Object) method, where the string is a key and the object a value.
So to start, I would recommend creating the parameters list.
In your case, you use a HashMap for the objects, which is not really wrong, but not really necessary either, I would just stick to a JSONObject, which is not all that different than a HashMap<string, Object>.
So instead of map.put(...), you could do something like:
JSONObject param = new JSONObject();
param.put("user_id", "teer");
param.put("comp_id", "97");
param.put("contact_group_name", "Test01");
Now, one of the objects values should be an array (invite_user_id) and the easiest way to add an array to the object is to create a JSONArray and then just add it.
JSONArray inviteList = new JSONArray();
// Then you need to add an object to the array for each `user` that has invited.
// For-loop here maybe?
JSONObject invitee = new JSONObject();
invitee.put("invite_user_id", user);
inviteList,add(invitee); // This will make it into an array with objects, I.E., [ { "invite_user_id": "Steve" } ]
After creating the invite list, add it to the param object like:
param.put("invite_user_list", inviteList);
// Now, param should be in its own list too, so you should probably create a JSONArray for the params.
// Ill leave that to you, and we pretend we have a list of the param objects named "params".
And then at the end, you create the root object and set its values:
JSONObject root = new JSONObject();
root.put("parameters", params);
root.put("function", "create_contact_group");
And that should be it.
This should create a JSON-string with the structure that you made above. But I would recommend testing (and writing unit tests!) for this (especially as I have written this code in the browser!).
But why?!
I guess I should try to describe why your code was not working as the one I described above.
You start by creating a root object, so far so good (can create it at start or at the time you need it, doesn't really matter), after that you create a HashMap and add the properties to it.
So far this is also legit (you could later create a JSONObject from the map).
In the next part, you create an ArrayList (im not really sure why) and add a name to it, and then another HashMap which you add the single name to (key invite_user_list) inside a for-loop.
This is either not necessary (cause its just one name) or wrong (if there is supposed to be more names in a real life execution of the code), in case of unnecessary, the for-loop shouldn't be there and in case of "not like real life" it should not be added to a Map!
Instead the invieList should have been an array, and each entry added should have been a object which had the "invite_user_id" key set to the name.
After that, you add the inviteList HashMap to a newly created JSONArray, I guess this could be kinda okay, if you only want one object ever in the array, else I would recommend creating it before the loop and add each entry into it!
The inviteArray is then put inside the root object with the key invite_user_list, after that you create another JSONArray and add both the map (your parameters created at the start) and the JSONObject (root) created first of all.
But the thing you do after that, is why you are getting a stackoverflow exception, you add the parameterlist (which contains the jsonObject (root)) to the jsonObject, which makes the jsonObject exist inside an array that is inside itself!
This creates a cyclic JSON structure which will never end if the whole thing was to be unrolled, hence the computer throws the exception.
The structure of the resulting object would also be wrong, cause it would look something like this:
{ // Root (jsonObject)
"invite_user_list": [
{ "invite_user_id": "steve" }
]
"parameters": [
{ // The "map" hashmap
"user_id", "teer",
"comp_id": "97",
"contact_group_name": "Test01"
},
{ // The jsonObject object (which is also the root!)
"invite_user_list": [
{ "invite_user_id": "steve" }
],
"parameters": [
{ // The "map" hashmap
"user_id", "teer",
"comp_id": "97",
"contact_group_name": "Test01"
},
{
// The jsonObject object again (which is also the root and the parent!)
// ... and so on til eternity!
}
],
"function": "create_contact_group"
}
],
"function": "create_contact_group"
}
Extra...
I would like to add here at the end (where I hope you end up after reading the whole wall of text that I wrote above, cause you might have learnt something!) that there is a easier way of doing it.
Now, I haven't used this lib myself, but from what I understand, it should be able to serialize a whole object, the lib can be found at Googles github repos which can be used as a json serializer and convert a class-instance to a json string, then you could just create a class for the whole object and fill it up and serialize it at the end of the function, without using either JSONArray nor JSONObject.

The issue is due to recursion process that occurs when you are trying to add the JsonObject to JsonArray and viceVersa.
The thing you are doing is,
JSONArray parameterlist = new JSONArray();
parameterlist.add(map);
parameterlist.add(jsonObject);
And then
jsonObject.put("parameters", parameterlist);
The problem is when you print the object using jsonObject.toJSONString(), Then at first it will fetch the parameterlist then as jsonObject is part of the keyvalue pair on the parameterlist JsonArray it will refetch the jsonObject which then again fetch the parameterlist and this process continues on and hence causing the StackOverflow Issue.
The Quick Solution is to create new JsonObject while assigning the parameterList,
JSONArray parameterlist = new JSONArray();
parameterlist.add(map);
parameterlist.add(jsonObject);
JSONObject newJson = new JSONObject();
newJson.put("parameters", parameterlist);
System.out.println(newJson.toJSONString());

Associativity array in Java

I receive a List<org.apache.avro.generic.GenericRecord> with the data contents as shown below (JSON notation used for clarity). How can I best hold these record types using Java?
Record 1:
[
{
"serial_no" : "x",
"data1" : "d"
},
{
"serial_no" : "y",
"data2" : "d2"
},
............................MANY MORE
]
Record 2:
[
{
"id":"x",
"type":"A"
},
{
"id" : "x",
"type" : "B"
},
{
"id" : "y",
"type" : "A",
},
{
"id" : "y",
"type" : "B"
}
]
As you see here, each serial number has two records in record2. serial_no in record1 is same as id in record2.
My Goal is:
Fatsest way to find these two records.
Solution I think:
Create a map like
map.put("x", [map.put("A",List), map.put("B",List)]);
But I feel like, its a complex structure. Because map contains list of maps[each map is Map<String,List<Map<String,String>>>].
Any suggestions?
EDIT
Each entries in records are avro GenericRecord

It looks as if you are trying to parse JSON using Java. Why not use a specific library for that?
Like the basic http://www.json.org/java/ or Google's https://github.com/google/gson
Otherwise, I do not think that the complex structure you are proposing is especially slow. You might want to design your own object class to hold the data if you think it is more efficient or easier to get to the data.
EDIT
Based on your question I assumed JSON was the format you received it in, sorry.
I would just create a wrapper for GenericRecord, or subclass it. Then add the methods that you need to extract the data, or make it Comparable for sorting.
Something along the lines of
public class MyRecord extends GenericRecord implements Comparable<MyRecord>
{
// Determine the type
public int getType()
{
if ( this.get( "id") != null )
return 2;
return 1;
}
// Add methods that allow you to retrieve the serial field from any of the two record types
public String getId()
{
if ( this.get( "id") != null )
return (String)this.get("id");
return (String)this.get("serial_no");
}
// add comparator methods that will allow you to sort the list, compare based on Id, etc
#Override
public int compareTo(MyRecord another)
{
// Just a simple example
return this.getId().compareTo( another.getId() );
}
}

Define classes for repeated entries:
class SerialNoData {
String serialNo;
Object data;
}
and
class IdType {
String id;
String type;
}
; once parsed put the instances into arrays or Lists to get the desired format.

How complex the map is doesn't really make a difference for the speed. Depending on the type of Map you use getting a list of records will be constant time (with a reasonably small overhead). Finding something in the sublists will then be O(n), since you need to iterate through the list and look at all the Maps.

Define following classes
class Serial{
String serial-no;
String data;
List<IdType> idTypes;
}
class IdType{
String id;
String type;
}
After that you can use jackson or any kind of JSON processing library.