GAE Datastore Text messes up JSON representation - java

The following code excerpt builds a Google App Engine Datastore Entity from a Java Map. Short strings (less than 450 characters) are stored directly, long strings are converted to Text objects.
for (String key : movieMap.keySet()) {
String value = movieMap.get(key).trim();
if (value.length()>0 && value.length()<450) { // Short String - store as is
movieEntity.setProperty(key, value);
} else if (value.length()>=450) { // Long String - convert to text
movieEntity.setProperty(key, new Text(value));
}
}
The following code retrieves the Entity as a Json object:
Map<String, Object> result = new LinkedHashMap<String, Object>();
result.put("numberOfMovies", String.valueOf(movies.size()));
result.put("movies", movies);
Gson gson = new GsonBuilder().setPrettyPrinting().create();
resp.getWriter().print(gson.toJson(result));
The result JSON for a String description fields:
{
"propertyMap": {
"title": "טאנו מתחתנת עם מאנו - Tanu Weds Manu",
"duration": "154 דקות",
"movieId": "2315",
"image_url": "http://www.sderot-cin.org.il/media/sal/movies2009/2315/f18_tano mithatenet iim mano.jpg",
"description": "סינמסטה - מועדון הסרט ההודי רופא ממוצא הודי חוזר מלונדון במטרה לפגוש את כלתו המיועדת. הוא מופתע לגלות כי היא מאוהבת במישהו אחר ואין בכוונתה להינשא לו. התנהגותה הפראית גורמת לו לשקול מחדש את צעדיו.",
"explorationDate": "Apr 26, 2013 8:42:36 PM",
"actors": "קנגנה רנאוט, אר מדווהין",
"year": "2011",
"language": "הינדית",
"director": "אנאנד ראי",
"country": "הודו"
}
},
{
The result JSON for a Text description fields, with a strange inner dictionary:
"propertyMap": {
"title": "הנעליים של אדון סימון - הצגת תיאטרון לילדים",
"movieId": "2316",
"image_url": "http://www.sderot-cin.org.il/media/sal/movies2009/2316/f18_anaaliim shel adon simon.jpg",
"description": {
"value": {
"value": "תיאטרון אורנה פורת לגילאי 3-7 אדון סימון עוזב את ארצו ועובר לגור בארץ אחרת. בביתו החדש הוא מחליט להיפרד מנעליו הישנות שליוו אותו כל חייו ולהחליפן בנעליים חדשות, אך משום מה הוא לא מצליח להיפטר מהישנות. בכל פעם שהוא זורק אותן, רחוק ככל שיהיה, הן תמיד חוזרות אליו בדרך פלא. הצגה מרגשת ומצחיקה שמלמדת אותנו לקבל בשמחה את הדברים החדשים שבחיינו ובאותו זמן להבין שאפשר גם להתגעגע לדברים הישנים. משך ההצגה כ50 דקות. כרטיסים: מכירה מוקדמת-30 ₪, ביום ההצגה-40 ₪ מנוי/מלווה : מכירה מוקדמת 20 ₪, ביום ההצגה 30 ₪."
}
},
"explorationDate": "Apr 26, 2013 8:42:36 PM"
}
},
The problem is that the description field returns normaly (key:value pair) for String field, but has a strange inner value key for Text fields - compare description in both examples.
Any idea how can I make Text fields behave normally and appear as the standard

Looks like the inner dictionary is a result of serializing the complex GAE Datastore Text object.
Since you're accessing your fields of your data with keys then I guess annotations and some getter/setter methods isn't possible?
In such case a simple way to fix it may be to iterate through the Map after it is retrieved and set (replace) the Text "description" fields with standard String if they were of type Text.
Then when you serialize to JSON those strings will be handled as the other strings.

Related

Get json object in Array based on key and value in Java

I have a Json body like the example below. I need to extract the value from a key that has another key with a specific value in an array. I am passing in a JsonNode with everything in the detail component of the message, I can easily extract from each level, however, I'm struggling with the array.
In this case, I need to extract the value of "value" (Police/Fire/Accident Report) from the object in the array which has a key/value pair of "name":"documentTitle". I understand this is a JSONArray, but I can't find a good example that shows me how to extract the values for an object in the array that contains a certain key/value pair, I don't think I can rely on getting the object in position [2] in the array as the same objects may not always be present in the additionalMetadata array.
Sample Json:
"sourceVersion": "1.0",
"eventId": "8d74b892-810a-47c3-882b-6e641fd509eb",
"clientRequestId": "b84f3a7b-03cc-4848-a1e8-3519106c6fcb",
"detail": {
"stack": "corona",
"visibilityIndicator": null,
"documentUid": "b84f3a7b-03cc-4848-a1e8-3519106c6fcb",
"additionalMetadata": [
{
"name": "lastModifiedDate",
"value": "2021-05-21T04:53:53Z"
},
{
"name": "documentName",
"value": "Police/Fire Report, 23850413, 2021-05-20 14:51:23"
},
{
"name": "documentTitle",
"value": "Police/Fire/Accident Report"
},
{
"name": "documentAuthor",
"value": "System Generated"
},
{
"name": "lastModifiedBy",
"value": "System Updated"
},
{
"name": "createdBy",
"value": "System Generated"
},
{
"name": "documentDescription",
"value": "Police/Fire Report received"
},
{
"name": "organizationCode",
"value": "Claims"
}
]
}
}```
Loop through the json array and extract the json object with name documentTitile. From that json object you can get the value
Well, either the JSON framework you're using supports this out of the box (check the documentation) or you could convert it manually to a map:
List<AdditionalMetadataEntry> additionalMetadata;
[...]
Map<String, String> additionalMetadataMap = additionalMetadata.stream().collect(Collectors.toMap(AdditionalMetadataEntry::getName, AdditionalMetadataEntry::getValue));
I was able to figure it out. I created a new node off the existing notificationBody JsonNode, then parsed through the metadata key/value pairs:
String docTitle = "";
JsonNode additionalMetadata = notificationBody.get("detail").get("additionalMetadata");
for (JsonNode node: additionalMetadata) {
String name = node.get("name").asText();
String value = node.get("value").asText();
if(name.equals("documentTitle")){
docTitle = value;
}
}

Java Spring - How to get one specific key value from a very nested json without mapping to a specific pojo?

Small java and spring question regarding how to get a specific key value from a very nested json, without having to map back to java pojos please.
I am consuming an api, where the json response is gigantic. The raw response does not fit in a screen.
The response is also very nested. Meaning, it has fields inside fields inside fields... etc
I have no access, no way to modify this API.
Nonetheless, it is a very interesting API, and in this very gigantic payload, very nested, there is always exactly one "the-key-i-need": "the-value-i-need",
Furthermore, there is no way to know in advanced how nested (which layer, which child) and no way to know where will "the-key-i-need": "the-value-i-need", be.
Also, there is no way to map the response back to any POJO, it changes always, the only information, "the-key-i-need": "the-value-i-need", exists, and it is always there.
I am well aware of GSON or fasterxml libraries, that can help map the json string back to existing pojos.
However, in my case, such will not help, since the existing pojo does not exists. the response is always different, the structure, the level of nesting is always different.
My question is, instead of trying to map back to pojos that will always change, and very nested, is there a simpler way, to just use some kind of regex, or something else, to extract the key value, "the-key-i-need": "the-value-i-need", and only this please?
I already tried mapping to all kinds of pojos, and unfortunately the response structure is too dynamic.
Thank you
Since you're using Spring, you very likely already have Jackson FasterXML in you classpath.
Normally, Spring uses the Databind module, which relies on the Streaming module, aka the Core module.
In this case, you want to use Streaming directly, so get the JSON text as a String, and start the parser.
static String getFieldValue(String json, String field) throws JsonParseException, IOException {
JsonFactory factory = new JsonFactory();
try (JsonParser parser = factory.createParser(json)) {
for (JsonToken token; (token = parser.nextToken()) != null; ) {
if (token == JsonToken.FIELD_NAME && parser.getCurrentName().equals(field)) {
token = parser.nextToken();
// check token here if object or array should throw exception instead of returning null
return parser.getValueAsString();
}
}
}
return null; // or throw exception if "not found" shouldn't return null
}
Test
String json = "{ \"A\": { \"B\": [ 5, { \"C\": \"D\" }, true ], \"E\": null, \"F\": 42, \"G\": false }}";
System.out.println("C: " + getFieldValue(json, "C")); // "D"
System.out.println("E: " + getFieldValue(json, "E")); // null (null-value)
System.out.println("F: " + getFieldValue(json, "F")); // "42"
System.out.println("G: " + getFieldValue(json, "G")); // "false"
System.out.println("H: " + getFieldValue(json, "H")); // null (not found)
System.out.println("B: " + getFieldValue(json, "B")); // null (not a value)
JsonPath
If using an external library is an option, then JsonPath might help.
Maven
<dependency>
<groupId>com.jayway.jsonpath</groupId>
<artifactId>json-path</artifactId>
<version>2.5.0</version>
</dependency>
StackOverflow Tag for JsonPath
Example usage as shared in the README
{
"store": {
"book": [
{
"category": "reference",
"author": "Nigel Rees",
"title": "Sayings of the Century"
},
{
"category": "fiction",
"author": "Evelyn Waugh",
"title": "Sword of Honour",
"price": 12.99
}
],
"bicycle": {
"color": "red",
"price": 19.95
}
},
"expensive": 10
}
To access all the author names
List<String> authors = JsonPath.read(json, "$..author");
System.out.println(authors);
Output
["Nigel Rees","Evelyn Waugh"]
To access all the prices (across both book and bicycle)
List<Double> prices = JsonPath.read(json, "$..price");
System.out.println(prices);
Output
[12.99, 19.95]
Note
Some missing keys can cause remapping extracted data across two fields difficult
Say fetching category and price from the above example will make it difficult to summarize a category to price mapping
List<String> categories = JsonPath.read(json, "$..category");
System.out.println(categories);
Output
["reference", "fiction"]
Based on the above example, price and category does not have correct 1-1 mapping

How do I read multiple JSON objects in Java?

So this is my sample Json Text,
{
"Date": [
1545,
"20 January 2014"
],
"docText": "ABCD",
"docSource": "",
"docID": 99,
"docDate": "",
"Date": [
1930,
"1995/11",
"past decade",
"today",
"Today"
],
"docText": "JJJJJJJ\"",
"Organization": [
"x",
"y",
"z"
],
"docSource": "",
"docID": 98,
"docDate": "",
"Person": "xxxxxx"
}
Now I need a Java code to Read from this file and Display all the docText, docID Entities. I'm able to retrieve only one Entity.
This is the part of the code im using.
JSONParser jsonParser = new JSONParser();
try {
FileReader fileReader = new FileReader(jsonFilePath);
JSONObject jsonObject = (JSONObject) jsonParser.parse(fileReader);
String docid = (String) jsonObject.get("docText");
System.out.println("DocText: " +docid);
long members = (long) jsonObject.get("docID");
System.out.println("docID: " + members);
JSONArray names = (JSONArray) jsonObject.get("Organization");
Iterator i = names.iterator();
while (i.hasNext()) {
System.out.println("Organization: "+i.next());
I really need this working soon! Thank you
The JSON file has duplicate keys so you probably can't read them all using a standard parser.
From http://www.ietf.org/rfc/rfc4627.txt:
An object structure is represented as a pair of curly brackets surrounding zero or more name/value pairs (or members). A name is a string. A single colon comes after each name, separating the name from the value. A single comma separates a value from a following name. The names within an object SHOULD be unique.
Instead, I'd expect each entity to be a separate object within an array.
AFAIK there is no option to fetch multiple values from a JSON object in a single get...

Converting Nested Json files to CSV in java

{
"Employee": [
{
"empMID": "mock:1",
"comments": [],
"col1": "something",
"contact": [{"address":"2400 waterview", "freetext":true}
],
"gender": "male"
},
{
"empMID": "mock:2",
"comments": [],
"col1": "something",
"contact": [{"address":"2200 waterview", "freetext":true}
],
"gender": "female"
}
],
"cola": false,
"colb": false
}
This is how my Json file looks .I m required to convert this json to a csv .(I m trying to convert a multi-dimesional data to 2d).I m using gson for my purpose.I cannot use gson.fromgson() function to object map with a template because it should be generic .
I know we can use CDL to convert jsonarray to csv format but It wont work in my case .
my csv format looks like
Employee*
empMID,comment.$,contact.address,contact.freetext,gender
mock:1,,2400 waterview,TRUE,male
mock:123,,2200 waterview,TRUE,female
colA#
TRUE
colB#
FALSE
I tried using google-GSON api to convert to this format .But I m not able to convert to this format .I have used * to represent its a json array and # to represent its a primitive type and contact.address to represent nested array inside another json array .I having problem relating this nested structure .I m able to traverse everything recursively like a column. Thanks in advance
public static void main(String[] args) throws IOException{
BufferedReader reader=null;
StringBuilder content=null;
String result=null;
reader = new BufferedReader(new FileReader("temp.json"));
String line = null;
content= new StringBuilder();
while ((line = reader.readLine()) != null) {
content.append(line);
}
reader.close();
result= content.toString();
JsonElement jelement = new JsonParser().parse(result);
printJsonRecursive(jelement);
}
public static void printJsonRecursive(JsonElement jelement){
if(jelement.isJsonPrimitive()){
System.out.println(jelement.getAsString());
return;
}
if(jelement.isJsonArray()){
JsonArray jarray= jelement.getAsJsonArray();
for(int i=0;i<jarray.size();i++){
JsonElement element= jarray.get(i);
printJsonRecursive(element);
}
return;
}
JsonObject jobject= jelement.getAsJsonObject();
Set<Entry<String, JsonElement>> set= jobject.entrySet();
for (Entry<String, JsonElement> s : set) {
printJsonRecursive(s.getValue());
}
}
}
You can achieve this thru reflection if you have a object mapped to the json.
use gson/jackson to convert json to java object
append fields using reflection by iterating the class and get any field you interested in.
append value with reflection by getting value from the target object.
More detail look at my blog post below:
vcfvct.wordpress.com/2015/06/30/converting-nested-json-files-to-csv-in-java-with-reflection/
You are not printing the key. This should fix it.
for (Entry<String, JsonElement> s : set) {
System.out.println(s.getKey()); //Added
printJsonRecursive(s.getValue());
}
You can take care of \ns from here.
EDIT
If you want to print the keys just once for repeating json objects, create a Java bean to hold the data and populate it during your recursion. Once the bean is complete, add a method there to print all the data in the format you want (printing keys only once and so on).
You can use the library json2flat for converting your JSON to CSV.
This library doesn't require any POJO's. It simply takes your JSON as string and returns a 2D representation of it in the format of List<Object[]>.
For example for the JSON:
{
"Employee": [
{
"empMID": "mock:1",
"comments": [],
"col1": "something",
"contact": [{"address":"2400 waterview", "freetext":true}
],
"gender": "male"
},
{
"empMID": "mock:2",
"comments": [],
"col1": "something",
"contact": [{"address":"2200 waterview", "freetext":true}
],
"gender": "female"
}
],
"cola": false,
"colb": false
}
It gives an output:
/cola,/colb,/Employee/empMID,/Employee/col1,/Employee/gender,/Employee/contact/address,/Employee/contact/freetext
,,"mock:1","something",,"2400 waterview",true
,,"mock:2","something",,"2200 waterview",true
false,false,,,,,
/**
* Get separated comlumns used a separator (comma, semi column, tab).
*
* #param headers The CSV headers
* #param map Map of key-value pairs contains the header and the value
*
* #return a string composed of columns separated by a specific separator.
*/
private static String getSeperatedColumns(Set<String> headers, Map<String, String> map, String separator) {
List<String> items = new ArrayList<String>();
for (String header : headers) {
String value = map.get(header) == null ? "" : map.get(header).replaceAll("[\\,\\;\\r\\n\\t\\s]+", " ");
items.add(value);
}
return StringUtils.join(items.toArray(), separator);
}

Parsing generic tag in JSON from GSON

I have a JSON which contains generic values. I tried it by using Maps but couldn't get the results. My problem is the generic tags starts from second level. Here is the JSON I am trying to parse through gson.
{
"success": true,
"status": 200,
"events": {
"Sep 2013": [
{
"artist_id": 1,
"created_at": "2013-05-18T15:21:00Z",
"duration": 2,
"end_time": "2013-09-19T22:00:00Z",
"event_desc": "",
"event_facebook_link": "",
"event_link": "https://www.smtickets.com/marketing/view/1316",
"feature_small": false,
"featured_status": false,
"id": 90,
In this JSON tag after "events" is generic i.e., "Sep 2013".
What I am trying right now is:
public Event event ;
public class Event {
public ArrayList<Map<String, String>> dates;
}
And I'm accessing it:
obj = gson.fromJson(reader, AllShowsActivityData.class);
Can anyone tell me that how can I make a class of dates. So Gson can serialize it. Or is there any other way to iterate first level tag and then I can declare it something like this
Type fooType = new TypeToken<Foo<Bar>>() {}.getType();
gson.toJson(foo, fooType);
You can indeed use a Map, but you're not using it correctly. Note that what you have is a field "events", which is an object that contains a number of pairs string and array of objects:
{ "events": { "Sep 2013": [ {}, {}, ... ], ... } }
To parse your JSON you'd need a class structure like this (in pseudo-code):
class AllShowsActivityData
Map<String, List<Event>> events
class Event
int artist_id
String created_at //parsing as String to simplify...
int duration
...
And then your code to parse:
AllShowsActivityData obj = gson.fromJson(reader, AllShowsActivityData.class);
And finally, if what you want is to access the dates, you'll have all of them in:
Set<String> dates = obj.getEvents().keySet(); //"Sep 2013", ...
Note: in fact, if you only want those dates, you don't even need the class Event and you could use just Map<String, List<Object>> events...

Categories