I've a Dynamodb table with streaming enabled. Also I've created a trigger for this table which calls an AWS Lambda function. Within this lambda function, I'm trying read the new image (Dynamodb item after the modification) from the Dynamodb stream and trying to get the pure json string out of it. My Question is how can i get the pure json string of the DynamoDB item that's been sent over the stream? I'm using the code snippet given below to get the new Image, but I've no clue how to get the json string out of it. Appreciate your help.
public class LambdaFunctionHandler implements RequestHandler<DynamodbEvent, Object> {
#Override
public Object handleRequest(DynamodbEvent input, Context context) {
context.getLogger().log("Input: " + input);
for (DynamodbStreamRecord record : input.getRecords()){
context.getLogger().log(record.getEventID());
context.getLogger().log(record.getEventName());
context.getLogger().log(record.getDynamodb().toString());
Map<String,AttributeValue> currentRecord = record.getDynamodb().getNewImage();
//how to get the pure json string of the new image
//..............................................
}
return "Successfully processed " + input.getRecords().size() + " records.";
}
}
Below is the complete code for converting from Dynamo JSON to Standard JSON:
import com.amazonaws.services.dynamodbv2.document.Item;
import com.amazonaws.services.dynamodbv2.document.internal.InternalUtils;
import com.amazonaws.services.dynamodbv2.model.AttributeValue;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.DynamodbEvent;
import com.amazonaws.services.lambda.runtime.events.DynamodbEvent.DynamodbStreamRecord;
import com.google.gson.Gson;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
/**
* Main Lambda class to receive event stream, parse it to Survey
* and process them.
*/
public class SurveyEventProcessor implements
RequestHandler<DynamodbEvent, String> {
private static final String INSERT = "INSERT";
private static final String MODIFY = "MODIFY";
public String handleRequest(DynamodbEvent ddbEvent, Context context) {
List<Item> listOfItem = new ArrayList<>();
List<Map<String, AttributeValue>> listOfMaps = null;
for (DynamodbStreamRecord record : ddbEvent.getRecords()) {
if (INSERT.equals(record.getEventName()) || MODIFY.equals(record.getEventName())) {
listOfMaps = new ArrayList<Map<String, AttributeValue>>();
listOfMaps.add(record.getDynamodb().getNewImage());
listOfItem = InternalUtils.toItemList(listOfMaps);
}
System.out.println(listOfItem);
try {
// String json = new ObjectMapper().writeValueAsString(listOfItem.get(0));
Gson gson = new Gson();
Item item = listOfItem.get(0);
String json = gson.toJson(item.asMap());
System.out.println("JSON is ");
System.out.println(json);
}catch (Exception e){
e.printStackTrace();
}
}
return "Successfully processed " + ddbEvent.getRecords().size() + " records.";
}
}
In c# you can convert newImage to pure json by use of DynamoDB Document class
using Amazon.DynamoDBv2.DocumentModel;
var streamRecord = dynamoEvent.Records.First();
var jsonResult=Document.FromAttributeMap(streamRecord.Dynamodb.NewImage).ToJson();
and if you want to go further ahead to convert json to object you can use Newtonsoft
using Newtonsoft.Json;
TModel model = JsonConvert.DeserializeObject(jsonResult);
Found a way of doing it cleanly. Using InternalUtils from aws-java-sdk-dynamodb-1.11.15.jar
com.amazonaws.services.dynamodbv2.model.Record streamRecord = ((RecordAdapter) record).getInternalObject();
// get order ready //
OrderFinal order = Utils.mapO2Object(
InternalUtils.toSimpleMapValue(streamRecord.getDynamodb().getNewImage().get("document").getM()),
OrderFinal.class );
Just summarizing the answer of Himanshu Parmar:
Map<String, AttributeValue> newImage = record.getDynamodb().getNewImage();
List<Map<String, AttributeValue>> listOfMaps = new ArrayList<Map<String, AttributeValue>>();
listOfMaps.add(newImage);
List<Item> itemList = ItemUtils.toItemList(listOfMaps);
for (Item item : itemList) {
String json = item.toJSON();
}
For those stuck with a Map<String, ?> where objects are plain Map, but not Attributes value, you can do the following:
Map<String, AttributeValue> dynamoDbAttributes =
objectMapper.convertValue(dynamoDbMap, new TypeReference<Map<String, AttributeValue>>() {});
and then convert this DynamoDB Map into a plain Map (equivalent to the json originally pushed into DynamoDb):
asMap = InternalUtils.toSimpleMapValue(dynamoDbAttributes);
For the ones facing issues with AttributeValue conversion refer the below code:
https://github.com/aws/aws-lambda-java-libs/blob/master/aws-lambda-java-events-sdk-transformer/README.md
Map<String, AttributeValue> stringAttributeValueMap = DynamodbAttributeValueTransformer.toAttributeValueMapV1(dynamodb.getNewImage());
List stringAttributeValueMapList = new ArrayList();
stringAttributeValueMapList.add(stringAttributeValueMap);
List<Item> listOfItem = InternalUtils.toItemList(stringAttributeValueMapList);
Gson gson = new GsonBuilder().setPrettyPrinting().create();
String updatedJSON = gson.toJson(listOfItem.get(0).asMap());
Below is the Method which converts DynamoDB JSON to normal JSON
/**
* Converts DynamoDB JSON to normal JSON.
*
* #param map Input map of String to AttributeValue.
* #return Returns an ObjectNode containing the normal JSON.
*/
public JsonObject toJsonObject(final Map<String, AttributeValue> map) {
final JsonNode result = mapToJsonObject(map);
final ObjectNode objectNode = (ObjectNode) result;
final ObjectMapper objectMapper = new ObjectMapper();
String recordObjectString;
try {
recordObjectString = objectMapper.writeValueAsString(objectNode);
} catch (JsonProcessingException e) {
throw new RuntimeException(e);
}
final JsonParser jsonParser = new JsonParser();
final JsonObject jsonObject = jsonParser.parse(recordObjectString)
.getAsJsonObject();
return jsonObject;
}
So in your Case simple call below method like this
// here record is of type DynamodbStreamRecord
toJsonObject(record.getDynamodb().getNewImage());
This library do the job: dynamoDb-marshaler
var unmarshalJson = require('dynamodb-marshaler').unmarshalJson;
console.log('jsonItem Record: %j', unmarshalJson(record.dynamodb.NewImage));
Related
I have a JsonObject named "mapping" with the following content:
{
"client": "127.0.0.1",
"servers": [
"8.8.8.8",
"8.8.4.4",
"156.154.70.1",
"156.154.71.1"
]
}
I know I can get the array "servers" with:
mapping.get("servers").getAsJsonArray()
And now I want to parse that JsonArray into a java.util.List...
What is the easiest way to do this?
Definitely the easiest way to do that is using Gson's default parsing function fromJson().
There is an implementation of this function suitable for when you need to deserialize into any ParameterizedType (e.g., any List), which is fromJson(JsonElement json, Type typeOfT).
In your case, you just need to get the Type of a List<String> and then parse the JSON array into that Type, like this:
import java.lang.reflect.Type;
import com.google.gson.reflect.TypeToken;
JsonElement yourJson = mapping.get("servers");
Type listType = new TypeToken<List<String>>() {}.getType();
List<String> yourList = new Gson().fromJson(yourJson, listType);
In your case yourJson is a JsonElement, but it could also be a String, any Reader or a JsonReader.
You may want to take a look at Gson API documentation.
Below code is using com.google.gson.JsonArray.
I have printed the number of element in list as well as the elements in List
import java.util.ArrayList;
import com.google.gson.Gson;
import com.google.gson.JsonArray;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;
public class Test {
static String str = "{ "+
"\"client\":\"127.0.0.1\"," +
"\"servers\":[" +
" \"8.8.8.8\"," +
" \"8.8.4.4\"," +
" \"156.154.70.1\"," +
" \"156.154.71.1\" " +
" ]" +
"}";
public static void main(String[] args) {
// TODO Auto-generated method stub
try {
JsonParser jsonParser = new JsonParser();
JsonObject jo = (JsonObject)jsonParser.parse(str);
JsonArray jsonArr = jo.getAsJsonArray("servers");
//jsonArr.
Gson googleJson = new Gson();
ArrayList jsonObjList = googleJson.fromJson(jsonArr, ArrayList.class);
System.out.println("List size is : "+jsonObjList.size());
System.out.println("List Elements are : "+jsonObjList.toString());
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
OUTPUT
List size is : 4
List Elements are : [8.8.8.8, 8.8.4.4, 156.154.70.1, 156.154.71.1]
I read solution from official website of Gson at here
And this code for you:
String json = "{"client":"127.0.0.1","servers":["8.8.8.8","8.8.4.4","156.154.70.1","156.154.71.1"]}";
JsonObject jsonObject = new Gson().fromJson(json, JsonObject.class);
JsonArray jsonArray = jsonObject.getAsJsonArray("servers");
String[] arrName = new Gson().fromJson(jsonArray, String[].class);
List<String> lstName = new ArrayList<>();
lstName = Arrays.asList(arrName);
for (String str : lstName) {
System.out.println(str);
}
Result show on monitor:
8.8.8.8
8.8.4.4
156.154.70.1
156.154.71.1
I was able to get the list mapping to work with just using #SerializedName for all fields.. no logic around Type was necessary.
Running the code - in step #4 below - through the debugger, I am able to observe that the List<ContentImage> mGalleryImages object populated with the JSON data
Here's an example:
1. The JSON
{
"name": "Some House",
"gallery": [
{
"description": "Nice 300sqft. den.jpg",
"photo_url": "image/den.jpg"
},
{
"description": "Floor Plan",
"photo_url": "image/floor_plan.jpg"
}
]
}
2. Java class with the List
public class FocusArea {
#SerializedName("name")
private String mName;
#SerializedName("gallery")
private List<ContentImage> mGalleryImages;
}
3. Java class for the List items
public class ContentImage {
#SerializedName("description")
private String mDescription;
#SerializedName("photo_url")
private String mPhotoUrl;
// getters/setters ..
}
4. The Java code that processes the JSON
for (String key : focusAreaKeys) {
JsonElement sectionElement = sectionsJsonObject.get(key);
FocusArea focusArea = gson.fromJson(sectionElement, FocusArea.class);
}
Kotlin Extension
for Kotlin developers you can use this extension
inline fun <reified T> String.convertToListObject(): List<T>? {
val listType: Type = object : TypeToken<List<T?>?>() {}.type
return Gson().fromJson<List<T>>(this, listType)
}
Given you start with mapping.get("servers").getAsJsonArray(), if you have access to Guava Streams, you can do the below one-liner:
List<String> servers = Streams.stream(jsonArray.iterator())
.map(je -> je.getAsString())
.collect(Collectors.toList());
Note StreamSupport won't be able to work on JsonElement type, so it is insufficient.
So I have an object with some fields...
protected String name;
protected String relativePathAndFileName;
protected DateTime next_Run;
protected ArrayList<String> hosts;
Which gets serialized to JSON like this:
public void serialize(){
Gson gson = Converters.registerDateTime(new GsonBuilder()).setPrettyPrinting().create();
String json = gson.toJson(this);
try {
FileWriter writer = new FileWriter(this.relativePathAndFileName);
writer.write (json);
writer.close();
} catch (IOException e) {
logger.error("Error while trying to write myAlert to json: ", e);
}
}
Later when I need to read in this json file, I try to do so like this:
try {
for (File f : alertConfigFiles) {
Gson gson = new Gson();
JsonReader reader = new JsonReader(new FileReader(f));
Type type = new TypeToken<Map<String, String>>(){}.getType();
Map<String, String> myMap = gson.fromJson(reader, type);
Alert tempAlert = new Alert(myMap);
myAlerts.add(tempAlert);
logger.debug("Imported: " + f.toString());
}
The error that I'm getting is:
Unhandled exception when trying to import config files:
com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected a string but was BEGIN_ARRAY at line 28 column 13 path $.
The JSON inside the file is something to the effect of:
{
"name": "Logs Per Host - past 24 hours",
"relativePathAndFileName": "./Elk-Reporting/Alerts/logs_per_host24h.json",
"next_Run": "2017-06-07T22:24:56.682-04:00",
"hosts": [
"app-12c",
"app1-18",
"wp-01",
"app-02",
"wp-02",
"cent-04",
"app-06",
"app-05"
]
}
It seems to be choking when it tries to import the ArrayList of hosts, but it was able to write them out without issues.
Can anyone offer some advice on how to get my import working without issues?
try to keep it simple. Using maps and so on, is a way to have issues.
Here is a working code to deserialise / serialise :
package com.rizze.beans.labs.sof;
import static org.junit.Assert.assertTrue;
import org.junit.Test;
import com.google.gson.Gson;
public class SOFGson {
public String json = "{ \"name\": \"Logs Per Host - past 24 hours\", \"relativePathAndFileName\": \"./Elk-Reporting/Alerts/logs_per_host24h.json\", \"next_Run\": \"2017-06-07T22:24:56.682-04:00\", \"hosts\": [ \"bos-qa-app-12c\", \"bos-qa-app1-18\", \"bos-qa-wp-01\", \"bos-lt-app-02\", \"bos-qa-wp-02\", \"bos-dev-cent-04.americanwell.com\", \"bos-qa-app-06\", \"bos-qa-app-05\" ]}";
public class MyObj{
protected String name;
protected String relativePathAndFileName;
protected String next_Run;
protected String[] hosts;
}
#Test
public void test() {
Gson gson = new Gson();
MyObj obj = gson.fromJson(json, MyObj.class);
assertTrue(obj!=null);
assertTrue(obj.hosts.length==8);
System.out.println(gson.toJson(obj));
}
}
here is the class in gist : https://gist.github.com/jeorfevre/7b32a96d4ddc4af68e40bf95f63f2c26
Those two lines seem to be the problem:
Type type = new TypeToken<Map<String, String>>(){}.getType();
Map<String, String> myMap = gson.fromJson(reader, type);
You serialize your object of some specific class. You then deserialize it to type. But your JSON does not fit into a Map. Better do it like this, so you can use your own class.
YourClass myMap = gson.fromJson(reader, YourClass.class);
If you want to use this approach, you might want to change your Java class to hold an array of strings instead of an ArrayList of strings.
Maybe this page helps you a bit. Especially the first case fits your situation.
Another option is a custom Deserialzer as described here.
I have a Json string which has a string message field.
String:
{ "Type" : "Text",
"Subject" : "data received",
"Message" :"{\\"language\\":\\"US\\",\\"data\\":\\"signature\\"}"
}
I want to convert it into the following structure:
Notification.java
public class Notification {
String type;
String subject;
Message message;
}
Message.java
public class Message {
String language;
String data;
}
Is there a way in which I can directly convert the string to a Java object of the above structure? I want to avoid deserializing twice.
You can create a custom Deserializer to deserialize the Message text into Message object and annotate the Message class with #JsonDeserialize:
#JsonDeserialize(using = MessageDeserializer.class)
public class Message {
String language;
String data;
}
public class MessageDeserializer extends JsonDeserializer<Message> {
public MessageDeserializer() {
super();
}
#Override
public Message deserialize(
final JsonParser jsonParser, final DeserializationContext deserializationContext) throws
IOException, JsonProcessingException {
final String messageText = jsonParser.getText();
// parse messageText into Message object
}
}
I am not sure my solution is acceptable since it does require additional explicit call to ObjectMapper to perform deserialization of the string value of Message.
However, this is it is done during the buildup of Notification object and does not require a String message property.
You need to add a ctor with String argument to Message class, where you can deserialize the String into Map and extract the instance propertieds:
public Message(String str) {
try {
#SuppressWarnings("unchecked")
Map<String, Object> map =
(Map<String, Object>)new ObjectMapper().readValue(str, Map.class);
language = map.containsKey("language") ? map.get("language").toString() : null ;
data = map.containsKey("data") ? map.get("data").toString() : null ;
} catch (Exception e) {
e.printStackTrace();
}
}
the new ctor will be called by Jackson when you deserialize a Notification object:
Notification n = (Notification)new ObjectMapper().readValue(reader, Notification.class);
You can convert json string into key-value pairs in Map.You will have to do twice as the Message value is again a json string.Use org.json for JSONObject
Map<String, String> map = new HashMap<String, String>();
JSONObject j = new JSONObject(str);
Iterator<String> keys = j.keys();
while( keys.hasNext() ){
String key = (String)keys.next();
String val = j.getString(key);
map.put(key, val);}
Then retrieve the values by iterating over the keys and pass the values into the class constructor
Then map.get(key) can be used to retrieve the values and will be passed into constructors of the classes.
The org.json library is easy to use:
//Create Json object to parse string
// str is input string
JSONObject obj = new JSONObject(str);
//Create Message
Message mess = new Message();
JSONObject obj2 = new JSONObject(obj.getString("Message"));
mess.data = obj2.getString("data");
mess.language = obj2.getString("language");
//Create Notification
Notification noti = new Notification();
noti.message = mess;
noti.subject = obj.getString("Subject");
noti.type = obj.getString("Type");
I have the following JSON Array as a string like this,
String output = "[{\"Symbol\":\"AMZN\",\"Name\":\"Amazon.com Inc\",\"Exchange\":\"NASDAQ\"},{\"Symbol\":\"VXAZN\",\"Name\":\"CBOE Amazon VIX Index\",\"Exchange\":\"Market Data Express\"}]";
I want to parse it and make a string array like this,
array = {"AMZN Amazon.com Inc NASDAQ", "VXAZN CBOE Amazon VIX Index Market Data Express"};
I came up with the following code to parse the string into a JSON Array using the json-simple-1.1.1.jar library,
import org.json.simple.*;
import org.json.simple.parser.JSONParser;
import org.json.simple.parser.ParseException;
public class RESTclient {
public static void main(String[] args) {
String output = "[{\"Symbol\":\"AMZN\",\"Name\":\"Amazon.com Inc\",\"Exchange\":\"NASDAQ\"},{\"Symbol\":\"VXAZN\",\"Name\":\"CBOE Amazon VIX Index\",\"Exchange\":\"Market Data Express\"}]";
JSONParser parser = new JSONParser();
JSONArray jsonArray = null;
try {
jsonArray = (JSONArray) parser.parse(output);
} catch (ParseException e) {
e.printStackTrace();
}
System.out.println(jsonArray);
}
}
This gives me the following OUTPUT,
[{"Name":"Amazon.com Inc","Exchange":"NASDAQ","Symbol":"AMZN"},{"Name":"CBOE Amazon VIX Index","Exchange":"Market Data Express","Symbol":"VXAZN"}]
Now after this, is there an elegant way to achieve my desired output?
You have to write extra code.
ArrayList<String> stringArray = new ArrayList<String>();
for (Iterator iterator = jsonArray.iterator(); iterator.hasNext();) {
JSONObject map = (JSONObject)iterator.next();
stringArray.add(map.get("Symbol")+" "+map.get("Name")+" "+map.get("Exchange"));
}
//stringArray is want you want
You can do it with Jackson like,
private ObjectMapper mapper = new ObjectMapper();
String[] outputArray = mapper.readValue(jsonString, String[].class);
I have a JsonObject named "mapping" with the following content:
{
"client": "127.0.0.1",
"servers": [
"8.8.8.8",
"8.8.4.4",
"156.154.70.1",
"156.154.71.1"
]
}
I know I can get the array "servers" with:
mapping.get("servers").getAsJsonArray()
And now I want to parse that JsonArray into a java.util.List...
What is the easiest way to do this?
Definitely the easiest way to do that is using Gson's default parsing function fromJson().
There is an implementation of this function suitable for when you need to deserialize into any ParameterizedType (e.g., any List), which is fromJson(JsonElement json, Type typeOfT).
In your case, you just need to get the Type of a List<String> and then parse the JSON array into that Type, like this:
import java.lang.reflect.Type;
import com.google.gson.reflect.TypeToken;
JsonElement yourJson = mapping.get("servers");
Type listType = new TypeToken<List<String>>() {}.getType();
List<String> yourList = new Gson().fromJson(yourJson, listType);
In your case yourJson is a JsonElement, but it could also be a String, any Reader or a JsonReader.
You may want to take a look at Gson API documentation.
Below code is using com.google.gson.JsonArray.
I have printed the number of element in list as well as the elements in List
import java.util.ArrayList;
import com.google.gson.Gson;
import com.google.gson.JsonArray;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;
public class Test {
static String str = "{ "+
"\"client\":\"127.0.0.1\"," +
"\"servers\":[" +
" \"8.8.8.8\"," +
" \"8.8.4.4\"," +
" \"156.154.70.1\"," +
" \"156.154.71.1\" " +
" ]" +
"}";
public static void main(String[] args) {
// TODO Auto-generated method stub
try {
JsonParser jsonParser = new JsonParser();
JsonObject jo = (JsonObject)jsonParser.parse(str);
JsonArray jsonArr = jo.getAsJsonArray("servers");
//jsonArr.
Gson googleJson = new Gson();
ArrayList jsonObjList = googleJson.fromJson(jsonArr, ArrayList.class);
System.out.println("List size is : "+jsonObjList.size());
System.out.println("List Elements are : "+jsonObjList.toString());
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
OUTPUT
List size is : 4
List Elements are : [8.8.8.8, 8.8.4.4, 156.154.70.1, 156.154.71.1]
I read solution from official website of Gson at here
And this code for you:
String json = "{"client":"127.0.0.1","servers":["8.8.8.8","8.8.4.4","156.154.70.1","156.154.71.1"]}";
JsonObject jsonObject = new Gson().fromJson(json, JsonObject.class);
JsonArray jsonArray = jsonObject.getAsJsonArray("servers");
String[] arrName = new Gson().fromJson(jsonArray, String[].class);
List<String> lstName = new ArrayList<>();
lstName = Arrays.asList(arrName);
for (String str : lstName) {
System.out.println(str);
}
Result show on monitor:
8.8.8.8
8.8.4.4
156.154.70.1
156.154.71.1
I was able to get the list mapping to work with just using #SerializedName for all fields.. no logic around Type was necessary.
Running the code - in step #4 below - through the debugger, I am able to observe that the List<ContentImage> mGalleryImages object populated with the JSON data
Here's an example:
1. The JSON
{
"name": "Some House",
"gallery": [
{
"description": "Nice 300sqft. den.jpg",
"photo_url": "image/den.jpg"
},
{
"description": "Floor Plan",
"photo_url": "image/floor_plan.jpg"
}
]
}
2. Java class with the List
public class FocusArea {
#SerializedName("name")
private String mName;
#SerializedName("gallery")
private List<ContentImage> mGalleryImages;
}
3. Java class for the List items
public class ContentImage {
#SerializedName("description")
private String mDescription;
#SerializedName("photo_url")
private String mPhotoUrl;
// getters/setters ..
}
4. The Java code that processes the JSON
for (String key : focusAreaKeys) {
JsonElement sectionElement = sectionsJsonObject.get(key);
FocusArea focusArea = gson.fromJson(sectionElement, FocusArea.class);
}
Kotlin Extension
for Kotlin developers you can use this extension
inline fun <reified T> String.convertToListObject(): List<T>? {
val listType: Type = object : TypeToken<List<T?>?>() {}.type
return Gson().fromJson<List<T>>(this, listType)
}
Given you start with mapping.get("servers").getAsJsonArray(), if you have access to Guava Streams, you can do the below one-liner:
List<String> servers = Streams.stream(jsonArray.iterator())
.map(je -> je.getAsString())
.collect(Collectors.toList());
Note StreamSupport won't be able to work on JsonElement type, so it is insufficient.