Currently I am using Avro 1.8.0 to serialize / deserialize objects but facing issue especially for java.util.Map object. Not facing issue with other type of objects.
Sample code here -
class AvroUtils {
public byte[] serialize(Object payload) {
final ByteArrayOutputStream out = new ByteArrayOutputStream();
Schema schema = new ReflectDatumWriter().getData().induce(payload); //---> getting proper map schema as {"type":"map","values":"string"}
JsonEncoder jsonEncoder = EncoderFactory.get().jsonEncoder(schema, out);
final GenericDatumWriter<Object> writer = new GenericDatumWriter(schema);
writer.write(payload, jsonEncoder);
jsonEncoder.flush();
return out.toByteArray();
}
public <R> R deserialize(Object o, Class<R> aClass) {
Schema schema = new ReflectDatumWriter().getData().induce(o); //------> getting error - unable to get schema
final ByteArrayInputStream bin = new ByteArrayInputStream((byte[]) o);
JsonDecoder jsonDecoder = DecoderFactory.get().jsonDecoder(schema, bin);
final GenericDatumReader<R> reader = new GenericDatumReader<>(schema);
return reader.read(null, jsonDecoder);
}
public static void main(String[] args) {
Map<String, Object> map = new HashMap<String, Object>();
map.put("Key1", "Value1");
map.put("Key2", "Value2");
// Serialization
byte[] result = this.serialize(map);
System.out.println("Serialized Data : " + new String(mapDes, "UTF-8"));
// Deserialization
Map<String, Object> mapDes = (Map<String, Object>) this.deserialize(result, Map.class);
System.out.println("Deserialized Data : " + mapDes);
}
}
In deserialize method I am trying to get the schema based on input data but avro is throwing error -
`Exception in thread "main" java.lang.ClassCastException: [B cannot be cast to java.util.Collection
at org.apache.avro.reflect.ReflectData.getArrayAsCollection(ReflectData.java:196)
at org.apache.avro.generic.GenericData.induce(GenericData.java:612)`
Note: At the end both the methods will be placed in different libraries (avro-serializer / avro-deserializer).
Please suggest the best way to get schema in deserialization method.
Thanks.
You get java.lang.ClassCastException: [B cannot be cast to java.util.Collection cause you try to call induce() method with object which is byte array but not a Map.
If you want to serialize a map in one place and deserialize it in the other, you can use a better way:
Schema schema = Schema.createMap(Schema.create(Schema.Type.STRING));
If you do so, you will not need any additional parameters in desirialize method.
Also, GenericDatumWriter can be used only with generic records, so you need a ReflectDatumWriter.
Here is an example of your code with changes:
public class AvroUtils {
public static byte[] serialize(Object payload) throws IOException {
final ByteArrayOutputStream out = new ByteArrayOutputStream();
Schema schema = Schema.createMap(Schema.create(Schema.Type.STRING)); //---> getting proper map schema as {"type":"map","values":"string"}
JsonEncoder jsonEncoder = EncoderFactory.get().jsonEncoder(schema, out);
final DatumWriter<Object> writer = new ReflectDatumWriter<>(schema);
writer.write(payload, jsonEncoder);
jsonEncoder.flush();
return out.toByteArray();
}
public static <R> R deserialize(Object o) throws IOException {
Schema schema = Schema.createMap(Schema.create(Schema.Type.STRING));
JsonDecoder jsonDecoder = DecoderFactory.get().jsonDecoder(schema, new ByteArrayInputStream((byte[]) o));
final DatumReader<R> reader = new ReflectDatumReader<>(schema);
return reader.read(null, jsonDecoder);
}
public static void main(String[] args) throws IOException {
Map<String, Object> map = new HashMap<>();
map.put("Key1", "Value1");
map.put("Key2", "Value2");
// Serialization
byte[] result = serialize(map);
// Deserialization
Map<String, Object> mapDes = deserialize(result);
System.out.println("Deserialized Data : " + mapDes);
}
}
As a result, you'll get something like this:
Deserialized Data : {Key2=Value2, Key1=Value1}
Related
I'm using Jackson to deserialize objects in Java, e.g. here's one of hello world examples:
Foo foo = new ObjectMapper().readValue(jsonFoo, Foo.class);
There's a problem with my json string, though: it may be either
{
"error":{
"code":"404",
"message":"Not Found"
}
}
or
[{fooFields}, {fooFields}]
So usually for the latter case, I would use:
Foo[] foos = new ObjectMapper().readValue(jsonFoo, []Foo.class);
or
Error error = new ObjectMapper().readValue(jsonFoo, Error.class);
How can I try to parse jsonFoo to either Error or [Foo] and return another custom Error object if there's an exception?
Currently, I only managed to come up with a nested try-catch construction which doesn't look really nice.
try {
Foo[] foos = new ObjectMapper().readValue(jsonFoo, []Foo.class);
return new Bar(foos);
catch {
try {
Error error = new ObjectMapper().readValue(jsonFoo, Error.class);
return new Bar(error);
catch (IOException e)... // new Bar(InvalidInputError);
}
return new Bar(InvalidInputError);
Is there a way to rewrite it in a more concise way?
I usually use JsonNode.
public static void main(String[] args) throws IOException {
ObjectMapper objectMapper = new ObjectMapper();
String jsonStr1 = "[{\"key\":\"value\"}]";
JsonNode jsonNode1 = objectMapper.readTree(jsonStr1);
boolean array = jsonNode1.isArray();
String jsonStr2 = "{\"key\":\"value\"}";
JsonNode jsonNode2 = objectMapper.readTree(jsonStr2);
boolean object = jsonNode2.isObject();
}
More information
I'm trying to transform my HashMap into a JSON string using Google's lib. GSON.
I've tried using Jackson's ObjectMapper but it didn't work either. I've tried to put some transient and some things I found on other topics without success.
It is currently returning an error (without any line specified):
//ERROR
java.lang.StackOverflowError: null
at sun.reflect.misc.ReflectUtil.checkPackageAccess(Unknown Source) ~[?:1.8.0_201]
at sun.reflect.misc.ReflectUtil.checkPackageAccess(Unknown Source) ~[?:1.8.0_201]
at sun.reflect.generics.reflectiveObjects.TypeVariableImpl.getGenericDeclaration(Unknown Source) ~[?:1.8.0_201]
at com.google.gson.internal.$Gson$Types.declaringClassOf($Gson$Types.java:427) ~[$Gson$Types.class:?]
at com.google.gson.internal.$Gson$Types.resolveTypeVariable($Gson$Types.java:397) ~[$Gson$Types.class:?]
at com.google.gson.internal.$Gson$Types.resolve($Gson$Types.java:329) ~[$Gson$Types.class:?]
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.getBoundFields(ReflectiveTypeAdapterFactory.java:158) ~[ReflectiveTypeAdapterFactory.class:?]
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.create(ReflectiveTypeAdapterFactory.java:100) ~[ReflectiveTypeAdapterFactory.class:?]
at com.google.gson.Gson.getAdapter(Gson.java:423) ~[Gson.class:?]
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.createBoundField(ReflectiveTypeAdapterFactory.java:115) ~[ReflectiveTypeAdapterFactory.class:?]
// HashMap I need in JSON
public class Storage {
static #Getter #Setter HashMap<World, HashMap<Chunk, List<Particle>>> storage = new HashMap<>();
}
// Class to load/save the data file
public class StorageManager {
public static File folder, config;
private static final Gson GSON = new Gson();
public static void setup(File _folder){
folder = _folder;
}
public static void load() throws IOException {
if(!folder.exists())
folder.mkdir();
config = new File(folder, "particles.json");
if(!config.exists()) {
config.createNewFile();
FileWriter writer = new FileWriter(config, false);
writer.write(GSON.toJson(Storage.getStorage()));
writer.close();
}
FileReader content = new FileReader(config);
Type type = new TypeToken<HashMap<World, HashMap<Chunk, List<Particle>>>>(){}.getType();
Storage.setStorage(GSON.fromJson(content, type));
}
public static void save() throws IOException {
String json = GSON.toJson(Storage.getStorage());
FileWriter writer = new FileWriter(config, false);
writer.write(json);
writer.close();
}
}
I need to be able to load the file, get the JSON and replace my currently empty Hashmap with the data from the file. I also need to save the file with the new Hashmap data.
I've a Dynamodb table with streaming enabled. Also I've created a trigger for this table which calls an AWS Lambda function. Within this lambda function, I'm trying read the new image (Dynamodb item after the modification) from the Dynamodb stream and trying to get the pure json string out of it. My Question is how can i get the pure json string of the DynamoDB item that's been sent over the stream? I'm using the code snippet given below to get the new Image, but I've no clue how to get the json string out of it. Appreciate your help.
public class LambdaFunctionHandler implements RequestHandler<DynamodbEvent, Object> {
#Override
public Object handleRequest(DynamodbEvent input, Context context) {
context.getLogger().log("Input: " + input);
for (DynamodbStreamRecord record : input.getRecords()){
context.getLogger().log(record.getEventID());
context.getLogger().log(record.getEventName());
context.getLogger().log(record.getDynamodb().toString());
Map<String,AttributeValue> currentRecord = record.getDynamodb().getNewImage();
//how to get the pure json string of the new image
//..............................................
}
return "Successfully processed " + input.getRecords().size() + " records.";
}
}
Below is the complete code for converting from Dynamo JSON to Standard JSON:
import com.amazonaws.services.dynamodbv2.document.Item;
import com.amazonaws.services.dynamodbv2.document.internal.InternalUtils;
import com.amazonaws.services.dynamodbv2.model.AttributeValue;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.DynamodbEvent;
import com.amazonaws.services.lambda.runtime.events.DynamodbEvent.DynamodbStreamRecord;
import com.google.gson.Gson;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
/**
* Main Lambda class to receive event stream, parse it to Survey
* and process them.
*/
public class SurveyEventProcessor implements
RequestHandler<DynamodbEvent, String> {
private static final String INSERT = "INSERT";
private static final String MODIFY = "MODIFY";
public String handleRequest(DynamodbEvent ddbEvent, Context context) {
List<Item> listOfItem = new ArrayList<>();
List<Map<String, AttributeValue>> listOfMaps = null;
for (DynamodbStreamRecord record : ddbEvent.getRecords()) {
if (INSERT.equals(record.getEventName()) || MODIFY.equals(record.getEventName())) {
listOfMaps = new ArrayList<Map<String, AttributeValue>>();
listOfMaps.add(record.getDynamodb().getNewImage());
listOfItem = InternalUtils.toItemList(listOfMaps);
}
System.out.println(listOfItem);
try {
// String json = new ObjectMapper().writeValueAsString(listOfItem.get(0));
Gson gson = new Gson();
Item item = listOfItem.get(0);
String json = gson.toJson(item.asMap());
System.out.println("JSON is ");
System.out.println(json);
}catch (Exception e){
e.printStackTrace();
}
}
return "Successfully processed " + ddbEvent.getRecords().size() + " records.";
}
}
In c# you can convert newImage to pure json by use of DynamoDB Document class
using Amazon.DynamoDBv2.DocumentModel;
var streamRecord = dynamoEvent.Records.First();
var jsonResult=Document.FromAttributeMap(streamRecord.Dynamodb.NewImage).ToJson();
and if you want to go further ahead to convert json to object you can use Newtonsoft
using Newtonsoft.Json;
TModel model = JsonConvert.DeserializeObject(jsonResult);
Found a way of doing it cleanly. Using InternalUtils from aws-java-sdk-dynamodb-1.11.15.jar
com.amazonaws.services.dynamodbv2.model.Record streamRecord = ((RecordAdapter) record).getInternalObject();
// get order ready //
OrderFinal order = Utils.mapO2Object(
InternalUtils.toSimpleMapValue(streamRecord.getDynamodb().getNewImage().get("document").getM()),
OrderFinal.class );
Just summarizing the answer of Himanshu Parmar:
Map<String, AttributeValue> newImage = record.getDynamodb().getNewImage();
List<Map<String, AttributeValue>> listOfMaps = new ArrayList<Map<String, AttributeValue>>();
listOfMaps.add(newImage);
List<Item> itemList = ItemUtils.toItemList(listOfMaps);
for (Item item : itemList) {
String json = item.toJSON();
}
For those stuck with a Map<String, ?> where objects are plain Map, but not Attributes value, you can do the following:
Map<String, AttributeValue> dynamoDbAttributes =
objectMapper.convertValue(dynamoDbMap, new TypeReference<Map<String, AttributeValue>>() {});
and then convert this DynamoDB Map into a plain Map (equivalent to the json originally pushed into DynamoDb):
asMap = InternalUtils.toSimpleMapValue(dynamoDbAttributes);
For the ones facing issues with AttributeValue conversion refer the below code:
https://github.com/aws/aws-lambda-java-libs/blob/master/aws-lambda-java-events-sdk-transformer/README.md
Map<String, AttributeValue> stringAttributeValueMap = DynamodbAttributeValueTransformer.toAttributeValueMapV1(dynamodb.getNewImage());
List stringAttributeValueMapList = new ArrayList();
stringAttributeValueMapList.add(stringAttributeValueMap);
List<Item> listOfItem = InternalUtils.toItemList(stringAttributeValueMapList);
Gson gson = new GsonBuilder().setPrettyPrinting().create();
String updatedJSON = gson.toJson(listOfItem.get(0).asMap());
Below is the Method which converts DynamoDB JSON to normal JSON
/**
* Converts DynamoDB JSON to normal JSON.
*
* #param map Input map of String to AttributeValue.
* #return Returns an ObjectNode containing the normal JSON.
*/
public JsonObject toJsonObject(final Map<String, AttributeValue> map) {
final JsonNode result = mapToJsonObject(map);
final ObjectNode objectNode = (ObjectNode) result;
final ObjectMapper objectMapper = new ObjectMapper();
String recordObjectString;
try {
recordObjectString = objectMapper.writeValueAsString(objectNode);
} catch (JsonProcessingException e) {
throw new RuntimeException(e);
}
final JsonParser jsonParser = new JsonParser();
final JsonObject jsonObject = jsonParser.parse(recordObjectString)
.getAsJsonObject();
return jsonObject;
}
So in your Case simple call below method like this
// here record is of type DynamodbStreamRecord
toJsonObject(record.getDynamodb().getNewImage());
This library do the job: dynamoDb-marshaler
var unmarshalJson = require('dynamodb-marshaler').unmarshalJson;
console.log('jsonItem Record: %j', unmarshalJson(record.dynamodb.NewImage));
I am using the Jackson (1.9.x) library to parse JSON into a Map:
ObjectMapper mapper = new ObjectMapper();
Map<String,Object> map = (Map<String,Object>) mapper.readValue(jsonStr, Map.class);
Is there a way to tell the Jackson parser to lowercase all the names of the keys? I tried using a Jackson PropertyNamingStrategy, but that didn't work - it only seems to be useful when it is getting mapped onto some bean, not a Map.
Clarifications:
I do not want to have to precreate beans for the JSON - I only want dynamic Maps
The JSON keys coming in will not be lowercase, but I want all the map keys to be lowercase (see example below)
The JSON is rather large and heavily nested, so regular expression replacements of the incoming JSON or creating a new map manually after the Jackson parsing is not at all desired.
Incoming JSON:
{"CustName":"Jimmy Smith","Result":"foo","CustNo":"1234"}
The Java map would have:
"custname" => "Jimmy Smith"
"result" => "foo"
"custno" => "1234"
[UPDATE]: The answer I gave below doesn't fully solve the problem. Still looking for a solution.
(nb this solution is tested only with Jackson 2)
It's possible to do this by wrapping the JsonParser and simply applying .toLowerCase() to all field names:
private static final class DowncasingParser extends JsonParserDelegate {
private DowncasingParser(JsonParser d) {
super(d);
}
#Override
public String getCurrentName() throws IOException, JsonParseException {
if (hasTokenId(JsonTokenId.ID_FIELD_NAME)) {
return delegate.getCurrentName().toLowerCase();
}
return delegate.getCurrentName();
}
#Override
public String getText() throws IOException, JsonParseException {
if (hasTokenId(JsonTokenId.ID_FIELD_NAME)) {
return delegate.getText().toLowerCase();
}
return delegate.getText();
}
}
You then have to have a custom JsonFactory to apply your wrapper, as in this test:
#Test
public void downcase_map_keys_by_extending_stream_parser() throws Exception {
#SuppressWarnings("serial")
ObjectMapper mapper = new ObjectMapper(new JsonFactory() {
#Override
protected JsonParser _createParser(byte[] data, int offset, int len, IOContext ctxt) throws IOException {
return new DowncasingParser(super._createParser(data, offset, len, ctxt));
}
#Override
protected JsonParser _createParser(InputStream in, IOContext ctxt) throws IOException {
return new DowncasingParser(super._createParser(in, ctxt));
}
#Override
protected JsonParser _createParser(Reader r, IOContext ctxt) throws IOException {
return new DowncasingParser(super._createParser(r, ctxt));
}
#Override
protected JsonParser _createParser(char[] data, int offset, int len, IOContext ctxt, boolean recyclable)
throws IOException {
return new DowncasingParser(super._createParser(data, offset, len, ctxt, recyclable));
}
});
assertThat(
mapper.reader(Map.class)
.with(JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES)
.with(JsonParser.Feature.ALLOW_SINGLE_QUOTES)
.readValue("{CustName:'Jimmy Smith', CustNo:'1234', Details:{PhoneNumber:'555-5555',Result:'foo'} } }"),
equalTo((Map<String, ?>) ImmutableMap.of(
"custname", "Jimmy Smith",
"custno", "1234",
"details", ImmutableMap.of(
"phonenumber", "555-5555",
"result", "foo"
)
)));
}
I figured out one way to do it. Use a org.codehaus.jackson.map.KeyDeserializer, put it in a SimpleModule and register that module with the Jackson ObjectMapper.
import org.codehaus.jackson.map.KeyDeserializer;
import org.codehaus.jackson.map.ObjectMapper;
import org.codehaus.jackson.map.module.SimpleModule;
import org.codehaus.jackson.Version;
// ...
class LowerCaseKeyDeserializer extends KeyDeserializer {
#Override
public Object deserializeKey(String key, DeserializationContext ctx)
throws IOException, JsonProcessingException {
return key.toLowerCase();
}
}
// ...
ObjectMapper mapper = new ObjectMapper();
SimpleModule module = new SimpleModule("LowerCaseKeyDeserializer",
new Version(1,0,0,null));
module.addKeyDeserializer(Object.class, new LowerCaseKeyDeserializer());
mapper.registerModule(module);
Map<String,Object> map =
(Map<String,Object>) mapper.readValue(jsonStr, Map.class);
[UPDATE]: Actually this only will lowercase the top level map keys, but not nested keys.
If the input is:
{"CustName":"Jimmy Smith","CustNo":"1234","Details":{"PhoneNumber": "555-5555", "Result": "foo"}}
The output in the map, unfortunately, will be:
{"custname"="Jimmy Smith", "custno"="1234", "details"={"PhoneNumber"="555-5555", "Result"="foo"}}
With Jackson there isn't any function that will lower the keys in a nested fashion. Atleast not that I know of. I wrote this simple recursive code that does the job.
public JSONObject recursiveJsonKeyConverterToLower(JSONObject jsonObject) throws JSONException
{
JSONObject resultJsonObject = new JSONObject();
#SuppressWarnings("unchecked") Iterator<String> keys = jsonObject.keys();
while(keys.hasNext())
{
String key = keys.next();
Object value = null;
try
{
JSONObject nestedJsonObject = jsonObject.getJSONObject(key);
value = this.recursiveJsonKeyConverterToLower(nestedJsonObject);
}
catch(JSONException jsonException)
{
value = jsonObject.get(key);
}
resultJsonObject.put(key.toLowerCase(), value);
}
return resultJsonObject;
}
Passed String:
String json = "{'Music': 0, 'Books': {'Biology': 1.1, 'Chemistry': {'Inorganic': true, 'Organic': ['Atom', 'Molecule']}}, 'Food': {'Chicken': [1, 2, 3]}}";
Output:
{"music":0,"books":{"biology":1.1,"chemistry":{"inorganic":true,"organic":["Atom","Molecule"]}},"food":{"chicken":[1,2,3]}}
Its also easy to get Map<String, Object> instead of JSONObject (which is what you want) by making resultJsonObject to be of type Map and other little tweaks.
WARNING: for nested JSON, the result would be of type Map<String, Map<String, Object>> depending on how nested is your json object.
public void setKeyName(String systemName){
this.systemName = systemName.toLowerCase();
}
Below is the second JSON message:
{
"ModeL":"Tesla",
"YeaR":"2015"
}
Normally, default ObjectMapper cannot deserialize this message into a CarInfo object. With following configuration, it’s possible:
ObjectMapper objectMapper = new ObjectMapper();
objectMapper.configure(MapperFeature.ACCEPT_CASE_INSENSITIVE_PROPERTIES, true);
CarInfo info = objectMapper.readValue(data, CarInfo.class); //'data' contains JSON string
This deserialization is valid. his deserialization is valid.
https://mtyurt.net/post/jackson-case-insensitive-deserialization.html
I'm using a Jackson library to parse JSON:
{
"employees": [
{ "firstName":"John" , "lastName":"Doe" },
{ "firstName":"Anna" , "lastName":"Smith" },
{ "firstName":"Peter" , "lastName":"Jones" }
]
}
Here is what I'm doing:
public void testJackson() throws IOException {
JsonFactory factory = new JsonFactory();
ObjectMapper mapper = new ObjectMapper(factory);
File from = new File("emp.txt"); // JSON object comes from
TypeReference<HashMap<String, Object>> typeRef = new TypeReference<HashMap<String, Object>>() {};
HashMap<String, Object> o = mapper.readValue(from, typeRef);
Employees employees = new Employees();
employees.employees = (List<Employer>)o.get("employees"); // retrieving list of Employer(s)
employees.showEmployer(1); // choose second to print out to console
System.out.println("Got " + o); // just result of file reading
}
public static class Employees {
public List<Employer> employees;
public void showEmployer(int i) {
System.out.println(employees.get(i));
}
}
public static class Employer {
public String firstName;
public String lastName;
}
The output I'm getting:
{firstName=Anna, lastName=Smith}
Got {employees=[{firstName=John,
lastName=Doe}, {firstName=Anna, lastName=Smith}, {firstName=Peter,
lastName=Jones}]}
But I'm not expecting the elements in my List to be HashMap instances, but Employer objects. This is what Jackson library is supposed to be, isn't it? Could you guys correct me where I am wrong?
I haven't used Jackson, but it seems you're getting what you asked for - a HashMap of String, Object pairs. Perhaps you need to be more explicit in the 'value' portion of the map? Since the value is an array of Employee objects, you might try:
TypeReference<HashMap<String, List<Employee>>> typeRef = new TypeReference<HashMap<String, List<Employee>>>() {};
HashMap<String, List<Employee>> o = mapper.readValue(from, typeRef);