We Keep Coding

Jackson deserialization SNS message error MismatchedInputException

I'm coding a functionality of handling callbacks from Amazon Simple Email Service via SNS HTTP requests. I would like to parse message provided by Amazon to local object structure. Problem is that SNS is wrapping JSON message into String and it could not be parsed by Jackson. I'm getting an error:
com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot construct instance of `xxx.email.domain.aws.ses.Notification` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value ('{"notificationType":"Delivery","mail":{"timestamp":"2019-10-02T14:43:14.570Z" ... next values of the message ... }}')
Entire message from SNS looks like this one:
{
"Type" : "Notification",
"MessageId" : "4944xxxx-711d-57d4-91b8-8215cxxxxx",
"TopicArn" : "arn:aws:sns:eu-west-1:...",
"Message" : "{\"notificationType\":\"Delivery\",\"mail\":{\"timestamp\":\"2019-10-02T14:43:14.570Z\", ... next values of the message ... },\"delivery\":{\"timestamp\":\"2019-10-02T14:43:16.030Z\", ... next values of the message ... }}",
"Timestamp" : "2019-10-02T14:43:16.062Z",
"SignatureVersion" : "1",
"Signature" : "signature base64",
"SigningCertURL" : "cert url",
"UnsubscribeURL" : "unsubscribe url"
}
My actual local structure looks like this:
#Data
#JsonNaming(PropertyNamingStrategy.UpperCamelCaseStrategy.class)
public class MessageWrapper {
private String type;
private String messageId;
private String topicArn;
private Notification message;
private Date timestamp;
private String signatureVersion;
private String signature;
private String signingCertURL;
private String unsubscribeURL;
}
#Data
public class Notification {
private String notificationType;
private Mail mail;
}
#Data
public class Mail {
private String messageId;
private String source;
private String sourceArn;
private String sourceIp;
private String sendingAccountId;
private String[] destination;
}
I'm looking for some way to tell Jackson that Message should be extracted from a String and treated as a normal JSON.
Edit
deserialization
private MessageWrapper deserializeMessage(String message) throws IOException {
return new ObjectMapper().readValue(message, MessageWrapper.class);
}

I think to solve this you'll need a custom deserializer for Notification field in MessageWrapper class as well as one for the Mail field in the Notification class the like the following:
public class NotificationDeserializer extends JsonDeserializer<Notification> {
#Override
public Notification deserialize(JsonParser p, DeserializationContext ctxt)
throws IOException, JsonProcessingException {
String text = p.getText();
return new ObjectMapper().readValue(text, Notification.class);
}
}
public class MailDeserializer extends JsonDeserializer<Mail> {
#Override
public Mail deserialize(JsonParser p, DeserializationContext ctxt)
throws IOException, JsonProcessingException {
String text = p.getText();
return new ObjectMapper().readValue(text, Mail.class);
}
}
With some annotations on your classes like the following:
#Data
#JsonNaming(PropertyNamingStrategy.UpperCamelCaseStrategy.class)
public class MessageWrapper {
private String type;
private String messageId;
private String topicArn;
#JsonDeserialize(using = NotificationDeserializer.class)
private Notification message;
private Date timestamp;
private String signatureVersion;
private String signature;
private String signingCertURL;
private String unsubscribeURL;
}
#Data
public class Notification {
private String notificationType;
#JsonDeserialize(using = MailDeserializer.class)
private Mail mail;
}
#Data
public class Mail {
private String messageId;
private String source;
private String sourceArn;
private String sourceIp;
private String sendingAccountId;
private String[] destination;
}
EDIT 1
The MailDeserializer isn't actually needed. The NotificationDeserializer alone takes care of the issue.
EDIT 2
Using a new ObjectMapper in the custom deserializer is a must.

message property is of type Notification and Jackson expects JSON Object not string value. In that case you can create custom deserialiser or implement general solution with some kind of loop back implementation. If given payload is not a JSON Object read it as a String and invoke deserialisation again with this String.
To avoid StackOverflowError you need to use another instance of ObjectMapper or use BeanDeserializerModifier to keep BeanDeserializer instance and use it where JSON Object is encountered. Simple example could look like below:
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.JsonToken;
import com.fasterxml.jackson.databind.BeanDescription;
import com.fasterxml.jackson.databind.DeserializationConfig;
import com.fasterxml.jackson.databind.DeserializationContext;
import com.fasterxml.jackson.databind.DeserializationFeature;
import com.fasterxml.jackson.databind.JsonDeserializer;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.PropertyNamingStrategy;
import com.fasterxml.jackson.databind.annotation.JsonNaming;
import com.fasterxml.jackson.databind.deser.BeanDeserializer;
import com.fasterxml.jackson.databind.deser.BeanDeserializerBase;
import com.fasterxml.jackson.databind.deser.BeanDeserializerModifier;
import com.fasterxml.jackson.databind.module.SimpleModule;
import com.fasterxml.jackson.databind.node.TextNode;
import lombok.Data;
import lombok.ToString;
import java.io.File;
import java.io.IOException;
import java.util.Collections;
import java.util.Date;
import java.util.Objects;
import java.util.Set;
public class JsonApp {
public static void main(String[] args) throws Exception {
File jsonFile = new File("./resource/test.json").getAbsoluteFile();
SimpleModule loopBackModule = new SimpleModule();
loopBackModule.setDeserializerModifier(new LoopBackBeanDeserializerModifier(Collections.singleton(Notification.class)));
ObjectMapper mapper = new ObjectMapper();
mapper.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES);
mapper.registerModule(loopBackModule);
MessageWrapper wrapper = mapper.readValue(jsonFile, MessageWrapper.class);
System.out.println(wrapper.getMessage());
}
}
class LoopBackBeanDeserializerModifier extends BeanDeserializerModifier {
private final Set<Class> allowedClasses;
LoopBackBeanDeserializerModifier(Set<Class> allowedClasses) {
this.allowedClasses = Objects.requireNonNull(allowedClasses);
}
#Override
public JsonDeserializer<?> modifyDeserializer(DeserializationConfig config, BeanDescription beanDesc, JsonDeserializer<?> deserializer) {
if (allowedClasses.contains(beanDesc.getBeanClass())) {
return new LoopBackBeanDeserializer<>((BeanDeserializerBase) deserializer);
}
return deserializer;
}
}
class LoopBackBeanDeserializer<T> extends BeanDeserializer {
private final BeanDeserializerBase baseDeserializer;
protected LoopBackBeanDeserializer(BeanDeserializerBase src) {
super(src);
this.baseDeserializer = src;
}
#Override
public T deserialize(JsonParser p, DeserializationContext ctxt) throws IOException {
// if first token is VALUE_STRING we should read it as String and
// run deserialization process again based on this String.
if (p.currentToken() == JsonToken.VALUE_STRING) {
return (T) ((ObjectMapper) p.getCodec()).readValue(p.getText(), _valueClass);
}
// vanilla bean deserialization
return (T) baseDeserializer.deserialize(p, ctxt);
}
}
POJO model is the same. You just need to list classes for which you expect some problems and loop-back mechanism will work for them.

Jackson: Unresolved forward references when trying to deserialize JSON into java objects

I have been stuck on a problem with Jackson for quite some time now and I'm eager to solve it. I have a JSON that has objects in it that reference to each other with ID's and I need to deserialise it into objects, but I keep getting Unresolved forward references exception when trying to do so.
I have tried using the Jackson annotation #JsonIdentityInfo above classes in question but this did not yield any results.
Example of the JSON:
{
"customer": [
{
"id": 1,
"name": "Jane Gallow",
"age": 16
},
{
"id": 2,
"name": "John Sharp",
"age": 20
},
],
"shoppingcart": [
{
"id": 1,
"customer": 2,
"orderDate": "2015-02-19"
}
]
}
Customer class
#JsonIdentityInfo(generator= ObjectIdGenerators.PropertyGenerator.class, property="id",scope = Customer.class)
#JsonIdentityReference(alwaysAsId = true)
public class Customer {
private int id = -1;
private String name;
private int age;
//getters, setters
}
ShoppingCart class
<!-- language: java -->
#JsonIdentityInfo(generator= ObjectIdGenerators.PropertyGenerator.class, property="id",scope = ShoppingCart.class)
public class ShoppingCart {
private int id = -1;
private Customer customer;
#JsonDeserialize(using = LocalDateDeserializer.class)
#JsonSerialize(using = LocalDateSerializer.class)
private LocalDate orderDate = LocalDate.now();
//getters, setters
}
I expect Jackson to give me a ShoppinCart object, that has a reference to Customer object in with id of 2 (John Sharp in this case). But I just can't get it to work and it gives me "main" com.fasterxml.jackson.databind.deser.UnresolvedForwardReference when I try to read from JSON with ObjectMapper.readValue() method.

I had the same problem with Unresolved Forward Reference during deserialization. I used the example provided on https://www.logicbig.com/tutorials/misc/jackson/json-identity-reference.html to check and solve my problem.
There is a very simple class:
public class Order {
private int orderId;
private List<Integer> itemIds;
private Customer customer;
....
}
I added #JsonCreator static factory method to the Customer class and it did not cause any problems.
Now when I add any kind of #JsonCreator (a static method or a constructor) to the Order class I immediately get unresolved forward reference. What is interesting, when I add the following constructor, with only #JsonPropery annotations present:
public Order(#JsonProperty("orderId") int orderId, #JsonProperty("itemIds") List<Integer> itemIds, #JsonProperty("customer") Customer customer) {
this.orderId = orderId;
this.itemIds = itemIds;
this.customer = customer;
}
it also causes unresolved forward reference.
To solve the problem, for Order class you MUST create parameterless constructor (can be private). Otherwise you will get:
Exception in thread "main" com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of com.logicbig.example.Order (no Creators, like default construct, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
and there MUST not be any of #JsonCreator annotated methods and constructors, or constructors with #JsonProperty annotations.
And that is all.

You should use JsonIdentityReference on property.
#JsonIdentityInfo(generator = ObjectIdGenerators.PropertyGenerator.class, property = "id", scope = ShoppingCart.class)
class ShoppingCart {
private int id = -1;
#JsonIdentityReference
private Customer customer;
private LocalDate orderDate;
// getters, setters, toString
}
#JsonIdentityInfo(generator = ObjectIdGenerators.PropertyGenerator.class, property = "id", scope = Customer.class)
class Customer {
private int id = -1;
private String name;
private int age;
// getters, setters, toString
}
Simple example:
import com.fasterxml.jackson.annotation.JsonIdentityInfo;
import com.fasterxml.jackson.annotation.JsonIdentityReference;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.annotation.ObjectIdGenerators;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
import java.io.File;
import java.time.LocalDate;
import java.util.List;
public class JsonApp {
public static void main(String[] args) throws Exception {
File jsonFile = new File("./resource/test.json").getAbsoluteFile();
ObjectMapper mapper = new ObjectMapper();
mapper.registerModule(new JavaTimeModule());
mapper.enable(SerializationFeature.INDENT_OUTPUT);
System.out.println(mapper.readValue(jsonFile, Pojo.class));
}
}
For your JSON prints:
Pojo{customers=[Customer{id=1, name='Jane Gallow', age=16}, Customer{id=2, name='John Sharp', age=20}], shoppingCarts=[ShoppingCart{id=1, customer=Customer{id=2, name='John Sharp', age=20}, orderDate=2015-02-19}]}
See also:
Jackson JSON - Using #JsonIdentityReference to always serialize a
POJO by id
baeldung.com - More Jackson Annotations

Jackson - deserialize inner list of objects to list of one higher level

With Spring Boot and Jackson, how can I deserialize a wrapped/inner list into a list directly in the outer level?
For example, I have:
{
"transaction": {
"items": {
"item": [
{
"itemNumber": "193487654",
"itemDescription": "Widget",
"itemPrice": "599.00",
"itemQuantity": "1",
"itemBrandName": "ACME",
"itemCategory": "Electronics",
"itemTax": "12.95"
},
{
"itemNumber": "193487654",
"itemDescription": "Widget",
"itemPrice": "599.00",
"itemQuantity": "1",
"itemBrandName": "ACME",
"itemCategory": "Electronics",
"itemTax": "12.95"
}
]
},
...
}
}
In the JSON, item is a list under items; but I want to parse it as a list named items, directly under transaction, instead of defining a DTO Items which contains a list named item.
Is this possible? How to define this DTO Item?
public class TrasactionDTO {
private List<Item> items;
...
}
public class Item {
}
This question is similar but does not solve the problem.
Deserialize wrapped list using Jackson

We need to implement custom deserialiser. Because we want to skip one inner field our implementation should:
{ - skip start object
"any_field_name" - skip any field name. We assume that we have only one inner field.
[{}, ..., {}] - use default deserialiser for List.
} - skip end object
Using above concept implementation should be easy:
public class InnerListDeserializer extends JsonDeserializer<List> implements ContextualDeserializer {
private final JavaType propertyType;
public InnerListDeserializer() {
this(null);
}
public InnerListDeserializer(JavaType propertyType) {
this.propertyType = propertyType;
}
#Override
public List deserialize(JsonParser p, DeserializationContext context) throws IOException {
p.nextToken(); // SKIP START_OBJECT
p.nextToken(); // SKIP any FIELD_NAME
List list = context.readValue(p, propertyType);
p.nextToken(); // SKIP END_OBJECT
return list;
}
#Override
public JsonDeserializer<?> createContextual(DeserializationContext context, BeanProperty property) {
return new InnerListDeserializer(property.getType());
}
}
Let's assume we have JSON payload like this:
{
"transaction": {
"items": {
"item": [
{
"itemNumber": "193487654",
"itemDescription": "Widget",
"itemPrice": "599.00",
"itemQuantity": "1",
"itemBrandName": "ACME",
"itemCategory": "Electronics",
"itemTax": "12.95"
},
{
"itemNumber": "193487654",
"itemDescription": "Widget",
"itemPrice": "599.00",
"itemQuantity": "1",
"itemBrandName": "ACME",
"itemCategory": "Electronics",
"itemTax": "12.95"
}
]
},
"name": "Pickle Rick"
}
}
Above JSON we can map to below POJO classes:
#JsonRootName("transaction")
public class Transaction {
private String name;
private List<Item> items;
#JsonDeserialize(using = InnerListDeserializer.class)
public List<Item> getItems() {
return items;
}
// getters, setters, toString
}
public class Item {
private String itemNumber;
// getters, setters, toString
}
To show it works for many different models let's introduce one more JSON payload:
{
"product": {
"products": {
"innerArray": [
{
"id": "1234"
}
]
}
}
}
and two more POJO classes:
#JsonRootName("product")
class Product {
private List<ProductItem> products;
#JsonDeserialize(using = InnerListDeserializer.class)
public List<ProductItem> getProducts() {
return products;
}
// getters, setters, toString
}
class ProductItem {
private String id;
// getters, setters, toString
}
Now we can test our solution:
import com.fasterxml.jackson.annotation.JsonRootName;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.databind.BeanProperty;
import com.fasterxml.jackson.databind.DeserializationContext;
import com.fasterxml.jackson.databind.DeserializationFeature;
import com.fasterxml.jackson.databind.JavaType;
import com.fasterxml.jackson.databind.JsonDeserializer;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
import com.fasterxml.jackson.databind.deser.ContextualDeserializer;
import java.io.File;
import java.io.IOException;
import java.util.List;
public class JSoupTest {
public static void main(String[] args) throws Exception {
ObjectMapper mapper = new ObjectMapper();
mapper.enable(SerializationFeature.INDENT_OUTPUT);
mapper.enable(DeserializationFeature.UNWRAP_ROOT_VALUE);
mapper.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES);
File jsonFile = new File("Path to 1-st JSON").getAbsoluteFile();
File jsonFile1 = new File("Path to 2-nd JSON").getAbsoluteFile();
System.out.println(mapper.readValue(jsonFile, Transaction.class));
System.out.println(mapper.readValue(jsonFile1, Product.class));
}
}
Above example prints:
Transaction{items=[Item{itemNumber=193487654}, Item{itemNumber=193487654}], name='Pickle Rick'}
Product{products=[ProductItem{id='1234'}]}
For more info read:
Custom Jackson Deserializer Getting Access to Current Field Class
Getting Started with Custom Deserialization in Jackson
Jackson Exceptions – Problems and Solutions
Jackson UNWRAP_ROOT_VALUE
Configuring ObjectMapper in Spring

It seems that #JsonUnwrapped is what I need.
https://www.baeldung.com/jackson-annotations
#JsonUnwrapped defines values that should be unwrapped/flattened when serialized/deserialized.
Let's see exactly how that works; we'll use the annotation to unwrap the property name:
public class UnwrappedUser {
public int id;
#JsonUnwrapped
public Name name;
public static class Name {
public String firstName;
public String lastName;
}
}
Let's now serialize an instance of this class:
#Test
public void whenSerializingUsingJsonUnwrapped_thenCorrect()
throws JsonProcessingException, ParseException {
UnwrappedUser.Name name = new UnwrappedUser.Name("John", "Doe");
UnwrappedUser user = new UnwrappedUser(1, name);
String result = new ObjectMapper().writeValueAsString(user);
assertThat(result, containsString("John"));
assertThat(result, not(containsString("name")));
}
Here's how the output looks like – the fields of the static nested class unwrapped along with the other field:
{
"id":1,
"firstName":"John",
"lastName":"Doe"
}
So, it should be something like:
public class TrasactionDTO {
private List<Item> items;
...
}
public static class Item {
#JsonUnwrapped
private InnerItem innerItem;
...
}
public static class InnerItem {
private String itemNumber;
...
}

You can use a Map to represent the intermediate Items object.
Given this example (all fields public just for demonstration purposes):
public class Item {
public String itemNumber, itemDescription, itemPrice, itemQuantity, itemBrandName, itemCategory, itemTax;
}
...you can achieve what you want in two ways:
1. By using a constructor:
public class TransactionDTO {
private List<Item> items;
#JsonCreator
public TransactionDTO(#JsonProperty("items") final Map<String, List<Item>> items) {
this.items = items.get("item");
}
}
2. By using a setter:
public class TransactionDTO {
private List<Item> items;
public void setItems(final Map<String, List<Item>> items) {
this.items = items.get("item");
}
}

GSON failing to deserialize JSON with nested array of objects?

I have a pretty standard use-case here:
{
"title": "my awesome title",
"description": "all the awesome things",
"theThings": [
{
"thing": "coolThing1",
"association-type": "thing"
},
{
"thing": "coolThing2",
"association-type": "thing"
}
]
}
I'm utilizing the following class structure to use with GSON:
package things;
import java.io.Serializable;
import java.util.List;
import com.google.gson.annotations.Expose;
import com.google.gson.annotations.SerializedName;
public class ThingsPOJO implements Serializable
{
#SerializedName("title")
#Expose
public String title = "";
#SerializedName("description")
#Expose
public String description = "";
#SerializedName("theThings")
#Expose
public List<TheThing> theThings = null;
private class TheThing implements Serializable
{
#SerializedName("thing")
#Expose
public String thing = "";
#SerializedName("association-type")
#Expose
public String associationType = "";
}
}
title and description are good to go, but the collection of objects is null. I'm testing using that example payload from above. I'm using the fromJson call provided by GSON.

Why is the Jackson Object Mapper unable to deserialize this nested JSON?

I am in the process of updating some code to hit a new API which returns the same type of data but in a different JSON format. This is a sample of the return from a request:
{
"code": 200,
"message": "OK",
"data": [
{
"start_time": "2017-09-20T00:00:00.000-04:00",
"end_time": "2017-09-21T00:00:00.000-04:00",
"value": 8612.637512577203
},
{
"start_time": "2017-09-21T00:00:00.000-04:00",
"end_time": "2017-09-22T00:00:00.000-04:00",
"value": 8597.89155775999
},
{
"start_time": "2017-09-22T00:00:00.000-04:00",
"end_time": "2017-09-23T00:00:00.000-04:00",
"value": 24584.603303989123
}
],
"meta": {
"space_id": "e1c38410-f912-4ae3-9db9-10a1ad1e3bf5",
"channel_id": 1,
"aggregation_type": "period_beginning",
"pids_count": 1,
"timezone": "America/New_York"
}
}
I want to ignore the code and message properties, and put the array of data into a list within a map, with the key being the "space_id" property (Map<String, List<Reading>>) for compatibility with the old implementation. Here is the POJO that I have created to contain the deserialized object:
#JsonIgnoreProperties({"code", "message", "meta"})
public class GetReadingsResult {
#JsonIgnore
private Map<String, List<Reading>> readings;
#JsonIgnore
public GetReadingsResult(Map<String, List<Reading>> readings) {
this.readings = readings;
}
#JsonCreator
public GetReadingsResult(#JsonProperty("data")Reading[] data, #JsonProperty("space_id")String spaceId) {
this.readings.put(spaceId, Arrays.asList(data));
}
//...other getters and setters...
}
In my test I am calling readValue on a test JSON file and getting the following error:
com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class model.GetReadingsResult] value failed: null
What is the proper way to set up/annotate my POJO to deal with this nested file?

This is the correct way to implement.
Please pay attention to object point of view of all thing:
These are the pojos:
import com.fasterxml.jackson.annotation.JsonInclude;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.annotation.JsonPropertyOrder;
#JsonInclude(JsonInclude.Include.NON_NULL)
#JsonPropertyOrder({
"start_time",
"end_time",
"value"
})
public class Datum {
#JsonProperty("start_time")
public String startTime;
#JsonProperty("end_time")
public String endTime;
#JsonProperty("value")
public Double value;
}
import com.fasterxml.jackson.annotation.JsonInclude;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.annotation.JsonPropertyOrder;
#JsonInclude(JsonInclude.Include.NON_NULL)
#JsonPropertyOrder({
"space_id",
"channel_id",
"aggregation_type",
"pids_count",
"timezone"
})
public class Meta {
#JsonProperty("space_id")
public String spaceId;
#JsonProperty("channel_id")
public Integer channelId;
#JsonProperty("aggregation_type")
public String aggregationType;
#JsonProperty("pids_count")
public Integer pidsCount;
#JsonProperty("timezone")
public String timezone;
}
This is the wrapper root Object:
import java.util.List;
import com.fasterxml.jackson.annotation.JsonInclude;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.annotation.JsonPropertyOrder;
#JsonInclude(JsonInclude.Include.NON_NULL)
#JsonPropertyOrder({
"code",
"message",
"data",
"meta"
})
public class ExampleStack {
#JsonProperty("code")
public Integer code;
#JsonProperty("message")
public String message;
#JsonProperty("data")
public List<Datum> data = null;
#JsonProperty("meta")
public Meta meta;
}
And this is the working example:
import com.fasterxml.jackson.databind.ObjectMapper;//<--IMPORTANT!
public class TestJacksonObject {
public static void main(String[] args) {
ObjectMapper mapper = new ObjectMapper();
ExampleStack stack = null;
try {
stack = mapper .readValue( new FileInputStream(new File("C://test.json")) , ExampleStack.class);
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(stack.message);//<--Do whatever you want...
.....
If it's too hard to produce all these classes, which is actually tedious I suggest you to autoproduce them online through this useful site:
Json2Pojo
Hope it helps you out!