How to efficiently serialize POJO with LocalDate field in Flink? - java

Some of our POJOs contain fields from java.time API (LocalDate, LocalDateTime). When our pipelines are processing them we can see following information in the logs:
org.apache.flink.api.java.typeutils.TypeExtractor - Class class java.time.LocalDate cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance.
As I understand, LocalDate can't be classified as POJO, so instead of using POJO serializer flink falls back to Kryo, which is less efficient. However, since 1.9.0 version flink has dedicated serializers for java.time classes (for example LocalDateSerializer), so I would expect that these serializers would do the job here allowing POJO serializer to be used for our classes. Isn't that the case? If yes, is there any performance hit? If no, what is the optimal solution for such case?
In the project we use Flink 1.11 with Java 1.8.

Due to backwards compatibility, even if a new serializer is being introduced in Flink, it can't be used automatically. However, you can tell Flink to use that for your POJO like this (if you are starting without a previous savepoint using Kryo there):
#TypeInfo(MyClassTypeInfoFactory.class)
public class MyClass {
public int id;
public LocalDate date;
// ...
}
public class MyClassTypeInfoFactory extends TypeInfoFactory<MyClass> {
#Override
public TypeInformation<MyClass> createTypeInfo(
Type t, Map<String, TypeInformation<?>> genericParameters) {
return Types.POJO(MyClass.class, new HashMap<String, TypeInformation<?>>() { {
put("id", Types.INT);
put("date", Types.LOCAL_DATE);
// ...
} } );
}
}
You'll have to provide types for all your POJO's fields as shown, but there are a lot of helpers in the Types class that you can use. Also, by using a TypeInfoFactory like this, you don't have to worry about all the places Flink uses this type - it will always derive the given type information.
If you need to convert an old savepoint to use the new serializer, you may also want to look at Flink's State Processor API.

Related

Is there a Jackson annotation to use a wrapper class during deserialization as well as during serialization for Strings

Hi StackOverflow Community,
I am currently trying to deserialize JSON request bodies provided via Spring Boot #RestController.
The request body contains the following array:
{
...
"productIds": [
"123abc",
"234def"
],
...
}
However, I don't want to deserialize the product IDs into a list of Strings, but rather use a simple wrapper class (for various reasons, including but not limited to additional type safety and validation opportunities). Consequently the class looks like this (Lombok annotations were used to keep the code snippet short):
#Value
#AllArgsConstructor
public class TheRequest {
...
List<ProductId> productIds;
...
}
with ProductId being just a simple wrapper as already said (validation annotations are omitted for the sake of brevity):
#Value
#AllArgsConstructor
public class ProductId{
String id;
}
Looking at Stackoverflow I only found ways to achieve this using rather verbose custom deserialization methods.
However, I am a bit astonished, that Jackson does not provide this functionality out of the box. Consequently it would be great if anyone has any idea if
there is a more elegant way to achieve deserialization of a array of Strings into a List of WrapperObjects, ideally only using Jackson annotations?
there is an elegant way to achieve serialization of such a resulting List of ProductId wrapper objects back into String objects, ideally also using only Jackson annotations? I tried Jacksons #Value but that did not provide the required result.
To me still to verbose but it seems to be a working solution with Jacson 2.14+:
public record PayloadId(String id) {
#JsonCreator(mode = Mode.DELEGATING)
public PayloadId{}
#JsonValue
#Override
public String id() {
return id;
}
}
...and here is the records test https://github.com/FasterXML/jackson-databind/blob/2.14/src/test-jdk14/java/com/fasterxml/jackson/databind/records/RecordCreatorsTest.java

Why doesn't DefaultJackson2JavaTypeMapper.toJavaType() support abstract classes and interfaces as inferred types?

I would like to send messages in JSON format through RabbitMQ from one Java application to another using spring-amqp (1.7.4). The two applications do not share the same domain model classes.
I have a single generic #RabbitListener annotated method on the receiving end, that takes a single argument of type Event, an interface.
I have properly configured Jackson to handle the Event type hierarchy on both sides, yet, spring-rabbit won't convert my JSON message into the proper type because DefaultJackson2JavaTypeMapper does not support inferred abstract classes or interfaces.
If I define a custom JavaTypeMapper that extends DefaultJackson2JavaTypeMapper and does the following, it works perfectly fine:
#Override
public JavaType toJavaType(MessageProperties properties) {
boolean hasInferredTypeHeader = hasInferredTypeHeader(properties);
if (hasInferredTypeHeader && getTypePrecedence().equals(TypePrecedence.INFERRED)) {
// do not check for abstract classes and interfaces here
JavaType targetType = fromInferredTypeHeader(properties);
return targetType;
}
return super.toJavaType(properties);
}
Wouldn't it be better to leave the user in charge of how the conversion is to take place (either using spring-rabbit conventions or using Jackson directly)? Maybe add a flag that enables abstract classes and interfaces support? Is there something I'm missing?
Feel free to open an Improvement JIRA Issue.
Contributions are welcome along with suitable test cases.

Java serialization of non serializable third party class

I am currently developing a web application and I would like to make java objects persistent at the server so that they can be retrieved at any time. Since a database is an overkill for my application, I choose the easiest way of persisting java objects: serialization to xml or to bytes. Unfortunately a big part of the code I use are java classes which I cannot modify and these classes do not implement the interface 'serializable'. What are my options regarding to serializing objects of these classes, as well as other interacting objects of my own classes?
As I said in my comments, I'd go for a SerializationService which would find the proper Serializer<T> for every object you want to save.
Something like :
public interface Serializer<T> {
Serializable toSerializable(T objectToSerialize);
//to build a factory/service around it
boolean canDeserialize(Serializable serializedObject);
T fromSerializable(Serializable serializedObject);
}
And if you want a basic, concrete example : with the quite-common Path :
public class PathSerializer implements Serializer<Path> {
#Override
public Serializable toSerializable(Path objectToSerialize) {
return objectToSerialize.toString();
}
#Override
public Path fromSerializable(Serializable serializedObject) {
if(!canDeserialize(serializedObject)){
throw new IllegalArgumentException("Cannot deserialize this");
}
return Paths.get((String)serializedObject);
}
#Override
public boolean canDeserialize(Serializable serializedObject) {
return serializedObject != null && serializedObject instanceof String;
}
}
You could also very well store POJO containing the name your original object class and the list of parameters needed in its constructor an/or a map of its fields to be able to regenerate your objects by reflection.
It's all up to you and the complexity of your application.
I think JSON would be the go-to solution here. Take Googles GSON library for example. You don't need to annotate your classes, simply write
Gson gson = new Gson();
MyObj obj = gson.fromJson(jsonString);
String json = gson.toJson(obj);
For more general information about the JSON format see the official JSON documentation.
One option would be to extend the classes that you don't have access to, in order to save their internal state, and implement Serializable on those.
More info on this SO question:
Serializing a class variable which does not implement serializable
Besides this, I don't think there is any other option except building some wrappers and serializing the classes manually to XML or JSON.

Order matters with class metadata in Genson - Is there a work-around?

I'm using Genson to serialize + deserialize json in my android app into polymorphic objects. The JSON is coming from a variety of sources though and I can't guarantee that the #class metadata will be the first line item in the json. Walking through the Genson code and writing test cases it looks like the #class metadata has to be the first entry in the dictionary.
Has anyone had luck working around this constraint? Is it time to switch to something else, and if so, what?
public class Message {
Payload payload;
// getters & setters
}
public abstract class Payload {
//
}
public class Notification1 extends Payload {
String text;
// getters & setters
}
public class Notification2 extends Payload {
String otherText
// getters & setters
}
String correctOrder = {"#class":"Message","payload":{"#class":"Notification1","text":"Text"}}
String modifiedOrder = {"#class":"Message","payload":{"text":"Text", "#class":"Notification1"}}
Genson g = Genson.Builder()
.addAlias("Notification1", Notification1.class)
.addAlias("Notification2", Notification2.class)
.useRuntimeType(true)
.useClassMetadata(true)
.useMetadata(true)
.useFields(false)
.useIndentation(false)
.create();
g.deserialize(correctOrder, Message.class) // This works
g.deserialize(modifiedOrder, Message.class) // This barfs with the error: com.owlike.genson.JsonBindingException: Could not deserialize to type class com.ol.communication.messages.Message
Indeed the order matters. This was choosed on purpose, see the remarks in the user guide.
If we allow the #class property anywhere in the json object, then we will have to first deserialize all the json object (and its sub properties obj/arr etc) to an intermediary data structure and then to the correct type.
This would incur additional memory overhead and less speed but greater flexibility, true.
A solution would be to mark classes that are polymorphic (annotation/config in the builder), for whom Genson would search/produce the #class property in the stream. This would allow to have this overhead only for the polymorphic objects in the stream.
At the moment it is not implemented, but I opened an issue. It will come in a future release.
Outside of the technical aspects, I don't think you should have polymorphic logic (or any other fancy stuff) when you are dealing with multiple external API. I mean this kind of features is library specific, so if you don't use the same tool on both sides you can run into troubles. Usually people have a layer that will be used to communicate with the APIs and map the data to YOUR model. If you don't own the code on both ends, I think this would be a good solution on the long term.

Use class name as root key for JSON Jackson serialization

Suppose I have a pojo:
import org.codehaus.jackson.map.*;
public class MyPojo {
int id;
public int getId()
{ return this.id; }
public void setId(int id)
{ this.id = id; }
public static void main(String[] args) throws Exception {
MyPojo mp = new MyPojo();
mp.setId(4);
ObjectMapper mapper = new ObjectMapper();
mapper.configure(SerializationConfig.Feature.WRAP_ROOT_VALUE, true);
System.out.println(mapper.getSerializationConfig().isEnabled(SerializationConfig.Feature.WRAP_ROOT_VALUE));
System.out.println(mapper.writeValueAsString(mp));
}
}
When I serialize using the Jackson ObjectMapper, I just get
true
{"id":4}
but I want
true
{"MyPojo":{"id":4}}
I've searched all over, Jacksons documentation is really unorganized and mostly out of date.
By adding the jackson annotation #JsonTypeInfo in class level you can have the expected output. i just added no-changes in your class.
package com.test.jackson;
import org.codehaus.jackson.map.ObjectMapper;
import org.codehaus.jackson.map.SerializationConfig;
import com.fasterxml.jackson.annotation.JsonTypeInfo;
import com.fasterxml.jackson.annotation.JsonTypeInfo.As;
import com.fasterxml.jackson.annotation.JsonTypeInfo.Id;
#JsonTypeInfo(include=As.WRAPPER_OBJECT, use=Id.NAME)
public class MyPojo {
// Remain same as you have
}
output:
{
"MyPojo": {
"id": 4
}
}
I'm not using jackson, but searching I found this configuration that seems to be what you want: WRAP_ROOT_VALUE
Feature that can be enabled to make root value (usually JSON Object but can be any type) wrapped within a single property JSON object, where key as the "root name", as determined by annotation introspector (esp. for JAXB that uses #XmlRootElement.name) or fallback (non-qualified class name). Feature is mostly intended for JAXB compatibility.
Default setting is false, meaning root
value is not wrapped.
So that you can configure mapper:
objectMapper.configure(SerializationConfig.Feature.WRAP_ROOT_VALUE, true);
I hope it helps you...
Below is a way to achieve this
Map<String, MyPojo> singletonMap = Collections.singletonMap("mypojo", mp);
System.out.println(mapper.writeValueAsString(singletonMap));
Output
{ "mypojo" : { "id" : 4}}
Here the advantage is that we can give our on name for the root key of json object. By the above code, mypojo will be the root key. This approach will be most useful when we use java script template like Mustache.js for iteration of json objects
To achieve this you need to use the JsonTypeInfo annotation on your class and in particular WRAPPER_OBJECT
#JsonTypeName("foo")
#JsonTypeInfo(include = JsonTypeInfo.As.WRAPPER_OBJECT ,use = JsonTypeInfo.Id.NAME)
public class Bar(){
)
There is also a nice annotation for this:
#JsonRootName(value = "my_pojo")
public class MyPojo{
...
}
will generate:
{
"my_pojo" : {...}
}
How about simplest possible solution; just use a wrapper class like:
class Wrapper {
public MyPojo MyPojo;
}
and wrapping/unwrapping in your code?
Beyond this, it would help to know WHY you would like additional json object entry like this? I know this is done by libs that emulate json via xml api (because of impedance between xml and json, due to conversion from xml to json), but for pure json solutions it is usually not needed.
Is it to allow you do figure out what actual type is?
If so, perhaps you could consider enabled polymorphic type information, to let Jackson handle it automatically? (see 1.5 release notes, entry for PTH, for details).
there is another way i used and that worked for me.
I am working with a third party jar, so i have no control for annotations.
So i had to write through bit of hack.
Override: org.codehaus.jackson.map.ser.BeanSerializerFactory.findBeanProperties(SerializationConfig, BasicBeanDescription)
Add your property as below
List<BeanPropertyWriter> props = super.findBeanProperties(config, beanDesc);
BeanPropertyWriter bpw = null;
try {
Class cc = beanDesc.getType().getRawClass();
Method m = cc.getMethod("getClass", null);
bpw = new BeanPropertyWriter("$className", null, null, m, null,true, null);
} catch (SecurityException e) {
// TODO
} catch (NoSuchMethodException e) {
// TODO
}
props.add(bpw);
return props;
This way i get more control and can do other kind of filters too.
#JsonTypeInfo(include=As.WRAPPER_OBJECT, use=Id.NAME)
This annotation works perfectly, as suggested by Arun Prakash. I was trying to get json in this form:
{"Rowset":{"ROW":{"receiptno":"881604199388936","status":"SUCCESS"}}}
but getting like this:
{"ROW":{"receiptno":"881604199388936","status":"SUCCESS"}}
Now that annotation resolved my problem.
I would be interested in hearing the OP's solution for this. I'm having similar issues where my RESTful web service is serializing objects as either XML or JSON for clients. The Javascript clients need to know the wrapping type so that can parse it. Coupling the type to a URI pattern is not an option.
Thanks.
Edit: I noticed that Spring MappingJacksonJsonMarshaller adds the wrapping class when marshalling, so I stepped through the code in debug and noticed that Spring passes in a HashMap with a single key-value pair such that the key is the wrapping name and the value is the object. So, I extended JacksonJaxbJsonProvider, override the writeTo() method and added the following:
HashMap<String, Object> map = new HashMap<String, Object>();
map.put(value.getClass().getSimpleName(), value);
super.writeTo(map, type, genericType, annotations, mediaType, httpHeaders,entityStream);
It's a bit of a hack, but it works nicely.
use withRootName.
objectMapper.writer().withRootName(MyPojo.class.getName());
I have found through experience that it is a good idea for all JSON to include both the backend type (as a string) and the component type used to render it in the front end (if using something like angular or Vue).
The justification for doing this is so that you can process various types with a single set of code.
In vue, for example, having the name of the UI component in the data allows you, among other things, to have a screen rendering a list of children of different types using only a single tag in the parent template.
<component :is="child.componentType"/>.
For backend systems and web services - I prefer to use a single web service processor class that provides logging, auditing and exception handling for all web services by looking up the appropriate processor class based on the incoming payload. That makes the implementation of all my web services look exactly the same (about 3 lines of code), and I get detailed event logging through the lifecycle of the call without writing any per service code to do so.
Having the type wrapping the JSON makes it self documenting. If all you see are the properties, you have no idea what you are looking at until you find the corresponding end point.
If you want to write data driven software, being able to identify what you are processing is a basic requirement.

Categories