We Keep Coding

Kinesis Data Analytics - Flink state serializer incompatible after recovering from Snapshot

We have our Flink application(version 1.13.2) deployed on AWS KDA. The strategy is that we do not want the application to stop at all, so we always recover the application from a snapshot when updating the jar with new changes.
Recently, we found a problem where a lower-level POJO class is corrupted. It contains a few getters and setters with wrong namings. This early mistake essentially hinders us from adding the POJO class with new fields. So we decided to rename the getter/setter directly. But it led us to the following exception after updating the application.
org.apache.flink.util.StateMigrationException: The new state serializer (org.apache.flink.api.common.typeutils.base.ListSerializer#46c65a77) must not be incompatible with the old state serializer (org.apache.flink.api.common.typeutils.base.ListSerializer#30c9146c).
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.updateRestoredStateMetaInfo(RocksDBKeyedStateBackend.java:704) at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.tryRegisterKvStateInformation(RocksDBKeyedStateBackend.java:624)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.createInternalState(RocksDBKeyedStateBackend.java:837) at org.apache.flink.runtime.state.KeyedStateFactory.createInternalState(KeyedStateFactory.java:47) at org.apache.flink.runtime.state.ttl.TtlStateFactory.createStateAndWrapWithTtlIfEnabled(TtlStateFactory.java:71)
at org.apache.flink.runtime.state.AbstractKeyedStateBackend.getOrCreateKeyedState(AbstractKeyedStateBackend.java:301) at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.getOrCreateKeyedState(StreamOperatorStateHandler.java:315) at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.getOrCreateKeyedState(AbstractStreamOperator.java:494) at org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.open(WindowOperator.java:243) at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:442)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:582) at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55) at org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(StreamTask.java:562)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647) at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:537) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:764) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:571)
at java.base/java.lang.Thread.run(Thread.java:829)
As far as we understand, the failure happens specifically in the 2 CoGroup functions we implemented. They are both consuming the corrupted POJO class nested in another POJO class, Session. A code snippet of the Cogroup function is shown below. BTW, we are using google guava list here, not sure if it causes list serializer problem.
public class OutputCoGroup extends CoGroupFunction<Session, Event, OutputSession> {
#Override
public void coGroup(Iterable<Session> sessions, Iterable<Event> events,
Collector<OutputSession> collector) throws Exception {
// we are using google guava list here, not sure if it causes list serializer problem
if (Lists.newArrayList(sessions).size() > 0) {
...
if (events.iterator().hasNext()) {
List<Event> eventList = Lists.newArrayList(events);
...
As we can see in the input, the session is the POJO class that contains the problematic POJO class.
public class Session{
private problematicPOJO problematicpojo;
...
}
The problematic POJO class has 2 Boolean fields with the wrong getter/setter namings(literally missing Is :´<). Other fields in the class are ignored, they do not have any issues.
public class problematicPojo {
private Boolean isA;
private Boolean isB;
...
public getA(){ ... }
public setA(...){ ... }
public getB(){ ... }
public setB(...){ ... }
...
}
We have looked up some possible solutions.
Using State Processor API -> AWS does not provide access to KDA snapshots, so we're not able to modify it
Providing TypeInformation to the problematic POJO class -> did not seem to be working
We are thinking of specifying listStateDescriptor in the cogroup function(changing to RichCoGroup) to be able to manually update the states when recovering from a snapshot. But we could not get too much insight from the official docs. Is anyone here familiar with this method and can help us out?
Thank you!

How does Jackson #JsonProperty() work when used to annotate private fields?

Specifically I am wondering how when deserializing an object the deserializer could set a private field? Thinking of an example class like this:
public class MyClass {
#JsonProperty( "My String" );
private String myString;
}
If this is deserialized using objectMapper.readValue(json, MyClass.class); how does the resulting object have this field set if it is marked as private?

Calling Field.setAccessible(true) before reading or writing a value through reflection does the trick here.
For details see the corresponding javadoc: https://docs.oracle.com/javase/8/docs/api/java/lang/reflect/AccessibleObject.html#setAccessible-boolean-
But use with care ;-)

Quite a few frameworks allow access to private fields in this manner by using Field.setAccessible(true). It allows the application to ignore the Java language visibility rules (i.e. private) and read or change the value via an instance of the Reflection API Field class.
A little more can be found in this question:
Java reflection - impact of setAccessible(true)

The short answer is that it can't normally. We use lombok to generate the getter/setter for the variables, but you can of course write your own. Jackson has the same visibility as your main code does, so a private field cannot be mapped without some public getter/setter OR configuring the object mapper like so... objectMapper.setVisibility(PropertyAccessor.FIELD, Visibility.ANY);.
It wouldn't be able to serialize that either normally. You can use Lombok #Getter and #Setter on the class level so Jackson can work with myString, or put #JsonAutoDetect(fieldVisibility = Visibility.ANY) at the class level like below.
#JsonAutoDetect(fieldVisibility = Visibility.ANY)
public class MyClass {
#JsonProperty( "My String" );
private String myString;
}

How do I use Local Variable Annotations for Wicket Authorization?

I'm rolling my own IAuthorizationStrategy for Wicket 1.5.x I've setup type annotation for pages to use with isInstantiationAuthorized(). It works well and I'd like to use annotations for isActionAuthorized() as well. Ideally I'd like to be able annotate local variables and then check the annotations in my AuthStrategy. From what I've read Local variable Annotation doesn't work that way.
Is there any kind of known work around, maybe some sort of Compile time annotation processing to turn an annotated local variable into an "anonymous" subclass with the annotation as a type annotation?
For the record, the annotation I'm trying to use looks like this:
#Retention(RetentionPolicy.Runtime)
#Target(ElementType.Type, ElementType.LOCAL_VARIABLE)
public #interface AdminOnly
{
int isVisible() default 0;
int isEnabled() default 1;
}
UPDATE
So based on #Xavi López'es answer what I was hoping to do isn't exactly possible.
Annotated LocalVariables should be available at compile time though. Is there some way maybe I could use them as a shortcut for boiler-plating the meta-data code examples that are available in Wicket Examples or the excellent Apache Wicket Cookbook?

I've struggled with a similar issue some time ago with Wicket 1.3.x, and didn't find any way to achieve this with annotations. Annotations on local variables can't be retained at run-time, as explained in the JLS (9.6.3.2. #Retention):
An annotation on a local variable declaration is never retained in the binary representation.
In this related question: How can I create an annotation processor that processes a Local Variable? they talked about LAPT-javac, a patched javac version to allow this. On their site there's a link to the Type Annotations Specification (JSR 308), which will hopefully address this subject (JDK 8 ?).
I ended up defining a plain old interface with a related functionality code:
public interface RestrictedComponent {
Integer getFunction();
}
The main problem with this approach is that it's not possible to make instant anonymous subclasses of a specific class implement other interfaces (such as Component c = new TextField() implements AdminOnly { }) , but you can always define Component extensions that just implement RestrictedComponent in a class:
public abstract class RestrictedTextField extends TextField implements RestrictedComponent { }
Finally, I ended up implementing a RestrictedContainer that just subclassed WebMarkupContainer and put every secured component inside one, modelling it with a <wicket:container> in the markup.
public class RestrictedContainer extends WebMarkupContainer implements RestrictedComponent {
private final Integer function;
public RestrictedContainer(String id, IModel model, final Integer function) {
super(id, model);
this.function = function;
}
public RestrictedContainer(String id, final Integer funcionalitat) {
super(id);
this.function = function;
}
public Integer getFunction() {
return function;
}
}
And then in the Authorization Strategy checked for component instanceof RestrictedComponent and returned true or false depending on user permissions on the associated function.

Patterns: Populate instance from Parameters and export it to XML

I'm building a simple RESTFul Service; and for achieve that I need two tasks:
Get an instance of my resource (i.e Book) from request parameters, so I can get that instance to be persisted
Build an XML document from that instance to send the representation to the clients
Right now, I'm doing both things in my POJO class:
public class Book implements Serializable {
private Long id;
public Book(Form form) {
//Initializing attributes
id = Long.parseLong(form.getFirstValue(Book.CODE_ELEMENT));
}
public Element toXml(Document document) {
// Getting an XML Representation of the Book
Element bookElement = document.createElement(BOOK_ELEMENT);
}
I've remembered an OO principle that said that behavior should be where the data is, but now my POJO depends from Request and XML API's and that doesn't feels right (also, that class has persistence anotations)
Is there any standard approach/pattern to solve that issue?
EDIT:
The libraries i'm using are Restlets and Objectify.

I agree with you when you say that the behavior should be where the data is. But at the same time, as you say I just don't feel confortable polluting a POJO interface with specific methods used for serialization means (which can grow considerably depending on the way you want to do it - JSON, XML, etc.).
1) Build an XML document from that instance to send the representation to the clients
In order to decouple the object from serialization logic, I would adopt the Strategy Pattern:
interface BookSerializerStrategy {
String serialize(Book book);
}
public class XmlBookSerializerStrategy implements BookSerializerStrategy {
public String serialize(Book book) {
// Do something to serialize your book.
}
}
public class JsonBookSerializerStrategy implements BookSerializerStrategy {
public String serialize(Book book) {
// Do something to serialize your book.
}
}
You POJO interface would become:
public class Book implements Serializable {
private Long id;
private BookSerializerStrategy serializer
public String serialize() {
return serializer.serialize(this);
}
public void setSerializer(BookSerializerStrategy serializer) {
this.serializer = serializer;
}
}
Using this approach you will be able to isolate the serialization logic in just one place and wouldn't pollute your POJO with that. Additionally, returning a String I won't need to couple you POJO with classes Document and Element.
2) Get an instance of my resource (i.e Book) from request parameters, so I can get that instance to be persisted
To find a pattern to handle the deserialization is more complex in my opinion. I really don't see a better way than to create a Factory with static methods in order to remove this logic from your POJO.
Another approach to answer your two questions would be something like JAXB uses: two different objects, an Unmarshaller in charge of deserialization and a Marshaller for serialization. Since Java 1.6, JAXB comes by default with JDK.
Finally, those are just suggestions. I've become really interested in your question actually and curious about other possible solutions.

Are you using Spring, or any other framework, in your project? If you used Spring, it would take care of serialization for you, as well as assigning request params to method params (parsing as needed).

How can I tell jackson to ignore a property for which I don't have control over the source code?

Long story short, one of my entities has a GeometryCollection that throws an exception when you call "getBoundary" (the why of this is another book, for now let's say this is the way it works).
Is there a way I can tell Jackson not to include that specific getter? I know I can use #JacksonIgnore when I do own/control the code. But this is not case, jackson ends reaching this point through continuous serialization of the parent objects. I saw a filtering option in jackson documentation. Is that a plausible solution?
Thanks!

You can use Jackson Mixins. For example:
class YourClass {
public int ignoreThis() { return 0; }
}
With this Mixin
abstract class MixIn {
#JsonIgnore abstract int ignoreThis(); // we don't need it!
}
With this:
objectMapper.getSerializationConfig().addMixInAnnotations(YourClass.class, MixIn.class);
Edit:
Thanks to the comments, with Jackson 2.5+, the API has changed and should be called with objectMapper.addMixIn(Class<?> target, Class<?> mixinSource)

One other possibility is, if you want to ignore all unknown properties, you can configure the mapper as follows:
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);

Using Java Class
new ObjectMapper().configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
Using Annotation
#JsonIgnoreProperties(ignoreUnknown=true)

Annotation based approach is better. But sometimes manual operation is needed. For this purpose you can use without method of ObjectWriter.
ObjectMapper mapper = new ObjectMapper().configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
ObjectWriter writer = mapper.writer().withoutAttribute("property1").withoutAttribute("property2");
String jsonText = writer.writeValueAsString(sourceObject);

Mix-in annotations work pretty well here as already mentioned. Another possibility beyond per-property #JsonIgnore is to use #JsonIgnoreType if you have a type that should never be included (i.e. if all instances of GeometryCollection properties should be ignored). You can then either add it directly (if you control the type), or using mix-in, like:
#JsonIgnoreType abstract class MixIn { }
// and then register mix-in, either via SerializationConfig, or by using SimpleModule
This can be more convenient if you have lots of classes that all have a single 'IgnoredType getContext()' accessor or so (which is the case for many frameworks)

I had a similar issue, but it was related to Hibernate's bi-directional relationships. I wanted to show one side of the relationship and programmatically ignore the other, depending on what view I was dealing with. If you can't do that, you end up with nasty StackOverflowExceptions. For instance, if I had these objects
public class A{
Long id;
String name;
List<B> children;
}
public class B{
Long id;
A parent;
}
I would want to programmatically ignore the parent field in B if I were looking at A, and ignore the children field in A if I were looking at B.
I started off using mixins to do this, but that very quickly becomes horrible; you have so many useless classes laying around that exist solely to format data. I ended up writing my own serializer to handle this in a cleaner way: https://github.com/monitorjbl/json-view.
It allows you programmatically specify what fields to ignore:
ObjectMapper mapper = new ObjectMapper();
SimpleModule module = new SimpleModule();
module.addSerializer(JsonView.class, new JsonViewSerializer());
mapper.registerModule(module);
List<A> list = getListOfA();
String json = mapper.writeValueAsString(JsonView.with(list)
.onClass(B.class, match()
.exclude("parent")));
It also lets you easily specify very simplified views through wildcard matchers:
String json = mapper.writeValueAsString(JsonView.with(list)
.onClass(A.class, match()
.exclude("*")
.include("id", "name")));
In my original case, the need for simple views like this was to show the bare minimum about the parent/child, but it also became useful for our role-based security. Less privileged views of objects needed to return less information about the object.
All of this comes from the serializer, but I was using Spring MVC in my app. To get it to properly handle these cases, I wrote an integration that you can drop in to existing Spring controller classes:
#Controller
public class JsonController {
private JsonResult json = JsonResult.instance();
#Autowired
private TestObjectService service;
#RequestMapping(method = RequestMethod.GET, value = "/bean")
#ResponseBody
public List<TestObject> getTestObject() {
List<TestObject> list = service.list();
return json.use(JsonView.with(list)
.onClass(TestObject.class, Match.match()
.exclude("int1")
.include("ignoredDirect")))
.returnValue();
}
}
Both are available on Maven Central. I hope it helps someone else out there, this is a particularly ugly problem with Jackson that didn't have a good solution for my case.

If you want to ALWAYS exclude certain properties for any class, you could use setMixInResolver method:
#JsonIgnoreProperties({"id", "index", "version"})
abstract class MixIn {
}
mapper.setMixInResolver(new ClassIntrospector.MixInResolver(){
#Override
public Class<?> findMixInClassFor(Class<?> cls) {
return MixIn.class;
}
#Override
public ClassIntrospector.MixInResolver copy() {
return this;
}
});

One more good point here is to use #JsonFilter.
Some details here Feature: JSON Filter