We Keep Coding

Kinesis Data Analytics - Flink state serializer incompatible after recovering from Snapshot

We have our Flink application(version 1.13.2) deployed on AWS KDA. The strategy is that we do not want the application to stop at all, so we always recover the application from a snapshot when updating the jar with new changes.
Recently, we found a problem where a lower-level POJO class is corrupted. It contains a few getters and setters with wrong namings. This early mistake essentially hinders us from adding the POJO class with new fields. So we decided to rename the getter/setter directly. But it led us to the following exception after updating the application.
org.apache.flink.util.StateMigrationException: The new state serializer (org.apache.flink.api.common.typeutils.base.ListSerializer#46c65a77) must not be incompatible with the old state serializer (org.apache.flink.api.common.typeutils.base.ListSerializer#30c9146c).
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.updateRestoredStateMetaInfo(RocksDBKeyedStateBackend.java:704) at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.tryRegisterKvStateInformation(RocksDBKeyedStateBackend.java:624)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.createInternalState(RocksDBKeyedStateBackend.java:837) at org.apache.flink.runtime.state.KeyedStateFactory.createInternalState(KeyedStateFactory.java:47) at org.apache.flink.runtime.state.ttl.TtlStateFactory.createStateAndWrapWithTtlIfEnabled(TtlStateFactory.java:71)
at org.apache.flink.runtime.state.AbstractKeyedStateBackend.getOrCreateKeyedState(AbstractKeyedStateBackend.java:301) at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.getOrCreateKeyedState(StreamOperatorStateHandler.java:315) at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.getOrCreateKeyedState(AbstractStreamOperator.java:494) at org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.open(WindowOperator.java:243) at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:442)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:582) at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55) at org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(StreamTask.java:562)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647) at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:537) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:764) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:571)
at java.base/java.lang.Thread.run(Thread.java:829)
As far as we understand, the failure happens specifically in the 2 CoGroup functions we implemented. They are both consuming the corrupted POJO class nested in another POJO class, Session. A code snippet of the Cogroup function is shown below. BTW, we are using google guava list here, not sure if it causes list serializer problem.
public class OutputCoGroup extends CoGroupFunction<Session, Event, OutputSession> {
#Override
public void coGroup(Iterable<Session> sessions, Iterable<Event> events,
Collector<OutputSession> collector) throws Exception {
// we are using google guava list here, not sure if it causes list serializer problem
if (Lists.newArrayList(sessions).size() > 0) {
...
if (events.iterator().hasNext()) {
List<Event> eventList = Lists.newArrayList(events);
...
As we can see in the input, the session is the POJO class that contains the problematic POJO class.
public class Session{
private problematicPOJO problematicpojo;
...
}
The problematic POJO class has 2 Boolean fields with the wrong getter/setter namings(literally missing Is :´<). Other fields in the class are ignored, they do not have any issues.
public class problematicPojo {
private Boolean isA;
private Boolean isB;
...
public getA(){ ... }
public setA(...){ ... }
public getB(){ ... }
public setB(...){ ... }
...
}
We have looked up some possible solutions.
Using State Processor API -> AWS does not provide access to KDA snapshots, so we're not able to modify it
Providing TypeInformation to the problematic POJO class -> did not seem to be working
We are thinking of specifying listStateDescriptor in the cogroup function(changing to RichCoGroup) to be able to manually update the states when recovering from a snapshot. But we could not get too much insight from the official docs. Is anyone here familiar with this method and can help us out?
Thank you!

Multiple Java Objects to single endpoint

Instead of building a case statement for my Spring Boot Rest Controller, I want to have Spring use the correct endpoint. I am not even sure this is possible but I am hoping the universe could save me.
#PostMapping("/endpoint")
public String one(Greeting greet) {
return "Greeting Posted";
}
#PostMapping("/endpoint")
public String two(Address addr) {
return "Address Posted";
}
Current Error
Caused by: java.lang.IllegalStateException: Ambiguous mapping. Cannot map 'RController' method
public java.lang.String com.example.controller.RController.two(com.example.model.Address)
to {[/endpoint],methods=[POST]}: There is already 'RController' bean method
public java.lang.String com.example.controller.RController.one(com.example.model.Greeting) mapped.

This is not possible. It's ambiguous.
As a good practice, if 2 resources will handle the data differently, you must create a different endpoint for each one.
Or possible workaround for you, it's create an ViewModel object and handle it in just one method.
public class GreetingAddressVM {
private Address address;
private Greeting greeting;
}
I'd prefer creating different mapping for each action.

How to use a Serializable class

I'm using Backendless as a backend service, it offers a class called BackendlessUser for saving and retrieving users. I'm trying to pass a User between two activities on Android by passing it as Serializable extra:
Intent intent = new Intent(PeopleActivity.this, ConversationActivity.class);
intent.putExtra("withUser", contacts.get(position));
startActivity(intent);
Since the class BackendlessUser implements Serializable. However when I run it, it gives me this error:
java.lang.RuntimeException: Parcelable encountered IOException writing serializable object (name = com.backendless.BackendlessUser)
at android.os.Parcel.writeSerializable(Parcel.java:1468)
at android.os.Parcel.writeValue(Parcel.java:1416)
....
Caused by: java.io.NotSerializableException: java.lang.Object
....
Due to, I think, this variable:
public final class BackendlessUser implements Serializable {
private final Map<String, Object> properties = new HashMap<String, Object>();
...
How can I solve this considering that I cannot modify the BackendlessUser class?

In Android you should use Parcelable which offers better performances compared to Serializable. For an explanation about how to implement it take a look at this answer
Also, if you need to use Parcelable on your map object, see this answer
Edit: since Object is not Parcelable though you might want to follow Alexander's answer, or, even better, use a database for data persistence

Instead of passing the object, you can save the reference to it in a singleton so that it's available between the activities.
You can extend the Application class and save there. The Application class exists all the time while your app is running and is a singleton.
public class MyApp extends Application {
public BackendUser currentUser;
}
Then:
((MyApp)getApplication()).currentUser

You should be using Parcelable to pass objects between activities. Parcelable is Android's version of Serializable, so you use that. You can find more information on Parcelable here.
If you can't modify the backend user, your best bet would be to use Alexanders suggestion and create a Singleton instance of User. This would allow you to create/update you user from any activity.

Patterns: Populate instance from Parameters and export it to XML

I'm building a simple RESTFul Service; and for achieve that I need two tasks:
Get an instance of my resource (i.e Book) from request parameters, so I can get that instance to be persisted
Build an XML document from that instance to send the representation to the clients
Right now, I'm doing both things in my POJO class:
public class Book implements Serializable {
private Long id;
public Book(Form form) {
//Initializing attributes
id = Long.parseLong(form.getFirstValue(Book.CODE_ELEMENT));
}
public Element toXml(Document document) {
// Getting an XML Representation of the Book
Element bookElement = document.createElement(BOOK_ELEMENT);
}
I've remembered an OO principle that said that behavior should be where the data is, but now my POJO depends from Request and XML API's and that doesn't feels right (also, that class has persistence anotations)
Is there any standard approach/pattern to solve that issue?
EDIT:
The libraries i'm using are Restlets and Objectify.

I agree with you when you say that the behavior should be where the data is. But at the same time, as you say I just don't feel confortable polluting a POJO interface with specific methods used for serialization means (which can grow considerably depending on the way you want to do it - JSON, XML, etc.).
1) Build an XML document from that instance to send the representation to the clients
In order to decouple the object from serialization logic, I would adopt the Strategy Pattern:
interface BookSerializerStrategy {
String serialize(Book book);
}
public class XmlBookSerializerStrategy implements BookSerializerStrategy {
public String serialize(Book book) {
// Do something to serialize your book.
}
}
public class JsonBookSerializerStrategy implements BookSerializerStrategy {
public String serialize(Book book) {
// Do something to serialize your book.
}
}
You POJO interface would become:
public class Book implements Serializable {
private Long id;
private BookSerializerStrategy serializer
public String serialize() {
return serializer.serialize(this);
}
public void setSerializer(BookSerializerStrategy serializer) {
this.serializer = serializer;
}
}
Using this approach you will be able to isolate the serialization logic in just one place and wouldn't pollute your POJO with that. Additionally, returning a String I won't need to couple you POJO with classes Document and Element.
2) Get an instance of my resource (i.e Book) from request parameters, so I can get that instance to be persisted
To find a pattern to handle the deserialization is more complex in my opinion. I really don't see a better way than to create a Factory with static methods in order to remove this logic from your POJO.
Another approach to answer your two questions would be something like JAXB uses: two different objects, an Unmarshaller in charge of deserialization and a Marshaller for serialization. Since Java 1.6, JAXB comes by default with JDK.
Finally, those are just suggestions. I've become really interested in your question actually and curious about other possible solutions.

Are you using Spring, or any other framework, in your project? If you used Spring, it would take care of serialization for you, as well as assigning request params to method params (parsing as needed).

GWT AutoBean: annotation #PropertyName doesn't work anymore

I use AutoBean to code/decode data to JSON and that was all right in previous GWT versions. In my opinion AutoBean is very good and convenient tool to deal with JSON. Since GWT ver.2.4.0 this functionality has changed and I spent some time to restore it in my code. But only one part stays still unfixed - annotation #PropertyName. This annotation is used to add an "alias" to properties. It saves a lot of network traffic. And now it throws an exception. The code example is below:
import com.google.web.bindery.autobean.shared.AutoBean.PropertyName;
public interface IPersonInfo {
// Name
#PropertyName("n")
public String getName();
public void setName(String name);
// Surname
#PropertyName("s")
public String getSurname();
public void setSurname(String surname);
// other properties...
}
Then I try to decode this to JSON in this way:
AutoBean<IPersonInfo> user = factory.user();
// clone the userDto (it's a new way to clone an object in ver 2.4.0
// instad of deprecated clone() method)
Splittable data = AutoBeanCodex.encode(user);
IPersonInfo userDto = AutoBeanCodex.decode(factory, IPersonInfo.class, data).as();
userDto.setName("Name");
userDto.setSurname("Surname");
//... other properties
This piece of code worked perfectly in legacy code. But now (in GWT 2.4.0) I get an exception:
java.lang.IllegalArgumentException: name
at com.google.web.bindery.autobean.shared.impl.AutoBeanCodexImpl.doCoderFor(AutoBeanCodexImpl.java:524)
at com.google.web.bindery.autobean.shared.impl.AbstractAutoBean.setProperty(AbstractAutoBean.java:276)
at com.google.web.bindery.autobean.vm.impl.ProxyAutoBean.setProperty(ProxyAutoBean.java:253)
at com.google.web.bindery.autobean.vm.impl.BeanMethod$3.invoke(BeanMethod.java:103)
at com.google.web.bindery.autobean.vm.impl.SimpleBeanHandler.invoke(SimpleBeanHandler.java:43)
at $Proxy74.setName(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.google.appengine.tools.development.agent.runtime.Runtime.invoke(Runtime.java:104)
at com.google.web.bindery.autobean.vm.impl.ShimHandler.invoke(ShimHandler.java:81)
at $Proxy74.setName(Unknown Source)
If I remove the #PropertyName from my interface, then the exception won't occur.
I still waiting, that the official documentation will be updated, but it still stays with old code examples.
Can someone help me to solve this issue? Thank you for advice.
I use GWT ver. 2.4.0, GAE ver. 1.6.1.

I needed to put #PropertyName("XXXX") on my set methods as well. Give it a try.