We Keep Coding

Kinesis Data Analytics - Flink state serializer incompatible after recovering from Snapshot

We have our Flink application(version 1.13.2) deployed on AWS KDA. The strategy is that we do not want the application to stop at all, so we always recover the application from a snapshot when updating the jar with new changes.
Recently, we found a problem where a lower-level POJO class is corrupted. It contains a few getters and setters with wrong namings. This early mistake essentially hinders us from adding the POJO class with new fields. So we decided to rename the getter/setter directly. But it led us to the following exception after updating the application.
org.apache.flink.util.StateMigrationException: The new state serializer (org.apache.flink.api.common.typeutils.base.ListSerializer#46c65a77) must not be incompatible with the old state serializer (org.apache.flink.api.common.typeutils.base.ListSerializer#30c9146c).
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.updateRestoredStateMetaInfo(RocksDBKeyedStateBackend.java:704) at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.tryRegisterKvStateInformation(RocksDBKeyedStateBackend.java:624)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.createInternalState(RocksDBKeyedStateBackend.java:837) at org.apache.flink.runtime.state.KeyedStateFactory.createInternalState(KeyedStateFactory.java:47) at org.apache.flink.runtime.state.ttl.TtlStateFactory.createStateAndWrapWithTtlIfEnabled(TtlStateFactory.java:71)
at org.apache.flink.runtime.state.AbstractKeyedStateBackend.getOrCreateKeyedState(AbstractKeyedStateBackend.java:301) at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.getOrCreateKeyedState(StreamOperatorStateHandler.java:315) at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.getOrCreateKeyedState(AbstractStreamOperator.java:494) at org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.open(WindowOperator.java:243) at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:442)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:582) at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55) at org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(StreamTask.java:562)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647) at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:537) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:764) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:571)
at java.base/java.lang.Thread.run(Thread.java:829)
As far as we understand, the failure happens specifically in the 2 CoGroup functions we implemented. They are both consuming the corrupted POJO class nested in another POJO class, Session. A code snippet of the Cogroup function is shown below. BTW, we are using google guava list here, not sure if it causes list serializer problem.
public class OutputCoGroup extends CoGroupFunction<Session, Event, OutputSession> {
#Override
public void coGroup(Iterable<Session> sessions, Iterable<Event> events,
Collector<OutputSession> collector) throws Exception {
// we are using google guava list here, not sure if it causes list serializer problem
if (Lists.newArrayList(sessions).size() > 0) {
...
if (events.iterator().hasNext()) {
List<Event> eventList = Lists.newArrayList(events);
...
As we can see in the input, the session is the POJO class that contains the problematic POJO class.
public class Session{
private problematicPOJO problematicpojo;
...
}
The problematic POJO class has 2 Boolean fields with the wrong getter/setter namings(literally missing Is :´<). Other fields in the class are ignored, they do not have any issues.
public class problematicPojo {
private Boolean isA;
private Boolean isB;
...
public getA(){ ... }
public setA(...){ ... }
public getB(){ ... }
public setB(...){ ... }
...
}
We have looked up some possible solutions.
Using State Processor API -> AWS does not provide access to KDA snapshots, so we're not able to modify it
Providing TypeInformation to the problematic POJO class -> did not seem to be working
We are thinking of specifying listStateDescriptor in the cogroup function(changing to RichCoGroup) to be able to manually update the states when recovering from a snapshot. But we could not get too much insight from the official docs. Is anyone here familiar with this method and can help us out?
Thank you!

Java Play Controller Session Retrieval Understanding

The company I work for, use Java Play Framework. What I think mysterious is how can play retrieve the current session to me. As I see from source code,
package play.mvc;
public abstract class Controller extends Results implements Status, HeaderNames {
public static Request request() {
return Http.Context.current().request();
}
public static Session session() {
return Http.Context.current().session();
}
// ...
}
I found such class very strange, as all the methods are static methods. And I do not understand how Play can get the correct session in handling concurrent request (multi-thread environment ?).
In such case, how can Play handle / retrieve the session information correctly? Note that I can retrieve the session in the beginning or at the end of each Action (which also affect time). And in such case, how can Play retrieve the session correctly using a static method?
Or am I missing something here? It will be cool if one can tell me how Play can make such retrieval work. Thanks.
P.S. I am using quite a legacy Play Version, Play 2.2.0 in my work place and not learning Scala yet.

Java 14 records json serialization

Currently experimenting with the Records implements from java 14, everything looks nice but since the accessors are slightly different and jackson is not being able to deserialize and giving the following error:
Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: No serializer found for class com.x.x.x.xTracking and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS)
I checked all around the internet, including jackson and gson github to check the jep 359 support but havent found a single comment. Am i missing something really straight forward?
Yes i am aware that the java 14 is still not released and that records is only in preview in this version but would expect some comments at least.

Records support was added to Jackson 2.12.0 (https://github.com/FasterXML/jackson-future-ideas/issues/46). It will be released in the next days.

For someone else experimenting, i went around, not proudly, with the following:
#Bean
public Jackson2ObjectMapperBuilderCustomizer jacksonCustomizer(){
return builder ->
builder.visibility(PropertyAccessor.FIELD, JsonAutoDetect.Visibility.ANY);
}

With "pure" Json-B you can do it this way:
public class RecordPropertyVisibilityStrategy implements PropertyVisibilityStrategy {
#Override
public boolean isVisible(Field field) {return true;}
#Override
public boolean isVisible(Method method) {return false;}
}
Then
#JsonbVisibility(RecordPropertyVisibilityStrategy.class)
public record MyRecord(Long id, String attr) {}

I did Jackson dealing with this problem in Spring Boot 2.4.1 application annotating the record with
#JsonAutoDetect(fieldVisibility = JsonAutoDetect.Visibility.ANY)

Alternatively, for Spring Boot 2.4.5 and Java 16, it may be enabled via application.properties, like so:
spring.jackson.visibility.field=any
See more on the Spring Boot documentation.

java declare anonymous class using proxy api

This morning I fell into a particular case that never happened to me before. I'm developing a Minecraft plugin using the minecraft server API which is usually called NMS with reference to the name of its packages (eg net.minecraft.server.v1_13_R1 for version 1.13).
The main problem with the use of the minecraft server API is that it is difficult to write a cross version code: indeed the name of the packages changes with each new version.
When the plugin only supports two versions it is usually easier to use the interfaces to write two different codes depending on the version. But when you have to support a dozen different versions (and this is my case), it's a bad idea (the plugin would be much too heavy, it would have to import every jar in the IDE, and I would have to redo the code with each new version).
In these cases I usually use reflection but I do not think it's possible here:
packet = packetConstructor.newInstance(
new MinecraftKey("q", "q") {
#Override
public String toString() {
return "FML|HS";
}
},
packetDataSerializerConstructor.newInstance(Unpooled.wrappedBuffer(data)));
As you probably guessed MinecraftKey is a class from NMS and I was told to use Java Dynamic Proxy API. I have never used it and would like to know if you would know a place that would explain to me how to do it simply? If you know of another better method that interests me too!
When I think about it, I think that this is really a lot of trouble for a tiny piece of code x)
EDIT :
My plugin uses the PacketPlayOutCustomPayload (aka plugin messages) to communicate with the mods of the players. It allows me to send a message (a byte []) on a particular channel (a String). But with the 1.13 this String has been replaced by a MinecraftKey (a wrapper for the String that replaces some characters and requires the use of a ":"). This poses a problem when players connect to 1.12 on my 1.13 server so I do not have a choice: I have to override the MinecraftKey object in this case.

I don’t really think using proxy classes is good solution here, it will only make it harder to debug, but if you need something like that you should use library like ByteBuddy: (as java can’t generate proxy for a class, only interfaces are allowed)
import net.bytebuddy.ByteBuddy;
import net.bytebuddy.implementation.FixedValue;
import static net.bytebuddy.matcher.ElementMatchers.*;
public class Main {
public static void main(String[] args) throws Exception {
SomeKey someKey = new SomeKey("my", "key");
System.out.println(someKey); // :<
// this class should be cached/saved somewhere, do not create new one each time.
Class<? extends SomeKey> loaded = new ByteBuddy()
.subclass(SomeKey.class)
.method(named("toString").and(returns(String.class).and(takesArguments(0))))
.intercept(FixedValue.value("something"))
.make()
.load(Main.class.getClassLoader()).getLoaded();
someKey = loaded.getConstructor(String.class, String.class).newInstance("what", "ever");
System.out.println(someKey); // YeY
}
}
class SomeKey {
final String group;
final String name;
public SomeKey(String group, String name) {
this.group = group;
this.name = name;
}
public String getGroup() { return this.group; }
public String getName() { return this.name; }
#Override public String toString() {
return group+":"+name;
}
}
But I would just create separate modules in my project, one that does only work with real bukkit API and contains many interfaces to represent NMS types in some normalised and readable way.
And separate modules for each version, then you will not have much code to duplicate, as most of it will be abstracted and handled by that “core/base” module.
Then you can build it as one single fat jar or separate .jar per version.
Other solution might be using some template engines and preprocessors to generate java sources on build time, see how fastutil is doing this:
https://github.com/vigna/fastutil
And yet another solution for simple classes and parts of code would be to use build-in javascript or external script language like groovy to also create this pattern-line but in runtime. But I would use this only for simplest stuff.
Also for just using methods you can just use normal reflections.
You can also always inject into netty and instead of using default packet serializer just write own bytes, then you don't need that key at all.

WCF Serialization problems with WSDL file created by Java tools

My team is tasked with getting several in-house developed .NET client applications to connect to some new Java web services. The Java web service is a third party, vendor supplied WSDL file that our team has a limited ability to modify/control...meaning we probably have the power to request our vendor to make slight tweaks to the WSDL, but major changes would probably be either unfeasible or difficult to request.
That said, we are attempting to utilize WCF/.NET 4.0 to generate the .NET proxy class files we need on the client side. The proxy client class file generation process executes without issues.
The problem is when we attempt to use the proxy class file in a client app. I have verified through the web trace tool, Fiddler, that the raw SOAP message request fails to get sent across the wire to the server.
The specific .NET exception message I get when attempting to call the web service method in question, looks like this:
System.InvalidOperationException was unhandled
Message=XmlSerializer attribute System.Xml.Serialization.XmlAttributeAttribute is not valid in baseLanguage. Only XmlElement, XmlArray, XmlArrayItem, XmlAnyAttribute and XmlAnyElement attributes are supported when IsWrapped is true.
Source=System.ServiceModel
When I examine the .NET autogenerated proxy class file, Reference.cs, I noticed that the request and response messages for my web service method looks something like this:
[System.Diagnostics.DebuggerStepThroughAttribute()]
[System.CodeDom.Compiler.GeneratedCodeAttribute("System.ServiceModel", "4.0.0.0")]
[System.ServiceModel.MessageContractAttribute(WrapperName="QueryPBOT_MXWO_OS", WrapperNamespace="http://www.ibm.com/maximo", IsWrapped=true)]
public partial class QueryPBOT_MXWO_OSRequest {
[System.ServiceModel.MessageBodyMemberAttribute(Namespace="http://www.ibm.com/maximo", Order=0)]
public ConsoleApplication7.wsMaximo.PBOT_MXWO_OSQueryType PBOT_MXWO_OSQuery;
[System.ServiceModel.MessageBodyMemberAttribute(Namespace="http://www.ibm.com/maximo", Order=1)]
[System.Xml.Serialization.XmlAttributeAttribute()]
public string baseLanguage;
[System.ServiceModel.MessageBodyMemberAttribute(Namespace="http://www.ibm.com/maximo", Order=2)]
[System.Xml.Serialization.XmlAttributeAttribute()]
public string transLanguage;
[System.ServiceModel.MessageBodyMemberAttribute(Namespace="http://www.ibm.com/maximo", Order=3)]
[System.Xml.Serialization.XmlAttributeAttribute()]
public string messageID;
[System.ServiceModel.MessageBodyMemberAttribute(Namespace="http://www.ibm.com/maximo", Order=4)]
[System.Xml.Serialization.XmlAttributeAttribute()]
public string maximoVersion;
[System.ServiceModel.MessageBodyMemberAttribute(Namespace="http://www.ibm.com/maximo", Order=5)]
[System.Xml.Serialization.XmlAttributeAttribute()]
[System.ComponentModel.DefaultValueAttribute(false)]
public bool uniqueResult;
[System.ServiceModel.MessageBodyMemberAttribute(Namespace="http://www.ibm.com/maximo", Order=6)]
[System.Xml.Serialization.XmlAttributeAttribute(DataType="positiveInteger")]
public string maxItems;
[System.ServiceModel.MessageBodyMemberAttribute(Namespace="http://www.ibm.com/maximo", Order=7)]
[System.Xml.Serialization.XmlAttributeAttribute(DataType="integer")]
[System.ComponentModel.DefaultValueAttribute("0")]
public string rsStart;
public QueryPBOT_MXWO_OSRequest() {
}
public QueryPBOT_MXWO_OSRequest(ConsoleApplication7.wsMaximo.PBOT_MXWO_OSQueryType PBOT_MXWO_OSQuery, string baseLanguage, string transLanguage, string messageID, string maximoVersion, bool uniqueResult, string maxItems, string rsStart) {
this.PBOT_MXWO_OSQuery = PBOT_MXWO_OSQuery;
this.baseLanguage = baseLanguage;
this.transLanguage = transLanguage;
this.messageID = messageID;
this.maximoVersion = maximoVersion;
this.uniqueResult = uniqueResult;
this.maxItems = maxItems;
this.rsStart = rsStart;
}
}
I know that people reading this post will want to see the actual WSDL file we're trying to consume, but it is quite large, and I'm concerned the sheer size of it would make pinpointing the error quite difficult.
I'm hoping that the autogenerated client proxy file and the .NET exception will help someone recognize this WCF Serialization issue.
We've confirmed from our Java vendor that the style of WSDL they generate is doc-literal. After doing some research on the internet, it appears that WCF, by default. translates WSDL files with doc-literal wrapped, and that this may explain, at least in part, why we're seeing this WCF serialization issue with the WSDL file.
I've discovered, through trial and error, that the following attribute decorator in the proxy class file is the culprit behind the serialization issue:
[System.Xml.Serialization.XmlAttributeAttribute()]
If I comment out all instances of this attribute in the proxy class file and rerun my client app, the SOAP message successfully gets sent across the wire and I get a valid web service response come back from the server.
This fix is better than nothing, but I would very much prefer a solution that doesn't require myself or anyone on my team to constantly tweak these .NET autogenerated proxy class files.
I would like to know if there is something I can do, either through the various WCF tools or by modifying the WSDL file, that prevents that [System.Xml.Serialization.XmlAttributeAttribute()] from being applied to my request and response object properties?
Or at least a high level description of WHY we are seeing this serialization behavior in .NET with the Java WSDL file?
thanks in advance,
John

Use svcutil.exe utility with the /wrapped option on to generate proxy classes.
This will create slightly different classes then those created though Visual Studio in a way described by Ladislav Mrnka here. Resulting proxies should be free from the XmlAttribute issue when using on the client side.
Example:
svcutil /t:code wsdl.xml /out:wsdl.cs /serializer:XmlSerializer /wrapped

Here is how to do Mikhail G's solution within IDE:
Open Reference.svcmap under Service References
Add <Wrapped>true</Wrapped> under <ClientOptions> and Save
Right Click Reference.svcmap and hit "Run Custom Tool"
Visual Studio, where magic happens :)
Note: Tried with VS 2015. Prior versions may have same option with a
different name than "Run Custom Tool"

As a follow-on to stratovarius's answer, in VS 2017 the Service References folder is replaced by Connected Services, so you have to:
Open the {project}/Connected Services folder in Windows Explorer
Find and edit the Reference.svcmap with a text editor
Add <Wrapped>true</Wrapped> to the <ClientOptions> section
Save the file
In VS, right click on the service reference under Connected Services and select "Update Service Reference"
This cleared the exception from my service call.

Based on generated code it looks like your Java service expects request like:
<s:Envelope xmlns:s="...">
...
<s:Body>
<QueryPBOT_MXWO_OS xmlns="http://www.ibm.com/maximo" baseLanguage="..." transLanguage="..." ...>
<PBOT_MXWO_OSQuery>
...
</PBOT_MXWO_OSQuery>
</QueryPBOT_MXWO_OS>
</s:Body>
</s:Envelope>
The problem is that WCF recognized QueryPBOT_MXWO_OS as wrapper element for request. I'm not sure why it fires exception but probably there is some restriction that wrapper element can't have attributes. I'm suspicious that this is just global error handling shared with version which uses IsWrapped=false where usage of attributes is error.
You can try to modify your proxy in this way:
[System.Diagnostics.DebuggerStepThroughAttribute()]
[System.CodeDom.Compiler.GeneratedCodeAttribute("System.ServiceModel", "4.0.0.0")]
[System.ServiceModel.MessageContractAttribute(IsWrapped=false)]
public partial class QueryPBOT_MXWO_OSRequest
{
[MessageBodyMemberAttribute(Name="QueryPBOT_MXWO_OS", Namespace="http://www.ibm.com/maximo")]
public QueryPBOT_MXWO_OS QueryPBOT_MXWO_OS { get; set; }
}
[XmlRoot(ElementName="QueryPBOT_MXWO_OS", Namespace="http://www.ibm.com/maximo")]
public class QueryPBOT_MXWO_OS
{
[XmlElement(Namespace="http://www.ibm.com/maximo")]
public ConsoleApplication7.wsMaximo.PBOT_MXWO_OSQueryType PBOT_MXWO_OSQuery;
[XmlAttribute(Namespace="http://www.ibm.com/maximo")]
public string baseLanguage;
[XmlAttribute(Namespace="http://www.ibm.com/maximo")]
public string transLanguage;
[XmlAttribute(Namespace="http://www.ibm.com/maximo")]
public string messageID;
[XmlAttribute(Namespace="http://www.ibm.com/maximo")]
public string maximoVersion;
[XmlAttribute(Namespace="http://www.ibm.com/maximo")]
[System.ComponentModel.DefaultValueAttribute(false)]
public bool uniqueResult;
[XmlAttribute(Namespace="http://www.ibm.com/maximo")]
public string maxItems;
[XmlAttribute(Namespace="http://www.ibm.com/maximo")]
[System.ComponentModel.DefaultValueAttribute("0")]
public string rsStart;
}

I'm using an external "old" Java generated WSDL in my .Net Core app and the auto-generated Reference.cs did not work for me. I had to remove the [System.Xml.Serialization.XmlAttributeAttribute()] in order for it to work.