Using protobuf with flink - java

I'm using flink to read data from kafka and convert it to protobuf. The problem I'm facing is when I run the java application I get the below error. If I modify the unknownFields variable name to something else, it works but it's hard to make this change on all protobuf classes.
I also tried to deserialize directly when reading from kafka but I'm not sure what should be the TypeInformation to be returned for getProducedType() method.
public static class ProtoDeserializer implements DeserializationSchema{
#Override
public TypeInformation getProducedType() {
// TODO Auto-generated method stub
return PrimitiveArrayTypeInfo.BYTE_PRIMITIVE_ARRAY_TYPE_INFO;
}
Appreciate all the help. Thanks.
java.lang.RuntimeException: The field protected com.google.protobuf.UnknownFieldSet com.google.protobuf.GeneratedMessage.unknownFields is already contained in the hierarchy of the class com.google.protobuf.GeneratedMessage.Please use unique field names through your classes hierarchy
at org.apache.flink.api.java.typeutils.TypeExtractor.getAllDeclaredFields(TypeExtractor.java:1594)
at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1515)
at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1412)
at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
at org.apache.flink.api.java.typeutils.TypeExtractor.privateCreateTypeInfo(TypeExtractor.java:437)
at org.apache.flink.api.java.typeutils.TypeExtractor.getUnaryOperatorReturnType(TypeExtractor.java:306)
at org.apache.flink.api.java.typeutils.TypeExtractor.getFlatMapReturnTypes(TypeExtractor.java:133)
at org.apache.flink.streaming.api.datastream.DataStream.flatMap(DataStream.java:529)
Code:
FlinkKafkaConsumer09<byte[]> kafkaConsumer = new FlinkKafkaConsumer09<>("testArr",new ByteDes(),p);
DataStream<byte[]> input = env.addSource(kafkaConsumer);
DataStream<PBAddress> protoData = input.map(new RichMapFunction<byte[], PBAddress>() {
#Override
public PBAddress map(byte[] value) throws Exception {
PBAddress addr = PBAddress.parseFrom(value);
return addr;
}
});

Maybe you should try this follow:
env.getConfig().registerTypeWithKryoSerializer(PBAddress. class,ProtobufSerializer.class);
or
env.getConfig().registerTypeWithKryoSerializer(PBAddress. class,PBAddressSerializer.class);
public class PBAddressSerializer extends Serializer<Message> {
final private Map<Class,Method> hashMap = new HashMap<Class, Method>();
protected Method getParse(Class cls) throws NoSuchMethodException {
Method method = hashMap.get(cls);
if (method == null) {
method = cls.getMethod("parseFrom",new Class[]{byte[].class});
hashMap.put(cls,method);
}
return method;
}
#Override
public void write(Kryo kryo, Output output, Message message) {
byte[] ser = message.toByteArray();
output.writeInt(ser.length,true);
output.writeBytes(ser);
}
#Override
public Message read(Kryo kryo, Input input, Class<Message> pbClass) {
try {
int size = input.readInt(true);
byte[] barr = new byte[size];
input.read(barr);
return (Message) getParse(pbClass).invoke(null,barr);
} catch (Exception e) {
throw new RuntimeException("Could not create " + pbClass, e);
}
}
}

try this:
public class ProtoDeserializer implements DeserializationSchema<PBAddress> {
#Override
public TypeInformation<PBAddress> getProducedType() {
return TypeInformation.of(PBAddress.class);
}

https://issues.apache.org/jira/browse/FLINK-11333 is the JIRA ticket tracking the issue of implementing first-class support for Protobuf types with evolvable schema. You'll see that there was a pull request quite some time ago, which hasn't been merged. I believe the problem was that there is no support there for handling state migration in cases where Protobuf was previously being used by registering it with Kryo.
Meanwhile, the Stateful Functions project (statefun is a new API that runs on top of Flink) is based entirely on Protobuf, and it includes support for using Protobuf with Flink: https://github.com/apache/flink-statefun/tree/master/statefun-flink/statefun-flink-common/src/main/java/org/apache/flink/statefun/flink/common/protobuf. (The entry point to that package is ProtobufTypeInformation.java.) I suggest exploring this package (which includes nothing statefun specific); however, it doesn't concern itself with migrations from Kryo either.

Related

How to get the MethodInfo of a Java 14 method reference?

I'm essentially asking the same as this old question, but for Java 14 instead of Java 8. To spare answerers the trouble of navigating to the old question, I'll rephrase it here.
I want to get the name of a function from a referenced method. The following Java code should give you the idea:
public class Main
{
public static void main(String[] args)
{
printMethodName(Main::main);
}
private static void printMethodName(Consumer<String[]> theFunc)
{
String funcName = // somehow get name from theFunc
System.out.println(funcName)
}
}
The equivalent in C# would be:
public class Main
{
public static void Main()
{
var method = Main.Main;
PrintMethodName(method)
}
private static void PrintMethodName(Action action)
{
Console.WriteLine(action.GetMethodInfo().Name);
}
}
According to the accepted answer of the old question, this was not possible in Java 8 without considerable work, such as this solution. Is there a more elegant solution in Java 14?
Getting a method info from a method reference never was a goal on the JDK developer’s side, so no effort was made to change the situation.
However, the approach shown in your link can be simplified. Instead of serializing the information, patching the serialized data, and restoring the information using a replacement object, you can simply intercept the original SerializedLambda object while serializing.
E.g.
public class GetSerializedLambda extends ObjectOutputStream {
public static void main(String[] args) { // example case
var lambda = (Consumer<String[]>&Serializable)GetSerializedLambda::main;
SerializedLambda sl = GetSerializedLambda.get(lambda);
System.out.println(sl.getImplClass() + " " + sl.getImplMethodName());
}
private SerializedLambda info;
GetSerializedLambda() throws IOException {
super(OutputStream.nullOutputStream());
super.enableReplaceObject(true);
}
#Override protected Object replaceObject(Object obj) throws IOException {
if(obj instanceof SerializedLambda) {
info = (SerializedLambda)obj;
obj = null;
}
return obj;
}
public static SerializedLambda get(Object obj) {
try {
GetSerializedLambda getter = new GetSerializedLambda();
getter.writeObject(obj);
return getter.info;
} catch(IOException ex) {
throw new IllegalArgumentException("not a serializable lambda", ex);
}
}
}
which will print GetSerializedLambda main. The only newer feature used here, is the OutputStream.nullOutputStream() to drop the written information immediately. Prior to JDK 11, you could write into a ByteArrayOutputStream and drop the information after the operation which is only slightly less efficient. The example also using var, but this is irrelevant to the actual operation of getting the method information.
The limitations are the same as in JDK 8. It requires a serializable method reference. Further, there is no guaranty that the implementation will map to a method directly. E.g., if you change the example’s declaration to public static void main(String... args), it will print something like lambda$1 when being compiled with Eclipse. When also changing the next line to var lambda = (Consumer<String>&Serializable)GetSerializedLambda::main;, the code will always print a synthetic method name, as using a helper method is unavoidable. But in case of javac, the name is rather something like lambda$main$f23f6912$1 instead of Eclipse’s lambda$1.
In other words, you can expect encountering surprising implementation details. Do not write applications relying on the availability of such information.

Advice added using ByteBuddy AgentBuilder is not getting invoked

I am trying to add a advice in my application so that the onEnter and onExit gets called when a method CassandraFunctions.loadObjectByKey is invoked during execution flow. I used below code to register a advice.
protected void instrument(boolean t) {
Instrumentation instrument = null;
// Get loader initialized in premain class
try {
Class<?> c = Class.forName("my.loader.InstrumentLoader");
java.lang.reflect.Method m = c.getMethod("getInstrument");
instrument = (Instrumentation) m.invoke(null);
} catch (Exception e) {
e.printStackTrace();
}
if(instrument == null) {
return;
}
// Add an advice
String clzName = CassandraFunctionsAdvice.class.getName();
new AgentBuilder.Default()
.with(AgentBuilder.RedefinitionStrategy.RETRANSFORMATION)
.type(ElementMatchers.named("my.functions.CassandraFunctions"))
.transform(
new AgentBuilder.Transformer.ForAdvice()
.include(Class.class.getClassLoader())
.advice(ElementMatchers.named("loadObjectByKey"), clzName))
.installOn(instrument);
}
And the advice class looks like below:
public class CassandraFunctionsAdvice {
#Advice.OnMethodEnter
public static void onEnter(#Advice.Argument(0) String key) {
String debugText = "OnMethodEnter|loadObjectByKey|key=" + key;
System.out.println(debugText);
}
#Advice.OnMethodExit
public static void onExit(#Advice.Thrown Throwable throwable) {
String debugText = "OnMethodExit|loadObjectByKey";
System.out.println(debugText);
}
}
The class that is being instrumented looks like below:
public class CassandraFunctions {
public static Object loadObjectByKey(String key) {
....
return object;
}
}
The instrumented class my.functions.CassandraFunctions is loaded much before the function loadObjectByKey is called on a user request. I am not sure what is missing and why the advice is not getting invoked.
I have already answered your question on the GitHub issue:
The advice code is just a template. The private field is not visible to the code once it is inlined by Byte Buddy.
The question you need to ask yourself is: Could I copy-paste this code to the target class and would it still compile? If no, then you need to change your advice. If you want to manage shared state, you would need to move it to a class that is accessible to the class loader(s) in question and inject it into an appropriate location.

Receive custom exceptions from Restlet Framework in GWT client

I want to use Exception handling with the Restlet Framework and GWT clients.
The Restlet Framework supports the concept of annotated exceptions as described in this post;
http://restlet.com/company/blog/2015/12/21/exception-handling-with-restlet-framework/
In my project i created a LocationNameException
#Status(value = 409)
public class LocationNameException extends Exception
{
...
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
public LocationNameException(String pMessage, Throwable pCause)
{
super(pMessage, pCause);
}
}
And use this in my ServerResource;
#Override
#Transactional(rollbackOn = LocationNameException.class)
public LocationDto postLocation(LocationDto pLocationDto) throws LocationNameException
{
...
Location lLocation = new Location(pLocationDto);
try
{
LocationDao lLocationDao = getLocationDao();
lLocationDao.persist(lLocation);
}
catch (PersistenceException pPersistenceException)
{
throw new LocationNameException("Location requires unique Name", pPersistenceException);
}
...
return lLocationDto;
}
With the interface
public interface LocationListServerResourceInt
{
...
#Post
LocationDto postLocation(LocationDto pLocationDto) throws LocationNameException;
...
}
This works, in the case of an exception the call returns code 409;
And at the GWT client side onFailure() is called;
private class PostLocationCallback implements AsyncCallback<LocationDto>
{
...
#Override
public void onFailure(Throwable pCaught)
{
mCallback.onFailure(pCaught, mLocationDto);
}
}
But parameter pCaught only contains a ResourceException with the status code 409.
My LocationNameException isn't included in the underlying cause stack.
I need this LocationNameException for appropriate error message handling.
The reason is the generated ServerResourceProxy LocationListServerResourceProxyImpl by the Restlet GWT ClientProxyGenerator;
public void postLocation(LocationDto param1, final LocationDto> callback)
{
...
public void handle(Request request, Response response)
{
if (getClientResource().getStatus().isError())
{
callback.onFailure(new ResourceException(getClientResource().getStatus()));
}
else
{
...
}
I think i have to rewrite the Post method in the ClientProxyGenerator;
The LocationNameException is present in the Response data so the Basic approach using the getResponseEntity() method of the ClientResource class should be possible.
Is this the way to go? Or can i catch the LocationNameException exception somewhere else as suggested by Catching annotated exceptions?
It is really hard to try a different approach because of the generated code. Is there an easy way to circumvent the code generator with custom classes?
As already mentioned the LocationNameException is present in the Response data.
Therefore we can get it, just like a normal entity;
...
public void handle(Request request, Response response)
{
if (getClientResource().getStatus().isError())
{
LocationNameException lLocationNameException = null;
boolean serializationError = false;
try
{
if (response.isEntityAvailable())
{
if (MediaType.APPLICATION_JAVA_OBJECT_GWT.equals(response.getEntity().getMediaType()))
{
lLocationNameException = new ObjectRepresentation<LocationNameException>(
response.getEntity().getText(),
(SerializationStreamFactory) MyLocationListServerResourceProxyImpl.this, false)
.getObject();
}
else
{
throw new IOException("Can't parse the enclosed LocationNameException.");
}
}
}
catch (Throwable e)
{
serializationError = true;
callback.onFailure(new ResourceException(e));
}
if (!serializationError)
{
callback.onFailure(lLocationNameException);
}
}
else
{
...
The ClientProxyGenerator needs to know the exception type (in this case LocationNameException). Therefore we specify the exception in the ClientProxy interface;
#Post
void postLocation(LocationDto pLocationDto, AsyncCallback<LocationDto> pResult) throws LocationNameException;
And use getExceptionTypes() or getGenericExceptionTypes() in the ClientProxyGenerator;
Class<?>[] exceptionTypes = method.getExceptionTypes();
java.lang.reflect.Type[] genericExceptionTypes = method.getGenericExceptionTypes();
Of course not all REST methods use a custom exception. When getExceptionTypes() returns an empty list we just return the good old status code;
callback.onFailure(new ResourceException(getClientResource().getStatus()));
With help of Jerome Louvel and Thierry Boileau i created a new ClientProxyGenerator() that supports custom exceptions towards a GWT client;
Just specify the exception from the interface in the ServerResourceProxy (ClientProxy)
and voilà
It is possible to use this custom ClientProxyGenerator() in your project right away.
Download custom ClientProxyGenerator
And place it in a package on the server (for example package com.ludus.server.util)
In GWT module XML change the ClientProxyGenerator to the new version on the server;
And you're ready to go with your custom exceptions, but it would be nice if this extension would be integrated in the Restlet framework.

Migration from Play 2.4.1 to 2.5.6 - Sockets

I have updated my Play Framework version from 2.4.1 to 2.5.6 but now I have a problem with the web sockets management.
I have a Controller class where method liveUpdate() return a WebSocket<String> instance.
In this method I use WebSocket.whenReady() using Out<String> in a HashMap<Out<String>, String> where the key is the client output stream and the value is a String that contains the language information because when I need to send a broadcast message I iterate the HashMap.
Now all this is removed or deprecated in 2.5.6!
Searching the web I found that the new implementation is based on Akka Streams using the Flow class but I have no idea how to adapt my code.
WebSocket.whenReady() is replaced by WebSocket.Text.accept()
Out<?> is replaced by akka stream with Flow class
This is my code:
Alarms.java
public class Alarms extends Controller {
#Inject
private ActiveAlarms activeAlarms;
[...]
public WebSocket liveUpdate() {
return WebSocket.whenReady((in, out) -> {
in.onMessage(language ->{
activeAlarms.register(out, language);
});
in.onClose(() -> activeAlarms.unregister(out));
});
}
[...]
}
ActiveAlarms.java
public class ActiveAlarms{
private HashMap<Out<String>, String> websockets = new HashMap<>();
[...]
public void register(Out<String> out, String language) {
websockets.put(out, language);
updateWebsockets(out, language);
}
public void unregister(Out<String> out) {
websockets.remove(out);
}
private void updateWebsockets(Out<String> s, String lang){
if(s == null) return;
List<AlarmEvent> alarmsList = AlarmEvent.findActive();
ArrayList<AlarmEvent> translatedAlarmsList = new ArrayList<>();
//translate
alarmsList.forEach(e ->{
if(e != null) {
e.setAlarm(e.getAlarm().translate(checkLanguage(lang)));
translatedAlarmsList.add(e);
}
});
//WRITE TO SOCKET
String alarms = Json.stringify(Json.toJson(translatedAlarmsList));
try {
s.write(alarms);
} catch (Exception e2) {
Logger.debug("EX ActiveAlarms --> updateWebSocket " + e2.getMessage());
}
}
private void updateWebsockets(){
websockets.forEach(this::updateWebsockets);
}
[...]
}
Any idea on how to convert my code to the new implementation of WebSocket ?
I think you can easily migrate from one to the other. Did you check this documentation?
If you want to manage the Socket with Akka streams, this is the corresponding documentation part.
Let me know if you still need help to migrate, I'll help you.

Genson Polymorphic / Generic Serialization

I am trying to implement a JSON serialization in Java with Genson 1.3 for polymorphic types, including:
Numbers
Arrays
Enum classes
The SSCCE below demonstrates roughly what I am trying to achieve:
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import com.owlike.genson.Genson;
import com.owlike.genson.GensonBuilder;
/**
* A Short, Self Contained, Compilable, Example for polymorphic serialization
* and deserialization.
*/
public class GensonPolymoprhicRoundTrip {
// our example enum
public static enum RainState {
NO_RAIN,
LIGHT_RAIN,
MODERATE_RAIN,
HEAVY_RAIN,
LIGHT_SNOW,
MODERATE_SNOW,
HEAVY_SNOW;
}
public static class Measurement<T> {
public T value;
public int qualityValue;
public String source;
public Measurement() {
}
public Measurement(T value, int qualityValue, String source) {
this.value = value;
this.qualityValue = qualityValue;
this.source = source;
}
}
public static class DTO {
public List<Measurement<?>> measurements;
public DTO(List<Measurement<?>> measurements) {
this.measurements = measurements;
}
}
public static void main(String... args) {
Genson genson = new GensonBuilder()
.useIndentation(true)
.useRuntimeType(true)
.useClassMetadataWithStaticType(false)
.addAlias("RainState", RainState.class)
.useClassMetadata(true)
.create();
DTO dto = new DTO(
new ArrayList(Arrays.asList(
new Measurement<Double>(15.5, 8500, "TEMP_SENSOR"),
new Measurement<double[]>(new double[] {
2.5,
1.5,
2.0
}, 8500, "WIND_SPEED"),
new Measurement<RainState>(RainState.LIGHT_RAIN, 8500, "RAIN_SENSOR")
)));
String json = genson.serialize(dto);
System.out.println(json);
DTO deserialized = genson.deserialize(json, DTO.class);
}
}
Numbers and Arrays worked well out-of-the-box, but the enum class is providing a bit of a challenge. In this case the serialized JSON form would have to be IMO a JSON object including a:
type member
value member
Looking at the EnumConverter class I see that I would need to provide a custom Converter. However I can't quite grasp how to properly register the Converter so that it would be called during deserialization. How should this serialization be solved using Genson?
Great for providing a complete example!
First problem is that DTO doesn't have a no arg constructor, but Genson supports classes even with constructors that have arguments. You just have to enable it via the builder with 'useConstructorWithArguments(true)'.
However this will not solve the complete problem. For the moment Genson has full polymorphic support only for types that are serialized as a json object. Because Genson will add a property called '#class' to it. There is an open issue for that.
Probably the best solution that should work with most situations would be to define a converter that automatically wraps all the values in json objects, so the converter that handles class metadata will be able to generate it. This can be a "good enough" solution while waiting for it to be officially supported by Genson.
So first define the wrapping converter
public static class LiteralAsObjectConverter<T> implements Converter<T> {
private final Converter<T> concreteConverter;
public LiteralAsObjectConverter(Converter<T> concreteConverter) {
this.concreteConverter = concreteConverter;
}
#Override
public void serialize(T object, ObjectWriter writer, Context ctx) throws Exception {
writer.beginObject().writeName("value");
concreteConverter.serialize(object, writer, ctx);
writer.endObject();
}
#Override
public T deserialize(ObjectReader reader, Context ctx) throws Exception {
reader.beginObject();
T instance = null;
while (reader.hasNext()) {
reader.next();
if (reader.name().equals("value")) instance = concreteConverter.deserialize(reader, ctx);
else throw new IllegalStateException(String.format("Encountered unexpected property named '%s'", reader.name()));
}
reader.endObject();
return instance;
}
}
Then you need to register it with a ChainedFactory which would allow you to delegate to the default converter (this way it works automatically with any other type).
Genson genson = new GensonBuilder()
.useIndentation(true)
.useConstructorWithArguments(true)
.useRuntimeType(true)
.addAlias("RainState", RainState.class)
.useClassMetadata(true)
.withConverterFactory(new ChainedFactory() {
#Override
protected Converter<?> create(Type type, Genson genson, Converter<?> nextConverter) {
if (Wrapper.toAnnotatedElement(nextConverter).isAnnotationPresent(HandleClassMetadata.class)) {
return new LiteralAsObjectConverter(nextConverter);
} else {
return nextConverter;
}
}
}).create();
The downside with this solution is that useClassMetadataWithStaticType needs to be set to true...but well I guess it is acceptable as it's an optim and can be fixed but would imply some changes in Gensons code, the rest still works.
If you are feeling interested by this problem it would be great you attempted to give a shot to that issue and open a PR to provide this feature as part of Genson.

Categories