We Keep Coding

Using protobuf with flink

I'm using flink to read data from kafka and convert it to protobuf. The problem I'm facing is when I run the java application I get the below error. If I modify the unknownFields variable name to something else, it works but it's hard to make this change on all protobuf classes.
I also tried to deserialize directly when reading from kafka but I'm not sure what should be the TypeInformation to be returned for getProducedType() method.
public static class ProtoDeserializer implements DeserializationSchema{
#Override
public TypeInformation getProducedType() {
// TODO Auto-generated method stub
return PrimitiveArrayTypeInfo.BYTE_PRIMITIVE_ARRAY_TYPE_INFO;
}
Appreciate all the help. Thanks.
java.lang.RuntimeException: The field protected com.google.protobuf.UnknownFieldSet com.google.protobuf.GeneratedMessage.unknownFields is already contained in the hierarchy of the class com.google.protobuf.GeneratedMessage.Please use unique field names through your classes hierarchy
at org.apache.flink.api.java.typeutils.TypeExtractor.getAllDeclaredFields(TypeExtractor.java:1594)
at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1515)
at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1412)
at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
at org.apache.flink.api.java.typeutils.TypeExtractor.privateCreateTypeInfo(TypeExtractor.java:437)
at org.apache.flink.api.java.typeutils.TypeExtractor.getUnaryOperatorReturnType(TypeExtractor.java:306)
at org.apache.flink.api.java.typeutils.TypeExtractor.getFlatMapReturnTypes(TypeExtractor.java:133)
at org.apache.flink.streaming.api.datastream.DataStream.flatMap(DataStream.java:529)
Code:
FlinkKafkaConsumer09<byte[]> kafkaConsumer = new FlinkKafkaConsumer09<>("testArr",new ByteDes(),p);
DataStream<byte[]> input = env.addSource(kafkaConsumer);
DataStream<PBAddress> protoData = input.map(new RichMapFunction<byte[], PBAddress>() {
#Override
public PBAddress map(byte[] value) throws Exception {
PBAddress addr = PBAddress.parseFrom(value);
return addr;
}
});

Maybe you should try this follow:
env.getConfig().registerTypeWithKryoSerializer(PBAddress. class,ProtobufSerializer.class);
or
env.getConfig().registerTypeWithKryoSerializer(PBAddress. class,PBAddressSerializer.class);
public class PBAddressSerializer extends Serializer<Message> {
final private Map<Class,Method> hashMap = new HashMap<Class, Method>();
protected Method getParse(Class cls) throws NoSuchMethodException {
Method method = hashMap.get(cls);
if (method == null) {
method = cls.getMethod("parseFrom",new Class[]{byte[].class});
hashMap.put(cls,method);
}
return method;
}
#Override
public void write(Kryo kryo, Output output, Message message) {
byte[] ser = message.toByteArray();
output.writeInt(ser.length,true);
output.writeBytes(ser);
}
#Override
public Message read(Kryo kryo, Input input, Class<Message> pbClass) {
try {
int size = input.readInt(true);
byte[] barr = new byte[size];
input.read(barr);
return (Message) getParse(pbClass).invoke(null,barr);
} catch (Exception e) {
throw new RuntimeException("Could not create " + pbClass, e);
}
}
}

try this:
public class ProtoDeserializer implements DeserializationSchema<PBAddress> {
#Override
public TypeInformation<PBAddress> getProducedType() {
return TypeInformation.of(PBAddress.class);
}

https://issues.apache.org/jira/browse/FLINK-11333 is the JIRA ticket tracking the issue of implementing first-class support for Protobuf types with evolvable schema. You'll see that there was a pull request quite some time ago, which hasn't been merged. I believe the problem was that there is no support there for handling state migration in cases where Protobuf was previously being used by registering it with Kryo.
Meanwhile, the Stateful Functions project (statefun is a new API that runs on top of Flink) is based entirely on Protobuf, and it includes support for using Protobuf with Flink: https://github.com/apache/flink-statefun/tree/master/statefun-flink/statefun-flink-common/src/main/java/org/apache/flink/statefun/flink/common/protobuf. (The entry point to that package is ProtobufTypeInformation.java.) I suggest exploring this package (which includes nothing statefun specific); however, it doesn't concern itself with migrations from Kryo either.

Check if class exists without running into ClassNotFoundException

In order to schedule the execution of a job, i get the name of a class as a string-input.
This class may be in one of two packages, but i don't know which one so i have to check this.
By now, i have two try-catch-blocks
Class<Job> clazz;
String className; //the above mentioned string, will be initialized
try {
clazz = (Class<Job>) Class.forName("package.one." + className);
} catch (ClassNotFoundException ex) {
try {
clazz = (Class<Job>) Class.forName("package.two." + className);
} catch (ClassNotFoundException ex1) {
//some error handling code here, as the class is
//in neither of the two packages
}
}
For more packages this will get uglier and more unreadable. Furthermore, it is - for me - against the concept of exceptions, as exceptions should'nt be expected/used for flow-control!
Is there any way to rewrite this without the utilization of the ClassNotFoundException?

I'd stick to the Class.forName method for that.
You can store the class and package names in Collections or Sets and loop through those elements. When you get a ClassNotFoundException, you just continue your search. If you don't get an exception, you exit the loop using break, as you have found the class you were looking for.
The reason I'd go for Class.forName is that it loads the class for you if it had not been already loaded by the VM. This is quite a powerful side effect.
It basically relieves you of the trouble of digging through the whole CLASSPATH and looking for class files to load in the VM and dealing with issues such as whether the class has already been loaded or not by the VM.
EDIT:
Jakob Jenkov has written some really great articles/tutorials on Java. I've found them extremely useful when dealing with reflection, class loaders and concurrency (some of the "hardest" aspects of Java).
Here's a good article on the Java class loader by him, for if you still decide not to use Class.forName.

public Class<Job> getClass(String className) {
String packages[] = { "package.one.", "package.two." };
for (int j = 0; j < packages.length; j++) {
try {
return (Class<Job>) Class.forName(packages[j] + className);
} catch (ClassNotFoundException ex) {
System.out.println("Package "+packages[j]+" is not worked");
}
}
return null;
}

You can use Guava's Reflection utilities to get the ClassInfo of every class loaded in the classpath.
ClassLoader classLoader = ClassLoader.getSystemClassLoader();
ClassPath classPath = ClassPath.from(classLoader);
ImmutableSet<ClassInfo> set = classPath.getTopLevelClasses();
for (ClassInfo ci : set) {
System.out.println(ci.getName());
}
In the loop you can implement your custom logic to load the class with the className you're providing.

In this case, I wouldn't worry about using the ClassNotFoundException. While in general a case can be made to not use exceptions for flow control, here it hardly counts as such.
I'd probably wrap it in a function, like so
public static String getPackageForClass(String className, String... packageNames) {
for (String packageName : packageNames) {
try {
Class.forName(packageName + className);
return packageName;
} catch (ClassNotFoundException ignored) {
}
}
return "";
}
or return the class directly, if you so wish
public static Class getPackageForClass(String className, String... packageNames) {
for (String packageName : packageNames) {
try {
return Class.forName(packageName + className);
} catch (ClassNotFoundException ignored) {
}
}
return null;
}

Is there an utility to parse an URL without checked exception in java?

I'd like to create an URL constant, like so
public static final URL REMOTE_URL = new URL("http://example.com/");
But I can't since the constructor throw a checked exception. Right now I use
public static final URL REMOTE_URL = createUrl("http://example.com/");
private static URL createUrl(String url) {
try {
return new URL(url);
} catch (MalformedURLException error) {
throw new IllegalArgumentException(error.getMessage(), error);
}
}
But it feel like reinventing the wheel. I can't possibly be the only one who want to use a URL constant no? So I was wondering if there is third-party toolbox library (like guava or apache-commons, or something else, anything) or even better, something in standard Java that include this facilities? That would help me when we start a new project by reducing the size of our util package :) .

This complaint must have been a common one, because Java has (since 1.7) introduced the URI class. You can construct it using either of the following ways:
new URI(), which throws a checked exception
URI.create(), which throws an unchecked exception
For URIs/URLs like yours that are known to come from a safe source, you can use the URI.create() variant and not have to worry about catching the exception as you know it won't be thrown.
Unfortunately, sometimes you can't use a URI and you still need a URL. There's no standard method (that I have found so far) of converting a URI into a URL that doesn't throw a checked exception.

Since everyone else is just commenting, I will provide the answer which is that no there is no standard way to do what you want :)
Also, since you mention apache commons and google guava, I would point out that standard is not exactly the correct word to use either....maybe you want open-source, free, or just third-party.

Just throwing this one in the mix - there's a lot of stylistic variation for accomplishing the same thing, this one initializes in a static init block, but it can't be final.
public static URL g_url;
static {
try {
g_url = new URL("http://www.example.org");
} catch (Exception e) {
e.printStackTrace();
}
}

You cant extend URL because it is final, but you can create a new class, perhaps named MyURL, and have it proxy the URL methods to a private (or protected) URL member.
Here is the beginnings of such a class:
package blammo.url;
import java.net.MalformedURLException;
import java.net.URL;
public class MyURL
{
private URL containedURL;
public MyURL(final String spec)
{
try
{
containedURL = new URL(spec);
}
catch (MalformedURLException exception)
{
throw new RuntimeException(exception);
}
}
public String getAuthority()
{
return containedURL.getAuthority();
}
// required for stuff that needs a URL class.
public URL getContainedURL()
{
return containedURL;
}
}

Java CodeModel: annotation with class as value - class not on classpath

I need to generate code and in this case annotations using suns CodeModel library. The annotations value is a Class object. However that class is not known at compile time. The solution I have now is:
JAnnotationUse oneToMany = field.annotate(OneToMany.class)
.param("targetEntity", Class.forName(className);
However this obviously requires that the according class is on the classpath. I want to avoid the user having to deal with such issues.
Of course the other option would be to generate the java source code file and manipulate it afterwards (as pure string) but that seems very messy.
Is there any way with CodeModel to use the plain string className + ".class" instead?

You can handle the case where the class it not in the classpath by catching the ClassNotFoundException:
try {
JAnnotationUse oneToMany = field.annotate(OneToMany.class)
.param("targetEntity", Class.forName(className);
} catch (ClassNotFoundException e) {
// handle the exception
}
The part where you handle the exception really depends on what you're trying to achieve, depending on the situation you might want to log the error/add another anotation/throw an exception/...

Ok. Here my solution. Use javaassist to generate the class and the according Class object at runtime. Since the end result will be
#OneToMany(targetEntity = com.foo.Bar.class)
as normal text in a java source code file, it does not matter that this generated "dummy class" is not actually equivalent to the real actual class that was referenced.
So we adjust the creation of the annotation as Guillaume indicated and catch the ClassNotfoundExeption. In the exception handler we call the generateDummyClass method that uses javaassist to create a Class object:
JAnnotationUse oneToMany = field.annotate(OneToMany.class);
//...snipped...
try {
oneToMany.param("targetEntity", Class.forName(jpaProperty.getTargetEntity()));
} catch (ClassNotFoundException ex) {
try {
Class targetEntityClass = generateDummyClass(jpaProperty.getTargetEntity());
oneToMany.param("targetEntity", targetEntityClass);
} catch (CannotCompileException compileEx) {
throw ex;
}
}
}
Method generateDummyClass that create a class object at runtime using javaassist:
private Class<?> generateDummyClass(String fullQualifedClassName)
throws IOException, CannotCompileException {
ClassPool pool = ClassPool.getDefault();
pool.insertClassPath(new ClassClassPath(this.getClass()));
CtClass ctClass = pool.makeClass(fullQualifedClassName);
Class<?> clazz = ctClass.toClass();
return clazz;
}
And we get a java source code file with:
#OneToMany(targetEntity = org.foo.Bar.class)
private Set<Bar> bars = new HashSet<Bar>();

Trouble accessing fields of a serialized object in Java

I have instantized a class that implements Serializable and I am trying to stream that object like this:
try{
Socket socket = new Socket("localhost", 8000);
ObjectOutputStream toServer = new ObjectOutputStream(socket.getOutputStream());
toServer.writeObject(myObject);
} catch (IOException ex) {
System.err.println(ex);
}
All good so far right? Then I am trying to read the fields of that object like this:
//This is an inner class
class HandleClient implements Runnable{
private ObjectInputStream fromClient;
private Socket socket; // This socket was established earlier
try {
fromClient = new ObjectInputStream(socket.getInputStream());
GetField inputObjectFields = fromClient.readFields();
double myFristVariable = inputObjectFields.get("myFirstVariable", 0);
int mySecondVariable = inputObjectFields.get("mySecondVariable", 0);
//do stuff
} catch (IOException ex) {
System.err.println(ex);
} catch (ClassNotFoundException ex) {
System.err.println(ex);
} finally {
try {
fromClient.close();
} catch (Exception ex) {
ex.printStackTrace();
}
}
}
But I always get the error:
java.io.NotActiveException: not in call to readObject
This is my first time streaming objects instead of primitive data types, what am I doing wrong?
BONUS
When I do get this working correctly, is the ENTIRE CLASS passed with the serialized object (i.e. will I have access to the methods of the object's class)? My reading suggests that the entire class is passed with the object, but I have been unable to use the objects methods thus far. How exactly do I call on the object's methods?
In addition to my code above I also experimented with the readObject method, but I was probably using it wrong too because I couldn't get it to work. Please enlighten me.

To answer your first question:
You need to use ObjectInputStream.readObject to deserialize. You cannot read individual fields from the stream*.
fromClient = new ObjectInputStream(socket.getInputStream());
Object myObject = fromClient.readObject();
Don't forget to flush the output stream when writing!
The second question is a little more complex. What the serialization mechanism does is write a class identifier to the stream followed by the serialized object data. When it deserializes it will read the class identifier and attempt to load that class (if it isn't already loaded). It will then instantiate the object using the no-arg constructor and call the private readObject(ObjectInputStream) method. Yes, that's right, it calls a private method from outside the class. Java serialization is special.
If the class cannot be found (i.e. if it's not on the classpath) then an exception will be thrown; otherwise you'll get a fully deserialized object of the correct type assuming no other errors are found.
For example, suppose you have the following classes:
class Server {
public static void main(String[] args) {
// Set up an OutputStream sink, e.g. writing to a socket (not shown)
...
ObjectOutputStream out = new ObjectOutputStream(sink);
out.writeObject(new Data("data goes here"));
out.flush();
out.close();
}
}
class Client {
public static void main(String[] args) {
// Set up an InputStream source (not shown)
...
ObjectInputStream in = new ObjectInputStream(source);
Data d = (Data)in.readObject();
System.out.println(d.getData());
}
}
class Data implements java.io.Serializable {
private String data;
public Data(String d) {
data = d;
}
public String getData() {
return data;
}
}
Now suppose you put those classes into three jars (one class per jar): server.jar, client.jar and data.jar. If you run the following commands then it should all work:
java -cp server.jar:data.jar Server
java -cp client.jar:data.jar Client
But if you do this:
java -cp server.jar:data.jar Server
java -cp client.jar Client
then you'll get a ClassNotFoundException because the client doesn't know how to find the Data class.
Long story short: the class itself is not written to the stream. If deserialization succeeds then you will have access to the object as though it had been created locally, but you will have to downcast the result of readObject to the expected type.
There is some complexity around versioning that I've ignored for now. Take a look at serialVersionUID and how to deal with changes to serializable classes if versioning is likely to be an issue.
*Not strictly true. You can call readFields inside the serializable object's readObject method (or readResolve), but you cannot call it from outside the deserialization mechanism. Does that make sense? It's a little hard to explain.

Looking at the code for ObjectInputStream.readFields(), that exception is called because the curContext field is null. You should call fromClient.readObject() before calling readFields(), as it will set the curContext. Note that readObject() will return the instance that is being serialized, which may be of more use to you.