We Keep Coding

Copy fields across objects of different type in gRPC

Suppose I have two proto buffer types:
message MessageType1 {
SomeType1 field1 = 1;
SomeType2 field2 = 2;
SomeType3 field3 = 3;
}
message MessageType2 {
SomeType1 field1 = 1;
SomeType2 field2 = 2;
SomeType4 field4 = 3;
}
Then in Java I would like to be able to use one object as a template to another:
MessageType1 message1 = ...;
MessageType2 message2 = MessageType2.newBuilder()
.usingTemplate(message1) // sets field1 & field2 only
.setField4(someValue)
.build()
instead of
MessageType1 message1 = ...;
MessageType2 message2 = MessageType2.newBuilder()
.setField1(message1.getField1())
.setField2(message1.getField2())
.setField4(someValue)
.build()
Why do I need this? My gRPC service is designed to take incoming data of one type (message1) which is almost identical to another message of a different type (message2) -- which needs to be sent out. The amount of identical fields is huge and copy code is mundane. Manual solution also has a disadvantage of a miss if a new field gets added.
There exists a template method (object.newBuilder(template)) which allows templating object of the same type, but how about templating between different types?
I could, of course, write a small reflection utility which inspects all members (methods?) and manually copies data over, but generated code looks discouraging and ugly for this sort of quest.
Is there any good approach to tackle this?

It turned out to be not so complicated. I wrote a small utility which would evaluate and match FieldDescriptors (something that gRPC generates). In my world it is enough to match them by name and type. Full solution here:
/**
* Copies fields from source to dest. Only copies fields if they are set, have matching name and type as their counterparts in dest.
*/
public static void copyCommonFields(#Nonnull GeneratedMessageV3 source, #Nonnull com.google.protobuf.GeneratedMessageV3.Builder<?> destBuilder) {
Map<FieldDescriptorKeyElements, Descriptors.FieldDescriptor> elementsInSource = Maps.uniqueIndex(source.getDescriptorForType().getFields(), FieldDescriptorKeyElements::new);
Map<FieldDescriptorKeyElements, Descriptors.FieldDescriptor> elementsInDest = Maps.uniqueIndex(destBuilder.getDescriptorForType().getFields(), FieldDescriptorKeyElements::new);
// those two above could even be cached if necessary as this is static info
Set<FieldDescriptorKeyElements> elementsInBoth = Sets.intersection(elementsInSource.keySet(), elementsInDest.keySet());
for (Map.Entry<Descriptors.FieldDescriptor, Object> entry : source.getAllFields().entrySet()) {
Descriptors.FieldDescriptor descriptor = entry.getKey();
FieldDescriptorKeyElements keyElements = new FieldDescriptorKeyElements(descriptor);
if (entry.getValue() != null && elementsInBoth.contains(keyElements)) {
destBuilder.setField(elementsInDest.get(keyElements), entry.getValue());
}
}
}
// used for convenient/quick lookups in a Set
private static final class FieldDescriptorKeyElements {
final String fieldName;
final Descriptors.FieldDescriptor.JavaType javaType;
final boolean isRepeated;
private FieldDescriptorKeyElements(Descriptors.FieldDescriptor fieldDescriptor) {
this.fieldName = fieldDescriptor.getName();
this.javaType = fieldDescriptor.getJavaType();
this.isRepeated = fieldDescriptor.isRepeated();
}
#Override
public int hashCode() {
return Objects.hash(fieldName, javaType, isRepeated);
}
#Override
public boolean equals(Object obj) {
if (obj == null || !(obj instanceof FieldDescriptorKeyElements)) {
return false;
}
FieldDescriptorKeyElements other = (FieldDescriptorKeyElements) obj;
return Objects.equals(this.fieldName, other.fieldName) &&
Objects.equals(this.javaType, other.javaType) &&
Objects.equals(this.isRepeated, other.isRepeated);
}
}

Answering your specific question: no, there is no template based way to do this. However, there are some other ways to get the same effect:
If you don't care about performance and the field numbers are the same between the messages, you can serialize the first message to bytes and deserialize them back as the new message. This requires that all the fields in the first message must match the type and id number of those in the second message (though, the second message can have other fields). This is probably not a good idea.
Extract the common fields to another message, and share that message. For example:
proto:
message Common {
SomeType1 field1 = 1;
SomeType2 field2 = 2;
SomeType3 field3 = 3;
}
message MessageType1 {
Common common = 1;
// ...
}
message MessageType2 {
Common common = 1;
// ...
}
Then, you can share the messages in code:
MessageType1 message1 = ...;
MessageType2 message2 = MessageType2.newBuilder()
.setCommon(message1.getCommon())
.build();
This is the probably the better solution.
Lastly, as you mentioned, you could resort to reflection. This is probably the most verbose and slowest way, but it would allow you the most control (aside from manually copying over the fields). Not recommended.

open method is not being called in Flink RichMapFunction

I am trying to use apache flink for a simple example described at Shortcuts. However, I noticed the open method is never called and as a result I get null pointer exception on first line of map function.
public class MyMap extends RichMapFunction<Integer, Integer> {
private ValueState<Integer> test;
public void open(Configuration cfg) {
test = getRuntimeContext().getState(new
ValueStateDescriptor<Integer>("myTest", Integer.class));
System.out.println("1:" + test);
}
#Override
public Integer map(Integer i) throws Exception {
System.out.println("2:" + test.value()); //test is null here
test.update(test.value() == null? 1: test.value() + 1);
System.out.println("3:" + test.value());
return i;
}
}

Update:
Did you try to #Override the open function?
test test.value is supposed to be null the first time.
You are on keyed context, which means that each message has a key which flink already knows about. When you enter a stateful operator, flink will try to fetch a value for that key from the configured state backend. Unless you configure the ValueStateDescriptor to have a default (it is deprecated), the first time you process a message for a specific key the state will be null. Thus your application should handle the null value.
Try the following example (my java is rusty, this is in scala. Ask me if you need help converting it):
env.fromElements(("key1", 2),("key2", 4), ("key1", 5))
.keyBy(_._1)
.map {
new RichMapFunction[(String, Int), (String, Int)] {
lazy val stateTypeInfo: TypeInformation[Int] = implicitly[TypeInformation[Int]]
lazy val serializer: TypeSerializer[Int] = stateTypeInfo.createSerializer(getRuntimeContext.getExecutionConfig)
lazy val stateDescriptor = new ValueStateDescriptor[Int]("dummy state", serializer)
var testVar: ValueState[Int] = _
override def open(config: Configuration) = {
testVar = this.getRuntimeContext.getState(stateDescriptor)
}
override def map(in: (String, Int)): (String, Int) = {
println(s"message $in")
println(s"state ${testVar.value()}")
println()
val sum = Option(testVar.value()).getOrElse(0) + in._2
testVar.update(sum)
(in._1, sum)
}
}
}.print()
env.execute()
This should produce:
message (key1,2) (first time key1 is seen)
state null (state is null)
(key1,2) (output)
message (key2,4) (first time key2 is seen)
state null (state is null)
(key2,4) (output)
message (key1,5) (second time key1 is seen!! We stored something there!)
state 2 (we stored a 2)
(key1,7) (thus output is 2+5=7)

I had the similar problem. I could solve the problem by replacing the following import:
import java.lang.module.Configuration;
with this one:
import org.apache.flink.configuration.Configuration;

What does the following code analysis mean?

I have an object instantiated like the following in only one place in my code(AggregateFunctions).
private String selectColumns() {
String query = "SELECT ";
if (this.distinctResults) {
query = query + "DISTINCT ";
}
SelectColumn selectColumn = new SelectColumn(this);
if (!this.applyAggregation) {
for (Object object : this.columns) {
query = selectColumn.selectColumn(query, object);
}
} else {
AggregateFunctions aggregateFunctions = new AggregateFunctions(this);
query = query + aggregateFunctions.select();
}
//Remove extra ', '
query = query.substring(0, query.length() - 2) + " FROM ";
return query;
}
The constructors:
public AggregateFunctions(#NotNull SqlQueryGenerator sqlQueryGenerator) {
this.spaceEncloser = sqlQueryGenerator.getSpaceEncloser();
this.selectColumn = new SelectColumn(sqlQueryGenerator);
JSONObject formData = sqlQueryGenerator.getFormData();
this.columns = formData.getJSONArray("columns");
this.aggregateJson = formData.getJSONObject("functions").getJSONArray("aggregate");
this.aggregatesList = new ArrayList<Aggregate>();
prepareAggregates();
this.query = new StringBuilder();
}
public SelectColumn(SqlQueryGenerator sqlQueryGenerator) {
this.sqlQueryGenerator = sqlQueryGenerator;
}
But IntelliJ Code Analysis says the following about recursive calls. Basically I didn't understand the meaning. Can anyone elaborate to help me understand?
Problem synopsis
Constructor has usage(s) but they all belong to recursive calls chain that has no members reachable from entry points.   
Problem resolution
Safe delete
Comment out
Add as Entry Point

This is a warning from the Unused declaration inspection. IntelliJ IDEA thinks the constructor is not reachable from any entry points. The constructor is not unused however, but the usages are themselves unreachable.
If this is not the case for your code, it may be a bug in IntelliJ IDEA.

Probably in the constructor of AggregateFunctions in the code that you call you go back to the method selectColumns() in the other class. This way a recurrsion is never going to end.
My guess is that either here
JSONObject formData = sqlQueryGenerator.getFormData();
Or somewhere in here:
this.selectColumn = new SelectColumn(sqlQueryGenerator);
You go to the previous class and to the same method that creates a new aggreggate and a loop is happening.
You call the AggregateFunction with this - which is the same object. But then in the constructor you call methods of this. Check these methods and if any of them has another creation of AggregateFunction object - there is your problem.

I had this issue and it was because the object was not used anywhere else

Multiple #ImportStatic annotations on a single class using the Java Preon library

I'm wondering if there is a way to use multiple #ImportStatic annotations for a single class with Preon?
I tried:
#ImportStatic(classA.class)
#ImportStatic(classB.class)
// Also tried:
#ImportStatic(classA.class classB.class)
// And:
#ImportStatic(classA.class,classB.class)
None of these are valid however...
I have a specification that asks me to look at the enum value (classA) of the outer (parent) class, if it matches a specific value, then I must also do an enum check on on one of the parents other object's enum values (classB) before proceeding to read the next field.
EDIT:
Here's my first attempt at a solution, basically copy ImportStatic and make an ImportStatic2 annotation... Have to see how well it works.
diff --git a/preon-binding/src/main/java/org/codehaus/preon/el/ImportSupportingObjectResolverContext.java b/preon-binding/src/main/java/org/codehaus/preon/el/ImportSupportingObjectResolverContext.java
index b737719..8f4946c 100644
--- a/preon-binding/src/main/java/org/codehaus/preon/el/ImportSupportingObjectResolverContext.java
+++ b/preon-binding/src/main/java/org/codehaus/preon/el/ImportSupportingObjectResolverContext.java
## -89,14 +89,23 ## public class ImportSupportingObjectResolverContext implements
public static ObjectResolverContext decorate(ObjectResolverContext context,
Class<?> type) {
- if (type.isAnnotationPresent(ImportStatic.class)) {
+ if (type.isAnnotationPresent(ImportStatic.class) || type.isAnnotationPresent(ImportStatic2.class)) {
ImportSupportingObjectResolverContext replacement = new ImportSupportingObjectResolverContext();
Map<String, Reference<Resolver>> references = new HashMap<String, Reference<Resolver>>();
+ if (type.isAnnotationPresent(ImportStatic.class)) {
for (Class<?> imported : type.getAnnotation(ImportStatic.class)
.value()) {
references.put(imported.getSimpleName(), new ClassReference(
imported, replacement));
}
+ }
+ if (type.isAnnotationPresent(ImportStatic2.class)) {
+ for (Class<?> imported : type.getAnnotation(ImportStatic2.class)
+ .value()) {
+ references.put(imported.getSimpleName(), new ClassReference(
+ imported, replacement));
+ }
+ }
replacement.context = context;
replacement.references = references;
return replacement;

It appears using the syntax as follows works:
#ImportStatic({First.class,Second.class})

clojure/scala interop?

I am attempting to interop to this simple scala code, but am having some troubles.
package indicators
class DoubleRingBuffer(val capacity:Int=1000) {
var elements = new Array[Double](capacity);
private var head=capacity-1
private var max=0
def size ():Int = {
return max+1
}
def add(obj:Double):Double = {
head-=1
if (head<0) head=capacity-1
return set(max+1,obj)
}
def set(i:Int,obj:Double):Double = {
System.out.println("HI")
if (i>=capacity || i<0)
throw new IndexOutOfBoundsException(i+" out of bounds")
if (i>=max) max=i
var index = (head+i)%capacity
var prev = elements(index)
elements(index)=obj
return prev
}
def get(i:Int=0):Double = {
System.out.println("size is "+size())
if (i>=size() || i<0)
throw new IndexOutOfBoundsException(i+" out of bounds")
var index = (head+i)%capacity
return elements(index)
}
}
In clojure, i do this
(import 'indicators.DoubleRingBuffer)
(def b (DoubleRingBuffer. 100))
(pr (.size b)) ;;ERROR: No matching field found: size for class indicators.DoubleRingBuffer
(pr (.get b 33)) ;;returns 0: should throw an index out of bounds error!
(pr (.get b 100)) ;;throws index out of bounds error, as it should
In addition, i do not get any output to the console! Testing this code using scala works as expected. Whats going on here and how can i fix it so that clojure can use the scala code?

Try these in REPL:
(class b) will probably tell you it's indicators.DoubleRingBuffer.
(vec (.getDeclaredMethods (class b))) will give you a vector of all methods declared in your class as if it was a Java class, so you can see their signatures.
Now, call your methods as seen in the signatures, with these method names and parameters.
I have a feeling the problem is in Scala's dealing with default value for method parameter.
EDIT: As OP described in a comment, it isn't.
If that doesn't work you can try to decompile your Scala bytecode to Java to find out how does DoubleRingBuffer class look like.