We Keep Coding

Trying to create a generic Kafka consumer factory

I'm trying to implement a generic Kafka consumer factory. By generic I mean that the factory will receive generics K,V that will indicate the types of the key and the value of the consumed message.
public class KafkaConsumerFactory<K, V> {
private Properties properties;
In my constructor I'm calling a method that responsible for validating that the generics K,V are compatible with the deseralization classes :
public KafkaConsumerFactory(Properties p) {
....
....
validateGenericsWithDeserializationProperties();
}
The relevant function :
private void validateGenericsWithDeserializationProperties() throws Exception {
try {
String keyDeserializerClass = (String) (this.properties.get(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG));
String valueDeserializerClass = (String) (this.properties.get(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG));
Deserializer<K> keyDeserializer = (Deserializer<K>) Class.forName(keyDeserializerClass).getConstructor().newInstance();
Deserializer<V> valueDeserializer = (Deserializer<V>) Class.forName(valueDeserializerClass ).getConstructor().newInstance();
System.out.println(keyDeserializer.toString());
System.out.println(valueDeserializer.toString());
} catch (Exception e) {
throw new Exception("There is an incompatibility with the deserialization parameters in the properties with the generics <K,V>");
}
}
But it seems that when I pass generics K,V that aren't compatible with the deserializer class the code doesn't throw exception , for example:
K=String,
V=Long,
key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
value.deserializer=org.apache.kafka.common.serialization.StringDeserializer
I was expecting an exception on the downcasting on the following lines :
Deserializer<K> keyDeserializer = (Deserializer<K>)(Class.forName(keyDeserializerClass).getConstructor().newInstance());
Deserializer<V> valueDeserializer = (Deserializer<V>)Class.forName(valueDeserializerClass).getConstructor().newInstance();
I understand that those lines won't throw exception because of erasure of generic types.
Is there any other way to check for incompatibility between the deserialization class to the type of generic before trying to consume data?

Deserializer<K> keyDeserializer = (Deserializer<K>) Class.forName(keyDeserializerClass).getConstructor().newInstance();
Deserializer<V> valueDeserializer = (Deserializer<V>) Class.forName(valueDeserializerClass ).getConstructor().newInstance();
Why do you want to explicity create Deserializer object?
It will automatically be created as per the one specified in the properties.
If you want to verify (make sure) the appropriate deserializer is put for the appropriate key/value type, you can take the keyClass, valueClass in the constructor of your KafkaConsumerFactory
public KafkaConsumerFactory(Properties p, Class<K> keyClass, Class<V> valueClass) {
p.putIfAbsent("key.deserializer", getDeserializer(keyClass).getName());
p.putIfAbsent("value.deserializer", getDeserializer(valueClass).getName());
}
and then you can automatically map the keyClass, valueClass to appropriate deserializer and override them in the properties.
Class<Deserializer> getDeserializer(Class clazz) {
if(Long.class.equals(clazz)) { return LongDeserializer.class; }
else if (...)
...
}
If you want to support new custom deserializers, you can use an interface like CustomDeserializer for example that has a method called canBeProcessed(Class clazz) which determines if the object can be deserialized and ensure that this interface is implemented by all your custom deserializers and that the method is overriden.
If you are looking to support already existing deserializers, you can write the code to check the compatability of the deserializer with a certain class in your KafkaConsumerFactory itself.
Usually, serializers/deserializers are mapped to certain class(es), which means that you don't need an object to verify if it can be serialized/deserialized because the data in the object has pretty less (if not nothing) to do with the actual serialization/deserialization.
In case, if the class type information isn't enough for finding out whether it's object can be deserialized or not, the option left is to create a dummy object and then pass it to the deserialize() method. If it raises an exception, then it is invalid, not otherwise.

Getting behavior of Java's Class<? extends Map> in .NET

I have a generic class in java defined as:
public static class KeyCountMap<T>
{
private Map<T, MutableInt> map = new LinkedHashMap<T, MutableInt>();
// ... rest of the properties...
public KeyCountMap()
{ }
#SuppressWarnings({ "unchecked", "rawtypes" })
public KeyCountMap(Class<? extends Map> mapType) throws InstantiationException, IllegalAccessException
{
map = mapType.newInstance();
}
//... rest of the methods...
}
I have defined same class in .NET as:
public static class KeyCountMap<T>
{
private Dictionary<T, MutableInt> map = new Dictionary<T, MutableInt>();
// ... rest of properties...
public KeyCountMap()
{ }
public void KeyCountMap<T>(T obj) where T : Dictionary<T, MutableInt>
{
obj = new T(); // Unable to define new instance of T
map = obj; // Unable to convert T to base class
}
}
And then a method is defined to sort map of type KeyCountMap<T> by value in the descending order . The method is defined as:
public static KeyCountMap<T> SortMapByDescendValue<T>(KeyCountMap<T> _map)
{
List<KeyValuePair<T, MutableInt>> _list = new List<KeyValuePair<T, MutableInt>>(_map.EntrySet());
// whereas _map.EntrySet() return of type HashSet<KeyValuePair<T, MutableInt>>
_list = _list.OrderByDescending(_x => _x.Value).ToList();
KeyCountMap<T> _result = new KeyCountMap<T>();
foreach (KeyValuePair<T, MutableInt> _entry in _list)
{
_result.Put(_entry.Key, _entry.Value);
}
return _result;
}
How can I get corrected the class defined in .NET ?

I assume you know Java erases any generic type information after compiling (there's metadata for variables, but actual objects are void of generic type information). Moreover, your code is not type safe:
#SuppressWarnings({ "unchecked", "rawtypes" })
You're using this because you're creating a non-parameterized instance of Map.
In .NET, you don't get around the type system like this, because generic type information is kept and used at runtime.
Let's see your C# code:
public static class KeyCountMap<T>
A static class in C# is a class that cannot be instanced, it's used for its static members alone. I think you don't want this. Perhaps KeyCountMap is a static nested class in Java, as opposed to an inner class.
In C#, you don't have inner classes. Nested classes don't share data with an instance of the containing class, it's as if the name of the containing class is part of the namespace for the nested class. So, you don't need, and actually don't want, the static keyword here.
{
private Dictionary<T, MutableInt> map = new Dictionary<T, MutableInt>();
In .NET, Dictionary is a class. To keep the intent, you should use IDictionary, the corresponding interface, as the type for the map field.
// ... rest of properties...
public KeyCountMap()
{ }
public void KeyCountMap<T>(T obj) where T : Dictionary<T, MutableInt>
Why the void return type, isn't this a constructor?
In C#, constructors can't be generic. You probably want a Type.
Your C# code just doesn't make sense, so here's what you could do:
public KeyCountMap(Type dictionaryType)
{
if (!typeof(IDictionary<T, MutableInt>).IsAssignableFrom(dictionaryType))
{
throw new ArgumentException("Type must be a IDictionary<T, MutableInt>", nameof(dictionaryType));
}
map = (IDictionary<T, MutableInt>)Activator.CreateInstance(dictionaryType);
}
}
We're checking the type before creating an instance. If we didn't, we would create an instance, the cast would fail and the assignment wouldn't even happen, so the new instance would just be garbage.
It may be that the actual instance will be a proxy; if so, you may not want to check the type before creating an instance.
You can't just copy-paste Java as C# (or vice-versa) and expect to make just a few changes until it works, for some definition of works, e.g. it compiles. The languages are not that similar, and chances are that too many subtle things are wrong.
This approach might be fun at first, but you'll stumble so often it will soon stop being any fun at all. You should learn the basics and understand the way things are done in the target language before you start translating code line-by-line. Many times, you may find that something you had to do in one environment already exists in the other or vice-versa, or that something may take more or less steps to do in the other, etc.
In this particular case, Java made Class be a generic class, while .NET kept Type a non-generic class. In .NET only interfaces and delegates may state generic type covariance or contravariance. This is rather restrictive anyway, if Type was generic, the intended uses could be either covariant or contravariant. But remember that in Java, a generic Class<T> at runtime is as good as Class, it only has any value at compile time and you can tell the compiler you know better anyway, just like you did.

There are two problems. First, you need to tell the compiler that T has a parameterless constructor, so you can call new T(). You can do that by providing the new() argument to the class definition.
You also have to tell the compiler that T is actually the dictionary you are trying to assign, so we have to extend the class a little more:
public class KeyCountMap<K>
{
private Dictionary<K, MutableInt> map = new Dictionary<K, MutableInt>();
// ... rest of properties...
Note that K is the key type of the dictionary, which you didn't specify yet.
Second, the T in your method can be another T than in your class. Omitting that will do the trick:
public void Map()
{
var obj = new Dictionary<K, MutableInt>(); // Unable to define new instance of T
map = obj; // Unable to convert T to base class
}

Maybe this is what you want?
public class KeyCountMap<T>
where T : new()
{
private Dictionary<T, MutableInt> map = new Dictionary<T, MutableInt>();
// ... rest of properties...
public KeyCountMap()
{ }
public KeyCountMap(T obj)
{
obj = new T();
map = (Dictionary<T, MutableInt>)(object)obj;
}
}

Serialization Of Polymorphic Array Failing

I have enabled Polymorphic serialization support by adding annotations on the base class. I am able to seriazlize an individual object successfully and it is writing the type information as part of serialized data. However, the same is not happening if I store the objects in a list and serialize it.
It seems this issue was fixed in 1.6.3 (http://jira.codehaus.org/browse/JACKSON-362)
I am using Jackson 2.3.2 and still facing the issue.
Does somebody know how to fix this?
Code:
#JsonTypeInfo(use = JsonTypeInfo.Id.NAME, include = JsonTypeInfo.As.PROPERTY,property = "type")
#JsonSubTypes({#Type(value = Derived.class, name = "derived")})
public abstract class Base {
}
public class Derived extends Base {
private String field;
public String getField() {
return field;
}
public void setField(String field) {
this.field = field;
}
}
public class Test {
public static void main(String[] args) throws JsonProcessingException {
ObjectMapper mapper = new ObjectMapper();
Derived d = new Derived();
d.setField("Name");
Base b = d;
System.out.println(mapper.writeValueAsString(b));
List<Base> list = new ArrayList<Base>();
list.add(d);
System.out.println(mapper.writeValueAsString(list));
}
}
Output:
{"type":"derived","field":"Name"}
[{"field":"Name"}]
Thanks,
Praveen

The answer is at https://github.com/FasterXML/jackson-databind/issues/699
This is due to Java type erasure: when serializing a List, all Jackson see as a type is List (roughly equivalent to List). And since type Object does not have polymorphic type information (annotation), none will be written.
So this is not a bug in Jackson, but an unfortunate feature of Java Type Erasure.
It does not apply to arrays, since they retain element type information (arrays are not generic; arrays of different types are different classes, whereas generic typing is mostly compile-time syntactic sugar).
There are three main ways to deal with this:
pass full generic type using TypeReference (ObjectMapper has method like mapper.writerFor(new TypeReference<List<Base>>() { }).writeValue(....)
Sub-class List to something like public class BaseList extends ArrayList<Base>() { }, and pass that: this type WILL retain type information
Avoid using root-level List and Maps
I personally recommend doing (3), since this avoids all related problems with type erasure.
In my opinion JSON root value should always be a JSON Object, usually serialized to/from POJO.
Approach (2) will however work, and this is what most users do. It does require use of an additional helper class.
Approach (1) may or may not work; problem being that forcing type information does also affect actual value serialization. So while it will add type id, it may result in some properties not being serialized.

This problem can be solved by using arrays, instead of list (since list does type erasure):
for example, your above test-case could be written as:
public class Test {
public static void main(String[] args) throws JsonProcessingException {
ObjectMapper mapper = new ObjectMapper();
Derived d = new Derived();
d.setField("Name");
Base b = d;
System.out.println(mapper.writeValueAsString(b));
List<Base> list = new ArrayList<Base>();
list.add(d);
System.out.println(mapper.writeValueAsString(
list.toArray(new Base[list.size]) // <--This Part
));
}
}

I had the same issue with object array. Object[] doesn't carry type information but individual objects do. It's a shame that jackson doesn't handle that automatically.
Two possible solutions:
1. Typed array serialization works just fine:
Base[] myArray = Base[]{d};
mapper.writeValueAsString(myArray)
this will actually produce expected result as Base[] has type information.
I solved that my issue with custom serializer.
Serializer:
public class ObjectArraySerializer extends StdSerializer<Object[]> {
public ObjectArraySerializer(final Class<Object[]> vc) {
super(vc);
}
#Override
public void serialize(
final Object[] data,
final JsonGenerator gen,
final SerializerProvider provider) throws IOException {
gen.writeStartArray();
for (Object obj : data) {
gen.writeObject(obj);
}
gen.writeEndArray();
}
}
ObjectMapper configuration:
ObjectMapper objectMapper = new ObjectMapper();
SimpleModule module = new SimpleModule();
module.addSerializer(
Object[].class,
new ObjectArraySerializer(Object[].class));
objectMapper.registerModule(module);

How to get the Generic type for the Jackson ObjectMapper

Java erases normally the Generics data on compilation, but there is a possibility to get that information (the Jackson ObjectMapper does that pretty well).
My Problem: I have a Class with a property of List:
public class User {
public List<Long> listProp;//it is public only to keep the example simple
}
How can I get the correct TypeReference (or JavaType ?) so that I can map programatically a JSON String to the correct List Type, having the instance of the Class class (User.class) and the property name (listProp)? What I mean is this:
TypeReference typeReference = ...;//how to get the typeReference?
List<Long> correctList = om.readValue(jsonEntry.getValue(), typeReference);//this should return a List<Long> and not eg. a List<Integer>

Have you tried the mappers constructType method?
Type genericType = User.class.getField("listProp").getGenericType();
List<Long> correctList = om.readValue(jsonEntry.getValue(), om.constructType(genericType));

jackson use TypeReference to construct generic type
TypeReference typeReference =new TypeReference<List<Long>>(){}
jackson use JavaType to construct generic type
JavaType jt = om.getTypeFactory().constructArrayType(Long.class);
jackson support three types
Class
JavaType
TypeReference
i like use JavaType, it is more clear for generic type, for normal object use Class

Perhaps a less exotic approach to deserializing a generic type is to wrap it inside a concrete type:
class ListLongWrapper extends ArrayList<Long> {} // package scope
... or ...
static class ListLongWrapper extends ArrayList<Long> {} // class scope
then
String jsonStr = objMapper.writeValueAsString(user1.listProp); // serialize
user2.listProp = objMapper.readValue(jsonStr,ListLongWrapper.class); // deserialize
Notice that extends requires a class type (here I used ArrayList) instead of the interface List.
This suggests an even more direct approach for the given example -- User is already a wrapper (and listProp is public):
public class User {
public List<Long> listProp;
}
then
String jsonStr = objMapper.writeValueAsString(user1); // serialize
var user2 = objMapper.readValue(jsonStr,User.class); // deserialize
In this case, you can use the interface List as-is as a field type within the wrapping class, but this means you have no control over the concrete type that Jackson will use.

Avoiding Returning Wildcard Types

I have a class with a collection of Wildcard Types that is a singleton, something like:
public ObliviousClass{
private static final ObliviousClass INSTANCE = new ObliviousClass();
private Map<Key, Type<?>> map = new HashMap<Key, Type<?>>();
public void putType(Key key, Type<?> type){
map.put(type);
}
// returns the singleton
public static ObliviousClass getInstance(){
return INSTANCE;
}
}
I'd like to be able to add different Parameterized types to this collection in client code:
void clientMethod(){
ObliviousClass oc = ObliviousClass.getInstance();
Type<Integer> intType = ...
Type<String> stringType = ...
oc.putType(new Key(0), intType);
oc.putType(new Key(1), stringType);
}
Up to this point, as I understand it, everything is ok. But a client also needs to be able to get a Type<?> provided the Key. So a method something like the following would be added to ObliviousClass:
public Type<?> getType(Key key){
return map.get(key);
}
But in my handy copy of Effective Java, I read:
Do not use wildcard types as return types.
I understand the issue, as the client would have to cast the returned Type<?>. But I really do not want to make ObliviousClass a generic type, ObliviousClass<T>, because then my client code above would not work...
Is there a better design for what I am trying to do?
-My current solution is to provide a static method for the client; something along the lines of:
public static <T> void getType(ObliviousClass instance, Key key, Type<T> dest){
dest = (Type<T>)instance.getType(key);
}
I searched around, but wasn't able to find an answer that totally cleared my confusion.

Here's a type-safe way to store multiple instances of a given type in a map. The key is that you need to provide a Class instance when retrieving values in order to perform runtime type-checking, because static type information has been erased.
class ObliviousClass {
private final Map<Key, Object> map = new HashMap<Key, Object>();
public Object put(Key key, Object value)
{
return map.put(key, value);
}
public <T> T get(Key key, Class<? extends T> type)
{
return type.cast(map.get(key));
}
}
Usage would look like this:
oc.put(k1, 42);
oc.put(k2, "Hello!");
...
Integer i = oc.get(k1, Integer.class);
String s = oc.get(k2, String.class);
Integer x = oc.get(k2, Integer.class); /* Throws ClassCastException */

Simply type your class:
public ObliviousClass <T> {
private Map<Key, Type<T>> map = new HashMap<Key, Type<T>>();
public void putType(Key key, Type<T> type){
map.put(type);
}
public Type<T> getType(Key key){
map.get(key);
}
}
FYI, at this point you have the delegation pattern in play.
Your example client code would need to declare two instances of ObliviousClass: ObliviousClass<String> and ObliviousClass<Integer>.
Edit:
If you must have a mixed bag of Types, you can impose a type on your method, but you'll get a compiler warning for an unsafe cast:
public class ObliviousClass {
private final Map<Key, Type<?>> map = new HashMap<Key, Type<?>>();
public void putType(Key key, Type<?> value) {
map.put(key, value);
}
#SuppressWarnings("unchecked")
public <T> Type<T> getType1(Key key, Class<T> typeClass) {
return (Type<T>)map.get(key);
}
#SuppressWarnings("unchecked")
public <T> Type<T> getType2(Key key) {
return (Type<T>) map.get(key);
}
}
Clients can type the calls to these methods like this:
Type<Integer> x = obliviousClass.getType1(key, Integer.class);
Type<Integer> y = obliviousClass.<Integer>getType2(key);
Take your pick as to which one you prefer and use that.

For those landing on this question these many years later, this is not how Java generics are designed to be used. (I was going to comment but had more to details.)
The generic pattern manages a single parent class per type ID rather than multiple different classes. If we consider the simpler List<T>, a list of strings OR integers (as List<String> or List<Integer>) is how generics are defined. One class per type. This way, there is a consistent type when the values are referenced. Storing unrelated types would be the same as List<Object>. Only the programmer can know when multiple types are stored and how to retrieve them with casting.
It would be ok to store subclasses to a parent class, but when accessed from the collection without casting, the parent class contact is all that is known. For instance, a generic collection defined with an interface like Map<String, Runnable>. However, only the run() method is visible even if other public methods are added to implementations (unless the programmer explicitly casts). To access additional methods, casting is necessary.
This is a limitation in Java. A language could be defined to know the L-Value type - even Java. But it wasn't. When new features are added, there are many backward compatible considerations [Sun and] Oracle take into account. Code compiled with generics was designed to run on older JVMs with type erasure. Java uses type erasure at compile time once it has determined that the generics are consistently reference. The bytecode uses Object as if the instance was (sort of) defined as List. If the choice was made to abandon backward compatibility, like Java 9 and 11, then multiple types might have been workable.

Your ObliviousClass, by design, doesn't know the parameterized type of the item it holds. So to be type safe, you should avoid such design :-\
But if you want to keep it, first things is that you will have to cast. There is no way out of this. But the way you do it is very error prone. For example:
oc.put(k1, intType);
oc.put(k2, strType);
Type<Integer> tint = oc.get(k1, Integer.class)
Type<String> tstr = oc.get(k1, String.class) // typo in k2: compile fine
And worst, due to type erasure, it will fail at runtime only once you actually use tstr, not when you get it from ObliviousClass.
So you can improve safety by tracking the parameterized type in some other way. For example, you could associate the key to the type, not losing it:
#Value // lombok
class Key<T> {
private int index;
}
class Type<T> {}
class ObliviousClass {
// side note: static final can be public safely
public static final ObliviousClass instance = new ObliviousClass();
private List<Type<?>> map = new ArrayList<>();
public <T> Key<T> appendType(Type<T> type){
// here, I found it nicer that obliviousClass generates and return the key
// otherwise use: "public <T> void appendType(key<T> key, Type<T> type)"
// that binds parametrized type of both key and type arguments
map.add(type);
return new Key<>(map.size() - 1);
}
public <T> Type<T> get(Key<T> key){
return (Type<T>) map.get(key.index);
}
}
Then you can use it such as:
Type<Integer> intType = new Type<>();
Type<String> strType = new Type<>();
Key<Integer> k1 = ObliviousClass.instance.appendType(intType);
Key<String> k2 = ObliviousClass.instance.appendType(strType);
Type<Integer> t1 = ObliviousClass.instance.get(k1);
Type<String> t2 = ObliviousClass.instance.get(k2);
Type<String> t3 = ObliviousClass.instance.get(k1); // won't compile