We Keep Coding

How exactly type inference is working in java?

I am trying to create a method like following:
public <T> T getInstance(String key) {
Type type = new TypeToken<T>(){}.getType();
return deserialize(key, type); }
As far as I am not casting (T) in return statement, I expect compiler to infer type from the outside context or at least from a type witness like here:
Integer i = container.<Integer>getInstance(mKey);
But for some case, getInstance() method returns Double that was never mentioned (to be more precise, the serialization and deserialization are using google's Gson library, and the initial object was an instance of Integer). Hence, I get ClassCastException.
So how exactly does type inferring work in this case and why does the type witness not work? Is it possible to infer type from the outside context without specifying Class<T>.class as an argument?
Elaboration:
After some research, this FAQ helped me to understand the topic much better.

The general problem that you have here is that there is nothing to constrain the output to be related to the inputs.
So, this is valid:
Integer i = container.<Integer>getInstance(mKey); // I doubt you even need the <Integer>
but this is also valid:
String s = container.<String>getInstance(mKey);
All the getInstance method sees is the value of mKey: it doesn't know anything about the <Integer> or <String>; so this has to return the same result in both cases - but at least one of those is wrong (or they are both returning null).
Because getInstance only receives the value of mKey, the type token is the same in both cases. You can see this by implementing your own type token class:
abstract static class TypeToken<T> {
Type getType() {
return getClass().getGenericSuperclass();
}
}
static <T> TypeToken<T> getTypeToken() {
return new TypeToken<T>() {};
}
public static void main (String[] args) throws java.lang.Exception
{
TypeToken<String> stringTypeToken = getTypeToken();
TypeToken<Integer> integerTypeToken = getTypeToken();
System.out.println(stringTypeToken.getType());
System.out.println(integerTypeToken.getType());
System.out.println(stringTypeToken.getType().equals(integerTypeToken.getType()));
}
Output:
Ideone$TypeToken<T>
Ideone$TypeToken<T>
true
In your code, Type type = new TypeToken<T>(){}.getType(); isn't getting the TypeToken for the call site type, it's just getting T. So you're not getting a TypeToken<Integer>.
This works, in the sense that it compiles, but it doesn't work in the sense that it doesn't do what you're trying to get it to do.
The way to do this correctly is to inject the TypeToken<T> as a method parameter. It becomes an awful lot more cumbersome at call sites; but that's the price you pay for using a language with erased types.

How should Type type-hierarchy types be implemented?

When generics were added to 1.5, java.lang.reflect added a Type interface with various subtypes to represent types. Class is retrofitted to implement Type for the pre-1.5 types. Type subtypes are available for the new types of generic type from 1.5.
This is all well and good. A bit awkward as Type has to be downcast to do anything useful, but doable with trial, error, fiddling and (automatic) testing. Except when it comes to implementation...
How should equals and hashCode be implemented. The API description for the ParameterizedType subtype of Type says:
Instances of classes that implement this interface must implement an equals() method that equates any two instances that share the same generic type declaration and have equal type parameters.
(I guess that means getActualTypeArguments and getRawType but not getOwnerType??)
We know from the general contract of java.lang.Object that hashCode must also be implemented, but there appears to be no specification as what values this method should produce.
None of the other subtype of Type appear to mention equals or hashCode, other than that Class has distinct instances per value.
So what do I put in my equals and hashCode?
(In case you are wondering, I am attempting to substitute type parameters for actual types. So if I know at runtime TypeVariable<?> T is Class<?> String then I want to replace Types, so List<T> becomes List<String>, T[] becomes String[], List<T>[] (can happen!) becomes List<String>[], etc.)
Or do I have to create my own parallel type type hierarchy (without duplicating Type for presumed legal reasons)? (Is there a library?)
Edit: There's been a couple of queries as to why I need this. Indeed, why look at generic type information at all?
I'm starting with a non-generic class/interface type. (If you want a parameterised types, such as List<String> then you can always add a layer of indirection with a new class.) I am then following fields or methods. Those may reference parameterised types. So long as they aren't using wildcards, I can still work out actual static types when faced with the likes of T.
In this way I can do everything with high quality, static typing. None of these instanceof dynamic type checks in sight.
The specific usage in my case is serialisation. But it could apply to any other reasonable use of reflection, such as testing.
Current state of code I am using for the substitution below. typeMap is a Map<String,Type>. Present as an "as is" snapshot. Not tidied up in anyway at all (throw null; if you don't believe me).
Type substitute(Type type) {
if (type instanceof TypeVariable<?>) {
Type actualType = typeMap.get(((TypeVariable<?>)type).getName());
if (actualType instanceof TypeVariable<?>) { throw null; }
if (actualType == null) {
throw new IllegalArgumentException("Type variable not found");
} else if (actualType instanceof TypeVariable<?>) {
throw new IllegalArgumentException("TypeVariable shouldn't substitute for a TypeVariable");
} else {
return actualType;
}
} else if (type instanceof ParameterizedType) {
ParameterizedType parameterizedType = (ParameterizedType)type;
Type[] actualTypeArguments = parameterizedType.getActualTypeArguments();
int len = actualTypeArguments.length;
Type[] actualActualTypeArguments = new Type[len];
for (int i=0; i<len; ++i) {
actualActualTypeArguments[i] = substitute(actualTypeArguments[i]);
}
// This will always be a Class, wont it? No higher-kinded types here, thank you very much.
Type actualRawType = substitute(parameterizedType.getRawType());
Type actualOwnerType = substitute(parameterizedType.getOwnerType());
return new ParameterizedType() {
public Type[] getActualTypeArguments() {
return actualActualTypeArguments.clone();
}
public Type getRawType() {
return actualRawType;
}
public Type getOwnerType() {
return actualOwnerType;
}
// Interface description requires equals method.
#Override public boolean equals(Object obj) {
if (!(obj instanceof ParameterizedType)) {
return false;
}
ParameterizedType other = (ParameterizedType)obj;
return
Arrays.equals(this.getActualTypeArguments(), other.getActualTypeArguments()) &&
this.getOwnerType().equals(other.getOwnerType()) &&
this.getRawType().equals(other.getRawType());
}
};
} else if (type instanceof GenericArrayType) {
GenericArrayType genericArrayType = (GenericArrayType)type;
Type componentType = genericArrayType.getGenericComponentType();
Type actualComponentType = substitute(componentType);
if (actualComponentType instanceof TypeVariable<?>) { throw null; }
return new GenericArrayType() {
// !! getTypeName? toString? equals? hashCode?
public Type getGenericComponentType() {
return actualComponentType;
}
// Apparently don't have to provide an equals, but we do need to.
#Override public boolean equals(Object obj) {
if (!(obj instanceof GenericArrayType)) {
return false;
}
GenericArrayType other = (GenericArrayType)obj;
return
this.getGenericComponentType().equals(other.getGenericComponentType());
}
};
} else {
return type;
}
}

I've been solving this problem in unsatisfying ways for 10 years. First with Guice’s MoreTypes.java, copy-pasted and revised with Gson’s GsonTypes.java, and again in Moshi’s Util.java.
Moshi has my best approach, which isn't to say that it's good.
You can't call equals() on arbitrary implementations of Type and expect it to work.
This is because the Java Types APIs offers multiple incompatible ways to model arrays of simple classes. You can make a Date[] as a Class<Date[]> or as a GenericArrayType whose component type is Date. I believe you’ll get the former from reflection on a field of type Date[] and the latter from reflection as the parameter of a field of type List<Date[]>.
The hash codes aren't specified.
I also got to work on the implementation of these classes that Android uses. Very early versions of Android have different hash codes vs. Java, but everything you'll find in the wild today uses the same hash codes as Java.
The toString methods aren't good
If you're using types in error messages it sucks to have to write special code to print them nicely.
Copy Paste and Be Sad
My recommendation is to not use equals() + hashCode() with unknown Type implementations. Use a canonicalize function to convert into a specific known implementation and only compare within the ones you control.

Here is a little experiment that relies directly on the Sun API and reflection (that is, it uses reflection to work with classes that implement reflection):
import java.lang.Class;
import java.lang.reflect.*;
import java.util.Arrays;
import sun.reflect.generics.reflectiveObjects.*;
class Types {
private static Constructor<ParameterizedTypeImpl> PARAMETERIZED_TYPE_CONS =
((Constructor<ParameterizedTypeImpl>)
ParameterizedTypeImpl
.class
.getDeclaredConstructors()
[0]
);
static {
PARAMETERIZED_TYPE_CONS.setAccessible(true);
}
/**
* Helper method for invocation of the
*`ParameterizedTypeImpl` constructor.
*/
public static ParameterizedType parameterizedType(
Class<?> raw,
Type[] paramTypes,
Type owner
) {
try {
return PARAMETERIZED_TYPE_CONS.newInstance(raw, paramTypes, owner);
} catch (Exception e) {
throw new Error("TODO: better error handling", e);
}
}
// (similarly for `GenericArrayType`, `WildcardType` etc.)
/** Substitution of type variables. */
public static Type substituteTypeVariable(
final Type inType,
final TypeVariable<?> variable,
final Type replaceBy
) {
if (inType instanceof TypeVariable<?>) {
return replaceBy;
} else if (inType instanceof ParameterizedType) {
ParameterizedType pt = (ParameterizedType) inType;
return parameterizedType(
((Class<?>) pt.getRawType()),
Arrays.stream(pt.getActualTypeArguments())
.map((Type x) -> substituteTypeVariable(x, variable, replaceBy))
.toArray(Type[]::new),
pt.getOwnerType()
);
} else {
throw new Error("TODO: all other cases");
}
}
// example
public static void main(String[] args) throws InstantiationException {
// type in which we will replace a variable is `List<E>`
Type t =
java.util.LinkedList
.class
.getGenericInterfaces()
[0];
// this is the variable `E` (hopefully, stability not guaranteed)
TypeVariable<?> v =
((Class<?>)
((ParameterizedType) t)
.getRawType()
)
.getTypeParameters()
[0];
// This should become `List<String>`
Type s = substituteTypeVariable(t, v, String.class);
System.out.println("before: " + t);
System.out.println("after: " + s);
}
}
The result of substitution of E by String in List<E> looks as follows:
before: java.util.List<E>
after: java.util.List<java.lang.String>
The main idea is as follows:
Get the sun.reflect.generics.reflectiveObjects.XyzImpl classes
Get their constructors, ensure that they are accessible
Wrap the constructor .newInstance invocations in helper methods
Use the helper methods in a simple recursive method called substituteTypeVariable that rebuilds the Type-expressions with type variables substituted by concrete types.
I didn't implement every single case, but it should work with more complicated nested types too (because of the recursive invocation of substituteTypeVariable).
The compiler doesn't really like this approach, it generates warnings about the usage of the internal Sun API:
warning: ParameterizedTypeImpl is internal proprietary API and may be removed in a future release
but, there is a #SuppressWarnings for that.
The above Java code has been obtained by translating the following little Scala snippet (that's the reason why the Java code might look a bit strange and not entirely Java-idiomatic):
object Types {
import scala.language.existentials // suppress warnings
import java.lang.Class
import java.lang.reflect.{Array => _, _}
import sun.reflect.generics.reflectiveObjects._
private val ParameterizedTypeCons =
classOf[ParameterizedTypeImpl]
.getDeclaredConstructors
.head
.asInstanceOf[Constructor[ParameterizedTypeImpl]]
ParameterizedTypeCons.setAccessible(true)
/** Helper method for invocation of the `ParameterizedTypeImpl` constructor. */
def parameterizedType(raw: Class[_], paramTypes: Array[Type], owner: Type)
: ParameterizedType = {
ParameterizedTypeCons.newInstance(raw, paramTypes, owner)
}
// (similarly for `GenericArrayType`, `WildcardType` etc.)
/** Substitution of type variables. */
def substituteTypeVariable(
inType: Type,
variable: TypeVariable[_],
replaceBy: Type
): Type = {
inType match {
case v: TypeVariable[_] => replaceBy
case pt: ParameterizedType => parameterizedType(
pt.getRawType.asInstanceOf[Class[_]],
pt.getActualTypeArguments.map(substituteTypeVariable(_, variable, replaceBy)),
pt.getOwnerType
)
case sthElse => throw new NotImplementedError()
}
}
// example
def main(args: Array[String]): Unit = {
// type in which we will replace a variable is `List<E>`
val t =
classOf[java.util.LinkedList[_]]
.getGenericInterfaces
.head
// this is the variable `E` (hopefully, stability not guaranteed)
val v =
t
.asInstanceOf[ParameterizedType]
.getRawType
.asInstanceOf[Class[_]] // should be `List<E>` with parameter
.getTypeParameters
.head // should be `E`
// This should become `List<String>`
val s = substituteTypeVariable(t, v, classOf[String])
println("before: " + t)
println("after: " + s)
}
}

incompatible generic return type

I understand generics pretty well, but came across a seemingly simple problem that I don't quite understand. Take a look at the following example.
public <T extends Object> T getSetting(String settingName) {
return settings.getJsonObject(settingName).getValue("value");
}
settings is a instance attribute from type vertx.core.json.JsonObject. The result of getValue is Object.
I thought by setting the return type to T I would be able to return any type that is Object or an super type like String or Integer. This code however gives me the following error message.
incompatible types: java.lang.Object cannot be converted to T
What am I overlooking here?

That code does not compile because whatever T is, decided at compilation time, does not need to match what getValue will return which is determined at run-time
which is not restricted at all.
You can add a (T) cast and that would transform the error in a warning but that would not prevent the potential run-time cast error and that is why it stays as a warning.
I don't think there is any real workaround for this as you can never be certain of the type of the getValue return ... however you can provide some methods that at least would:
a. fail more gracefully (with a nicer error message),
b. return a default value if the type is not the one expected,
c. give the user the chance to provide the casting lambda to transform
any possible input type into the expected return value class.
E.g.
public <T> T getSetting(String settingName, Class<T> clazz) {
final Object obj = settings.getJsonObject(settingName).getValue("value");
if (obj == null)
return null;
else if (!clazz.isInstance(obj))
throw new IllegalArgumentException(String.format("wrong setting class; was expecting %s but found %s", clazz.getSimpleName(), obj.getClass().getName());
else {
return clazz.cast(obj);
}
}
...
Integer x = getSetting("mysetting", Integer.class);
Or:
public <T> T getSetting(String settingName, Function<Object, T> cast) {
final Object obj = settings.getJsonObject(settingName).getValue("value");
return obj == null ? null : cast.apply(obj);
}
...
Integer x = getSetting("mysetting", o -> Integer.parseInt(o.toString()));

Java generics cast based on return type?

The following code is from an Android library called ButterKnife. I'm figuring out how it works.
#SuppressWarnings("unchecked") // That's the point.
public <T> T castParam(Object value, String from, int fromPosition, String to, int toPosition) {
try {
return (T) value;
} catch (ClassCastException e) {
throw new IllegalStateException("Parameter #"
+ (fromPosition + 1)
+ " of method '"
+ from
+ "' was of the wrong type for parameter #"
+ (toPosition + 1)
+ " of method '"
+ to
+ "'. See cause for more info.", e);
}
}
I tried to recreate the behaviour of this function:
#SuppressWarnings("unchecked")
public static <T> T cast(Object o){
try {
return (T) o;
} catch (ClassCastException e){
throw new AssertionError("Error");
}
}
And usage:
Object o = new String("test");
Double d = cast(o);
But the exception is not never caught, it gets thrown at the line when the method is called. Why is that?
Also, how does this work exactly? How does the method know what to cast to?

Generics types are checked at compile time only, due to type erasure. This is done because there was no way to introduce generics in the runtime in Java 5 without breaking backwards compatibility and forcing to recompile all the already existing libraries.
Long history short, when you define a "generic" class or method, the actual code is compiled as Object instead of the type you are binding the method. All the checks of types are done at compile time.
So, your method code is not actually doing a cast in the return statement, since it is assigning something (a String) to an Object return value. The actual ClassCastException is returned by the calling line because it is the place when the reference variable is actually typed.

As SJuan67 explained, you cannot really use casts with generic types as Java compiler will
Replace all type parameters in generic types with their bounds or Object if the type parameters are unbounded. The produced bytecode, therefore, contains only ordinary classes, interfaces, and methods.
More info on all generics restrictions here.
So ButterKnife code will look like this:
public Object castParam(Object paramObject, String paramString1, int paramInt1, String paramString2, int paramInt2)
{
return paramObject;
}
So to your questions:
Q: But the exception is not never caught, it gets thrown at the line when the method is called. Why is that?
A: Well its not even in the bytecode.
Q: Also, how does this work exactly? How does the method know what to cast to?
A: It doesn't. At least not like you think it will. In practice it will throw ClassCastException not IllegalStateException or AssertionError as you observed.
You can even try it with ButterKnife sample app and Bind a known TextView to CheckBox:
#Bind(R.id.title) CheckBox title;
Q: How does the library work then?
A: Well IllegalStateException is just never called and you have ClassCastException. Why it is like that I an not really sure. However as ButterKnife generates code this could be intended to prevent from compile errors.
for example:
public interface Some {
}
public static void weWantSome(Some d) {
}
public static void test() {
String o = "test";
weWantSome((Some)o); //<-- compile error
weWantSome(Main.<Some>cast(o)); //<-- runtime error
}
Which is why in the previous example code compiles but does not run.

Why does the JVM allow be to pass a B[] to a method that expects an A[]?

I have the following generic test class:
public class BrokenGenerics<T> {
private T[] genericTypeArray;
public BrokenGenerics(T... initArray) {
genericTypeArray = initArray;
}
public void setArray(T[] newArray) {
genericTypeArray = newArray;
}
public T get(int idx) {
return genericTypeArray[idx];
}
public Class getType() {
return genericTypeArray.getClass().getComponentType();
}
public static boolean breakThis(BrokenGenerics any) {
any.setArray(new B[]{new B(2)});
return false;
}
public static void main(String[] args) {
BrokenGenerics<A> aBreaker = new BrokenGenerics<A>(new A("1"));
System.out.println(aBreaker.get(0));
System.out.println(aBreaker.getType());
breakThis(aBreaker);
System.out.println(aBreaker.get(0));
System.out.println(aBreaker.getType());
}
private static class A {
public String val;
public A(String init) {
val = init;
}
#Override
public String toString() {
return "A value: " + val;
}
}
private static class B {
public int val;
public B(int init) {
val = init;
}
#Override
public String toString() {
return "B value: " + val;
}
}
}
When I run it, I get this output, and no errors:
A value: 1
class BrokenGenerics$A
B value: 2
class BrokenGenerics$B
Now, I understand why this compiles; it can't know at compile-time that breakThis is being passed a generic of a bad type. However, once it runs the line any.setArray(new B[]{new B(2)});, shouldn't it throw a ClassCastException (NOTE THAT IT DOES NOT! Try it yourself!) because I'm trying to pass a B[] to a method that expects an A[]? And after that, why does it allow me to get() back the B?

After Type Erasure, T will be turned into Object since you didn't specify a bound on T. So, there is no problem at runtime assigning any type of array to genericTypeArray, which is now of type Object[] or calling the function setArray(...), which now also accepts an argument of type Object[]. Also, your get(...) method will simply return an Object.
Trouble starts when you access elements in the array with a wrong type expectation, since this might lead to (implicit or explicit) illegal type casts, for example by assigning the value returned by get(...) to a variable of type A.
You can also get a run-time ClassCastException if you try to type-cast the array itself, but, in my experience, that is a case that tends to come up less often, although it can be very obscure to find or even understand if it does happen. You can find some examples below.
All generics-checking happens only at compile-time. And if you use raw types, these checks can not be performed rigorously, and thus the best the compiler can do is to issue a warning to let you know that you are giving up an opportunity for more meaningful checks by omitting the type argument.
Eclipse with its standard settings (and probably the java compiler with the correct flags) shows these warnings for your code:
"Class is a raw type" where you define getType() (somewhat unrelated to your question)
"BrokenGenerics is a raw type" where you define breakThis(...)
"Type safety: The method setArray(Object[]) belongs to the raw type
BrokenGenerics" where you call setArray(...) inside breakThis(...).
Examples for causing ClassCastException due to illegal type-cast of the array:
You can get ClassCastExceptions at runtime if you expose the array to the outside world (which can often be a dangerous thing to do, so I try to avoid it) by adding the following to BrokenGenerics<T>:
public T[] getArray() {
return genericTypeArray;
}
If you then change your main method to:
BrokenGenerics<A> aBreaker = new BrokenGenerics<A>(new A("1"));
A[] array = aBreaker.getArray();
System.out.println(array[0]);
System.out.println(aBreaker.getType());
breakThis(aBreaker);
array = aBreaker.getArray(); // ClassCastException here!
System.out.println(array[0]);
System.out.println(aBreaker.getType());
You get the ClassCastException at runtime at the indicated position due to a cast of the array itself rather than one of its elements.
The same thing can also happen if you set the variable genericTypeArray to protected and use it from code that subclasses your generic class with a fixed type argument:
private static class C extends BrokenGenerics<A> {
public C(A... initArray) {
super(initArray);
}
public void printFirst() {
A[] result = genericTypeArray; // ClassCastException here!
System.out.println(result[0]);
}
}
To trigger the exception, add the following to you main method:
C cBreaker = new C(new A("1"));
cBreaker.printFirst();
breakThis(cBreaker);
cBreaker.printFirst();
Imagine this case coming up in a bigger project... How on earth would you even begin to understand how that line of code could possible fail?!? :) Especially since the stack trace might be of very little help trying to find the breakThis(...) call that is actually responsible for the error.
For more in-depth example cases, you can take a look at some tests I did a little while back.

shouldn't it throw a ClassCastException because I'm trying to pass a B[] to a method that expects an A[]?
No. As this post explains, your invocation of setArray in
public static boolean breakThis(BrokenGenerics any) {
any.setArray(new B[]{new B(2)});
return false;
}
is done on a reference expression of the raw type BrokenGenerics. When interacting with raw types, all corresponding generic parameters are erased. So setArray is actually expecting a Object[]. A B[] is a Object[].
why does it allow me to get() back the B?
Assuming you're asking about this
System.out.println(aBreaker.get(0));
PrintStream#println(Object) expects an Object, not an A. As such, there is no reason for the compiler to insert a cast here. Since there is no cast, there is no ClassCastException.
If you had instead done
A a = aBreaker.get(0);
or had a method like
void println(A a) {}
...
println(aBreaker.get(0));
then these would cause ClassCastException. In other words, the compiler will insert a cast (checkcast) anywhere a type needs to be converted from a generic type parameter. That was not the case with PrintStream#println.
Similarly,
System.out.println(aBreaker.getType());
doesn't even involve the generic parameter declared in BrokenGenerics
public Class getType() {...}
and also returns a value of the raw type Class. The compiler has no reason to add a checkcast to A.