Java Hadoop problems with ChainMapper Generics - java
Hi I am currently facing an issue with Java Generics:
The method I need to use has this signature
static<K1,V1,K2,V2> void addMapper(JobConf job,
Class<? extends Mapper<K1,V1,K2,V2>> klass,
Class<? extends K1> inputKeyClass,
Class<? extends V1> inputValueClass,
Class<? extends K2> outputKeyClass,
Class<? extends V2> outputValueClass,
boolean byValue, JobConf mapperConf)
and this is how I call it
ChainMapper.addMapper(conf, NameMapper.class,
Object.class, Object.class, Object.class, Object.class,
false, nameYearConf);
where NameMapper is defined as follows
public class NameMapper extends Mapper<Object, Object, Object, Object> { }
and also the other two parameters are correctly intialized
JobConf conf = new JobConf(); JobConf nameYearConf = new JobConf(false);
when I try to compile I obtain the following error
src\MeanYear.java:107: error: method addMapper in class ChainMapper cannot be applied to given types;
ChainMapper.addMapper(conf, NameMapper.class, Object.class,
^
required: JobConf,Class<? extends Mapper<K1,V1,K2,V2>>,Class<? extends K1>,Class<? extends V1>,Class<? extends K2>,Class<? extends V2>,boolean,JobConf
found: JobConf,Class<MeanYear.NameMapper>,Class<Object>,Class<Object>,Class<Object>,Class<Object>,boolean,JobConf
reason: no instance(s) of type variable(s) K1,V1,K2,V2 exist so that argument type Class<MeanYear.NameMapper> conforms to formal parameter type Class<? extends Mapper<K1,V1,K2,V2>>
where K1,V1,K2,V2 are type-variables:
K1 extends Object declared in method <K1,V1,K2,V2>addMapper(JobConf,Class<? extends Mapper<K1,V1,K2,V2>>,Class<? extends K1>,Class<? extends V1>,Class<? extends K2>,Class<? extends V2>,boolean,JobConf)
V1 extends Object declared in method <K1,V1,K2,V2>addMapper(JobConf,Class<? extends Mapper<K1,V1,K2,V2>>,Class<? extends K1>,Class<? extends V1>,Class<? extends K2>,Class<? extends V2>,boolean,JobConf)
K2 extends Object declared in method <K1,V1,K2,V2>addMapper(JobConf,Class<? extends Mapper<K1,V1,K2,V2>>,Class<? extends K1>,Class<? extends V1>,Class<? extends K2>,Class<? extends V2>,boolean,JobConf)
V2 extends Object declared in method <K1,V1,K2,V2>addMapper(JobConf,Class<? extends Mapper<K1,V1,K2,V2>>,Class<? extends K1>,Class<? extends V1>,Class<? extends K2>,Class<? extends V2>,boolean,JobConf)
I really cannot understand what I am doing wrong, I tried some similar examples and the compiles and they work. I have also tried some examples provided here, but without success.
Thanks.
At a glance my guess is you're mixing classes from the new and old api packages:
ChainMapper (as of 1.1.2) is only implemented for the old API package (org.apache.hadoop.mapred), and hasn't been ported to the new API package (org.apache.hadoop.mapreduce).
As such the second argument toChainMapper.addMapper (Class<? extends Mapper<K1,V1,K2,V2>> klass) is expected to be of type org.apache.hadoop.mapred.Mapper.
As your current mapper NameMapper extends the Mapper class, this leads me to believe that this is the new api Mapper class (org.apache.hadoop.mapreduce), as the new API is a class, not an interface
You should be able to fix by amending the signature of your NameMapper class to match that of the old API Mapper interface:
public class NameMapper extends MapReduceBase
implements Mapper<Object, Object, Object, Object> { }
This will also mean your current map method's signature will need to change from
protected void map(Object key, Object value, Context context) { }
// to
public void map(Object key, Object value, OutputCollector collector,
Reporter reporter) { }
And if you're using the setup and cleanup methods of the new Mapper API, you'll need to replace them too:
public void close() { } // instead of cleanup(Context) {}
public void configure(JobConf conf) { } // instead of setup(Context) {}
Related
Java weird generic's behavior
I found some weird behavior for java generics. I can't use parametrized class, which implements another parametrized class, in parametrized method. It sounds messy, so let's look at the code: class A { public<RESP extends Serializable, REQ extends Serializable> RESP sendData(REQ req, Class<? extends I<RESP, REQ>> c) { return null; } } interface I<RESP extends Serializable, REQ extends Serializable> { } class B implements I<String, Integer> { A a; public void test() { a.sendData(1, getClass()); } } It compiles fine. But we can break it easily. Just add a parameter variable to class B: class B<T> implements I<String, Integer> Now a.sendData(1, B.class); has the compiler error: method sendData in class A cannot be applied to given types; required: REQ,Class<? extends I<RESP,REQ>> found: int,Class<B> reason: cannot infer type-variable(s) RESP,REQ (argument mismatch; Class<B> cannot be converted to Class<? extends I<RESP,REQ>>) where REQ,RESP are type-variables: REQ extends Serializable declared in method <RESP,REQ>sendData(REQ,Class<? extends I<RESP,REQ>>) RESP extends Serializable declared in method <RESP,REQ>sendData(REQ,Class<? extends I<RESP,REQ>>) I just added unused parameter to class signature! Can anyone explain why this happens and is there way to use type variable in class B? Actualy my goal is signature for class B like this: class B<Data> implements I<String, ? extends List<Data>> By the way, I try this code in Java 8.
Java generics - Class<? extends Base>
I'm confused a bit about Java generics. I have a class Test which extends from Base: public class Test extends Base { private final static Test obj = new Test(); new Auto(obj); } The constructor of Auto is like this (not my class): public Auto (Class<? extends Base> programclass) {} And I'm getting this error: Test cannot be converted to Class<? extends Base> How should I declare obj variable so I can pass it to Auto constructor? Thanks
Issue with interface as Map key & value
In a library project, I have : public interface InterfaceA { } public interface InterfaceB { } public void myMethod(Map<? extends InterfaceA, List<? extends InterfaceB>> map) { //do something } Then I have another project (having this library as a dependency) that contains two object implementing these interfaces : public class ObjectA implements InterfaceA { } public class ObjectB implements InterfaceB { } When I try to call the library method myMethod like this : HashMap<ObjectA, List<ObjectB>> hashMap = new HashMap<>(); //populate hashmap myMethod(hashMap); I get a compilation warning saying there is an argument mismatch. What am I missing here ? Does it have something to do with the map ? EDIT : The exact error (it's not a warning actually) is : incompatible types: HashMap<ObjectA,List<ObjectB>> cannot be converted to Map<? extends InterfaceA,List<? extends InterfaceB>>
Generics are invariant. If your method declares: Map<? extends InterfaceA, List<? extends InterfaceB>> Then the second type parameter has to be exactly List<? extends InterfaceB>. You can fix it by using: Map<? extends InterfaceA, ? extends List<? extends InterfaceB>> Instead.
You either modify your Hashmap creation for this: Map<? extends InterfaceA, List<? extends InterfaceB>> hashMap = new HashMap<>(); or modify your method definition for this: public <A extends InterfaceA, B extends InterfaceB> void myMethod(Map<A, List<B>> map) { //do something }
Declare your map as HashMap<ObjectA, List<? extends InterfaceB>> hashMap = new HashMap<ObjectA, List<? extends InterfaceB>>();
Why nested wildcard capture is not possible?
I'm struggling to capture a wildcard when it is "nested in another wildcard". Is it possible? The code: public class ConvolutedGenerics { // listClass is a class implementing a List of some Serializable type public void doSomethingWithListOfSerializables( Class<? extends List<? extends Serializable>> listClass) { // Capture '? extends Serializable' as 'T extends Serializable' // The line does not compile with javac 7 captureTheWildcard(listClass); // <------ zonk here } // listClass is a class implementing a List of some Serializable type private <T extends Serializable> void captureTheWildcard( Class<? extends List</* ? extends */T>> listClass) { // Do something } } compiled with javac 7 produces: ConvolutedGenerics.java:18: error: method captureTheWildcard in class ConvolutedGenerics cannot be applied to given types; captureTheWildcard(listClass); ^ required: Class<? extends List<T>> found: Class<CAP#1> reason: no instance(s) of type variable(s) T exist so that argument type Class<CAP#1> conforms to formal parameter type Class<? extends List<T>> where T is a type-variable: T extends Serializable declared in method <T>captureTheWildcard(Class<? extends List<T>>) where CAP#1 is a fresh type-variable: CAP#1 extends List<? extends Serializable> from capture of ? extends List<? extends Serializable> 1 error Besides many more simpler cases I've found Incompatible generic wildcard captures Using Java wildcards but I could not infer an answer for my problem from those.
It is not possible, as you probably already know. Let me illustrate with a counter-example: List<Integer> foo = Arrays.asList(1,2,3); List<String> bar = Arrays.asList("hi","mom"); List<List<? extends Serializable>> baz = Arrays.asList(foo, bar); doSomethingWithListOfSerializables(baz); public void doSomethingWithListOfSerializables( List<? extends List<? extends Serializable>> listList) { captureTheWildcard(listList); } private <T extends Serializable> void captureTheWildcard( List<? extends List<T>> listList) { // Do something } What should T be?
The problem with your code is that you're trying to call captureTheWildcard passing different typed parameter then defined here: private <T extends Serializable> void captureTheWildcard(Class<? extends List<T>> listClass) You should explicitly say in your method definition that parameter passed is actually of type of Class<? extends List<? extends Serializable>>or modify the type of listClass like this: import java.util.List; import java.io.Serializable; public class ConvolutedGenerics { // listClass is a class implementing a List of some Serializable type public <T extends Serializable> void doSomethingWithListOfSerializables( Class<? extends List<T>> listClass) { // Capture '? extends Serializable' as 'T extends Serializable' // The line does not compile with javac 7 captureTheWildcard(listClass); // <------ zonk here } // listClass is a class implementing a List of some Serializable type private <T extends Serializable> void captureTheWildcard(Class<? extends List<T>> listClass) { // Do something } } Compiles well with javac 1.7.0_25
Java recursive generics class wildcard matching
Having these generic interface and class: interface TestIntf<T extends TestIntf<T>> { void test(T param); } class TestImpl<T extends TestIntf<T>> implements TestIntf<T> { #Override public void test(T param) { System.out.println(param); } } This fails: Class<? extends TestIntf<?>> clz = TestImpl.class; (Type mismatch: cannot convert from Class<TestImpl> to Class<? extends TestIntf<?>>) Why? How to properly provide a reference to TestImpl class to match Class<? extends TestIntf<?>>?
You can't. Use an unsafe cast. Class<? extends TestIntf<?>> clz = (Class<? extends TestIntf<?>>) TestImpl.class; or don't use the inner generics: Class<? extends TestIntf> clz = TestImpl.class; Update: When it regards annotations, there is nothing you can do - the annotation has to be changed. You cannot have a class literal represent a generic type.