I've got a word count example class called by a Jetty server's handle method. The job is submitted with:
JobClient.runJob(conf);
Sadly, the JobTracker sees nothing, the job does not get submitted, but it finishes and creates output. I've found out, that I have to inject my configurations with:
conf.addResource(new Path("/usr/local/hadoop/conf/mapred-site.xml"));
After adding this line, the JobTracker gets the job, but it fails with the following exception:
INFO: Task Id : attempt_201312071601_0005_m_000000_1, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: WC2$Reduce
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:889)
at org.apache.hadoop.mapred.JobConf.getCombinerClass(JobConf.java:1049)
at org.apache.hadoop.mapred.Task$CombinerRunner.create(Task.java:1385)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:986)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:422)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: WC2$Reduce
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:881)
... 10 more
Caused by: java.lang.ClassNotFoundException: WC2$Reduce
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:855)
... 11 more
My combiner class is the same as the reducer class. I've downloaded the Hadoop WordCount example without modifications. Only the mentioned addResource.
Related
I have created a jar using eclipse for the mapreduce jobs. If you extract the jar you can see all the class present there. When you run the jar in hadoop``usinghadoop` command it shows error shown below.
Its not able to recognize only one class i.e Test_project$TwoDArrayWritables. Test_project is the main class and TwoDArrayWritables is a class within Test_project. TwoDArrayWritables inherit TwoDArrayWritable built-in class in hadoop.
Jar extract Image:
Error:
16/04/05 15:48:28 INFO mapred.JobClient: Task Id : attempt_201604051120_0002_m_000000_1, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: mapreduce.Test_project$TwoDArrayWritables
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:889)
at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:747)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:966)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:422)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: mapreduce.Test_project$TwoDArrayWritables
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:881)
... 9 more
Caused by: java.lang.ClassNotFoundException: mapreduce.Test_project$TwoDArrayWritables
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:855)
... 10 more
16/04/05 15:48:34 INFO mapred.JobClient: Task Id : attempt_201604051120_0002_m_000000_2, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: mapreduce.Test_project$TwoDArrayWritables
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:889)
at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:747)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:966)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:422)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: mapreduce.Test_project$TwoDArrayWritables
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:881)
... 9 more
Caused by: java.lang.ClassNotFoundException: mapreduce.Test_project$TwoDArrayWritables
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:855)
... 10 more
Googled many solutions but nothing worked. Please help!
try this: bin/hadoop jar /path/to/jarfile/newproj.jar Test_project.TwoDArrayWritables /user/hduser/input /user/hduser/output1
Struggling for about weeks , something clicked in my mind i.e
I was using two reducers in my job so defining two Jobconf for each:
My earlier(wrong code)
JobConf conf = new JobConf(getConf(), Test_project.class);
JobConf conf2 = new JobConf(getConf());
As I thought that configuration is already defined so didn't mention class Test_project.class in conf2
My present (Correct Code)
JobConf conf = new JobConf(getConf(), Test_project.class);
JobConf conf2 = new JobConf(getConf(), Test_project.class);
The error was thrown because on execution it was searching for Test_project$TwoDArrayWritables , as it has no Test_project.class class , it can not locate TwoDArrayWritables
Now it works fine.
Can anyone tell me more about this error?
Exception in thread "main" java.lang.NoClassDefFoundError: org/rosuda/JRI/Rengine
at DataMining.RunMinerBack.checkRLibVersion(RunMinerBack.java:643)
at DataMining.RunMinerBack.init(RunMinerBack.java:831)
at DataMining.RunMinerBack.<init>(RunMinerBack.java:126)
at DataMining.ListMergeMembers.init(ListMergeMembers.java:1453)
at DataMining.ListMergeMembers.<init>(ListMergeMembers.java:95)
at DataMining.ListMergeMembers.main(ListMergeMembers.java:1585)
Caused by: java.lang.ClassNotFoundException: org.rosuda.JRI.Rengine
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 6 more
I have rJava installed, and I have JRI.jar in my CLASSPATH. I also checked and I do have Rengine.class in within JRI.jar, at the same path - org/rosuda/JRI/Rengine.class. I'm not sure what/where else to check, so any suggestions would be appreciated.
I'm attempting to make a very simple Kafka Producer and am currently following the producer example except my producer does not have a partitioner class.
After exporting required files into a jar I transfer them to my Linux image and try to run it.
I get this exception:
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.lang.NoClassDefFoundError: org/apache/kafka/common/utils/Utils
at kafka.client.ClientUtils$$anonfun$parseBrokerList$1.apply(ClientUtils.scala:103)
at kafka.client.ClientUtils$$anonfun$parseBrokerList$1.apply(ClientUtils.scala:102)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:194)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:194)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:194)
at scala.collection.mutable.ArrayBuffer.map(ArrayBuffer.scala:44)
at kafka.client.ClientUtils$.parseBrokerList(ClientUtils.scala:102)
at kafka.producer.BrokerPartitionInfo.<init>(BrokerPartitionInfo.scala:32)
at kafka.producer.async.DefaultEventHandler.<init>(DefaultEventHandler.scala:41)
at kafka.producer.Producer.<init>(Producer.scala:60)
at kafka.javaapi.producer.Producer.<init>(Producer.scala:26)
at producers.HelloWorldProducer.main(HelloWorldProducer.java:20)
... 5 more
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.utils.Utils
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 19 more
After looking at the kafka jar I see that the utils is its own package now and not located within common.
What would be the best way to solve this issue?
The answer ended up being real silly ... I needed to use the kafka-clients-0.8.2.0.jar instead.
I am trying to bulk load a sample data into HBase table using Java code I found at this link.
Although I have included the guava-11.0.2.jar, I am getting this error and my program is failing:
14/08/19 18:59:15 INFO mapred.JobClient: Task Id : attempt_201408181320_0291_r_000000_2,
Status : FAILED
Error: java.lang.ClassNotFoundException: com.google.common.primitives.Longs
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1541)
at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1523)
at java.util.TreeMap.compare(TreeMap.java:1188)
at java.util.TreeMap.put(TreeMap.java:531)
at java.util.TreeSet.add(TreeSet.java:255)
at org.apache.hadoop.hbase.mapreduce.KeyValueSortReducer.reduce(KeyValueSortReducer.java:41)
at org.apache.hadoop.hbase.mapreduce.KeyValueSortReducer.reduce(KeyValueSortReducer.java:35)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
I have just started programming in Java. Can someone please help with this?
When I execute a recently installed Gremlin by /bin/gremlin.sh I got:
Exception in thread "main" java.lang.NoClassDefFoundError: org/fusesource/jansi/AnsiRenderWriter
at com.tinkerpop.gremlin.groovy.console.Console.<init>(Console.java:71)
at com.tinkerpop.gremlin.groovy.console.Console.main(Console.java:96)
Caused by: java.lang.ClassNotFoundException: org.fusesource.jansi.AnsiRenderWriter
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
What could be wrong?
The jansi library is not in your classpath.