I'm trying Hadoop's Basic MapReduce Program whose tutorial is on http://java.dzone.com/articles/hadoop-basics-creating
The Full code of the class is(the code is present on net on above url)
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.KeyValueTextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;
public class Dictionary {
public static class WordMapper extends Mapper<Text, Text, Text, Text> {
private Text word = new Text();
public void map(Text key, Text value, Context context) throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString(), ",");
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(key, word);
}
}
}
public static class AllTranslationsReducer extends Reducer<Text, Text, Text, Text> {
private Text result = new Text();
public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
String translations = "";
for (Text val : values) {
translations += "|" + val.toString();
}
result.set(translations);
context.write(key, result);
}
}
public static void main(String[] args) throws Exception {
System.out.println("welcome to Java 1");
Configuration conf = new Configuration();
System.out.println("welcome to Java 2");
Job job = new Job(conf, "dictionary");
job.setJarByClass(Dictionary.class);
job.setMapperClass(WordMapper.class);
job.setReducerClass(AllTranslationsReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
job.setInputFormatClass(KeyValueTextInputFormat.class);
FileInputFormat.addInputPath(job, new Path("/tmp/hadoop-cscarioni/dfs/name/file"));
FileOutputFormat.setOutputPath(job, new Path("output"));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
But after running in eclipse; I'm getting the error,
welcome to Java 1
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:73)
at Dictionary.main(Dictionary.java:43)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 2 more
Please note that the exception is NoClassDefFoundError instead of ClassNotFoundException.
Note : NoClassDefFoundError is thrown when a class is not visible at run time but was visible at compile time. This may be something that can happen in the distribution or production of JAR files, where not all the required class files were included.
To fix : Please check for differences in your build time and runtime classpaths.
NoClassDefFoundError and ClassNotFoundException are different.
one is an Error and the other is an Exception.
NoClassDefFoundError: arises from the JVM having problems finding a class it expected to find. A program that was working at compile-time can't run because of class files not being found.
ClassNotFoundException: This exception indicates that the class was not found on the classpath i.e we are trying to load the class definition and class/jar containing the class does not exist in the classpath.
NoClassDefFoundError comes when a class is not visible at run time but was at compile time. Which may be related to JAR files, because all the required class files were not included.
So try adding in your class path commons-logging-1.1.1 jar which you can get from http://commons.apache.org/logging/download_logging.cgi
NoClassDefFoundError occurs when the named class is successfully located in the classpath, but for some reason cannot be loaded and verified. Most often the problem is that another class needed for the verification of the named class is either missing or is the wrong version.
Generally speaking, this error means "double-check that you have all the right JAR files (of the right version) in your classpath".
It's a very common error when you run a Hadoop Map/Reduce program in local IDE (Eclipse).
You should already added hadoop-core.jar in your build path, so no compile error detected in your program. But you get the error when you run it, because hadoop-core is dependent on commons-logging.jar (as well as some other jars). You may need to add the jars under /lib to your build path.
I recommend you to use Maven or other dependency management tool to manage the dependency.
Please read an article: http://kishorer.in/2014/10/22/running-a-wordcount-mapreduce-example-in-hadoop-2-4-1-single-node-cluster-in-ubuntu-14-04-64-bit/. It explains how to reference dependencies in Eclipse without Marven. However, Marven is preferred way, from what I understood.
Related
I am working on a project that is supposed to parse texts from PDF files.
Having multiple dependencies I have decided to build a combined JAR with all the dependencies and the classes.
However, when I build JAR including dependencies via Intellij IDEA even though the JAR file is added properly and I can import the class the program throws NoClassDefFoundError (Please refer to the screenshot).
Firstly, I thought the jar wasn't in the classpath. However, even if I add -cp TessaractPDF.jar through VM Options the class still get undetected.
I think it is worth to mention that, everything works smoothly if I build JAR without dependencies and add the dependencies manually.
What should I do?
Exception in thread "main" java.lang.NoClassDefFoundError: me/afifaniks/parsers/TessPDFParser
at Test.main(Test.java:20)
Caused by: java.lang.ClassNotFoundException: me.afifaniks.parsers.TessPDFParser
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 1 more
Code Snippet:
import me.afifaniks.parsers.TessPDFParser;
import java.io.IOException;
import java.util.HashMap;
public class Test {
public static void main(String[] args) throws IOException {
System.out.println(System.getProperty("java.classpath"));
HashMap<String, Object> arguments = new HashMap<>();
arguments.put("imageMode", "binary");
arguments.put("toFile", false);
arguments.put("tessDataPath", "/home/afif/Desktop/PDFParser/tessdata");
TessPDFParser pdfParser = new TessPDFParser("hiers15.pdf", arguments);
String text = (String) pdfParser.convert();
System.out.println(text);
}
}
I am trying to classify an instance using a .model file which I have created on the Weka GUI. It seems I have successfully classified the test instance, however, I am not sure whether I am able to successfully load my .model file and of the Stub compiler error.
I have tried to remove the extends AppCompatActivity and if that makes any difference in the .model upload. It turns out that to use getAssets(), the code must be in an activity. However, I an still unsure of whether the model has upload and the unusual compiler error. I have followed the basic framework of #davidmascharka's work on GitHub (he's also loading a WEKA model from assets), but mine does not compile.
Here's my code:
package com.example.owner.introductoryapplication;
import android.support.v7.app.AppCompatActivity;
import weka.classifiers.Classifier;
import weka.classifiers.rules.DecisionTable;
import weka.core.Attribute;
import weka.core.DenseInstance;
import weka.core.Instances;
import java.util.ArrayList;
public class Test extends AppCompatActivity {
public static void main(String[] args) {
Test test = new Test();
test.start();
}
public void start() {
//LOADS THE MODEL...------------------------------------------------------
String rootPath = "/assets/";
String fileName = "PGBD_DecisionTableUPD.model";
Classifier cls = null;
try {
//cls = (Classifier) weka.core.SerializationHelper.read(rootPath + fileName);
cls = (DecisionTable) weka.core.SerializationHelper.read(getAssets().open(fileName));
} catch (Exception e) {
e.printStackTrace();
}
}
}
And here's my error output:
Exception in thread "main" java.lang.RuntimeException: Stub!
at android.content.Context.<init>(Context.java:67)
at android.content.ContextWrapper.<init>(ContextWrapper.java:30)
at android.view.ContextThemeWrapper.<init>(ContextThemeWrapper.java:40)
at android.app.Activity.<init>(Activity.java:643)
at android.support.v4.app.SupportActivity.<init>(ComponentActivity.java:46)
at android.support.v4.app.FragmentActivity.<init>(FragmentActivity.java:68)
at android.support.v7.app.AppCompatActivity.<init>(AppCompatActivity.java:62)
at com.example.owner.introductoryapplication.Test.<init>(Test.java:13)
at com.example.owner.introductoryapplication.Test.main(Test.java:15)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMainV2.main(AppMainV2.java:131)
Process finished with exit code 1
I expect the program to at least compile! I have absolutely no clue why it's not. I tried switching the order of my dependencies, hoping that would make a difference, but to no luck.
Any ideas?
Thanks in advance.
This may have been covered before, but weka.jar only allows for Stub implementations. Essentially, you must configure the run setting to "app" instead of a specific file.
If you want to see how a specific file works, then you can use the debug option for your app.
I am trying to execute a sample program in Apache Flink in local mode.
import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.java.DataSet;
import org.apache.flink.api.java.ExecutionEnvironment;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.util.Collector;
public class WordCountExample {
public static void main(String[] args) throws Exception {
final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
DataSet<String> text = env.fromElements(
"Who's there?",
"I think I hear them. Stand, ho! Who's there?");
//DataSet<String> text1 = env.readTextFile(args[0]);
DataSet<Tuple2<String, Integer>> wordCounts = text
.flatMap(new LineSplitter())
.groupBy(0)
.sum(1);
wordCounts.print();
env.execute();
env.execute("Word Count Example");
}
public static class LineSplitter implements FlatMapFunction<String, Tuple2<String, Integer>> {
#Override
public void flatMap(String line, Collector<Tuple2<String, Integer>> out) {
for (String word : line.split(" ")) {
out.collect(new Tuple2<String, Integer>(word, 1));
}
}
}
}
It is giving me exception :
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/InputFormat
at WordCountExample.main(WordCountExample.java:10)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.InputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 1 more
What am I doing wrong?
I have used the correct jars also.
flink-java-0.9.0-milestone-1.jar
flink-clients-0.9.0-milestone-1.jar
flink-core-0.9.0-milestone-1.jar
Adding the three Flink Jar files as dependencies in your project is not enough because they have other transitive dependencies, for example on Hadoop.
The easiest way to get a working setup to develop (and locally execute) Flink programs is to follow the quickstart guide which uses a Maven archetype to configure a Maven project. This Maven project can be imported into your IDE.
NoClassDefFoundError extends LinkageError
Thrown if the Java Virtual Machine or a ClassLoader instance tries to
load in the definition of a class (as part of a normal method call or
as part of creating a new instance using the new expression) and no
definition of the class could be found. The searched-for class
definition existed when the currently executing class was compiled,
but the definition can no longer be found.
Your code/jar dependent to hadoop. Found it here download jar file and add it in your classpath org.apache.hadoop.mapreduce.InputFormat
Firstly, the flink jar files which you have included in your project are not enough, include all the jar files which are present in the lib folder present under the flink's source folder.
Secondly, " env.execute();
env.execute("Word Count Example");" These lines of code are not required since you are just printing your dataset onto the console; you're not writing the output into a file(.txt, .csv etc.). So, better to remove these lines (Sometimes throws errors if included in code if not required (observed a lot of times))
Thirdly, while exporting the jar files for your Java Project from your IDE, don't forget to select your 'Main' class.
Hopefully, after making the above changes, your code works.
I am using Hadoop 1.0.3 and HBase 0.94.22. I am trying to run a mapper program to read values from a Hbase table and output them to a file. I am getting the following error:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:340)
at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
The code is as below
import java.io.IOException;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter;
import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
import org.apache.hadoop.hbase.mapreduce.TableMapper;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class Test {
static class TestMapper extends TableMapper<Text, IntWritable> {
private static final IntWritable one = new IntWritable(1);
public void map(ImmutableBytesWritable row, Result value, Context context) throws IOException, InterruptedException
{
ImmutableBytesWritable userkey = new ImmutableBytesWritable(row.get(), 0 , Bytes.SIZEOF_INT);
String key =Bytes.toString(userkey.get());
context.write(new Text(key), one);
}
}
public static void main(String[] args) throws Exception {
HBaseConfiguration conf = new HBaseConfiguration();
Job job = new Job(conf, "hbase_freqcounter");
job.setJarByClass(Test.class);
Scan scan = new Scan();
FileOutputFormat.setOutputPath(job, new Path(args[0]));
String columns = "data";
scan.addFamily(Bytes.toBytes(columns));
scan.setFilter(new FirstKeyOnlyFilter());
TableMapReduceUtil.initTableMapperJob("test",scan, TestMapper.class, Text.class, IntWritable.class, job);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
System.exit(job.waitForCompletion(true)?0:1);
}
}
I get the above code exported to a jar file and on the command line I use the below command to run the above code.
hadoop jar /home/testdb.jar test
where test is the folder to which the mapper results should be written.
I have checked a few other links like Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.KeeperException where it has been suggested to include the zookeeper file in the classpath, but while creating the project in eclipse I have already included zookeeper file from the lib directory of hbase. The file I have included is zookeeper-3.4.5.jar. Ans also visited this link too HBase - java.lang.NoClassDefFoundError in java , but I am using a mapper class to get the values from the hbase table not any client API. I know I am making a mistake somewhere, guys could you please help me out ??
I have noted another strange thing, when I remove all of the code in the main function except the first line " HBaseConfiguration conf = new HBaseConfiguration();", then export the code to a jar file and try to compile the jar file as hadoop jar test.jar I still get the same error. It seems either I am defining the conf variable incorrectly or there is some issue with my environment.
I got the fix to the problem, I had not added the hbase classpath in the hadoop-env.sh file. Below is the one I added to make the job work.
$ export HADOOP_CLASSPATH=$HBASE_HOME/hbase-0.94.22.jar:\
$HBASE_HOME/hbase-0.94.22-test.jar:\
$HBASE_HOME/conf:\
${HBASE_HOME}/lib/zookeeper-3.4.5.jar:\
${HBASE_HOME}/lib/protobuf-java-2.4.0a.jar:\
${HBASE_HOME}/lib/guava-11.0.2.jar
I tried editing the hadoop-env.sh file, but the changes mentioned here didn't work for me.
What worked is this:
export HADOOP_CLASSPATH="$HADOOP_CLASSPATH:$HBASE_HOME/lib/*"
I just added that at the end of my hadoop-env.sh.
Do not forget to set your HBASE_HOME variable.
You can also replace the $HBASE_HOME with the actual path of your hbase installation.
In case there is someone who has different paths/configuration. Here is what I added to hadoop-env.sh in order to make it work:
$ export HADOOP_CLASSPATH="$HBASE_HOME/lib/hbase-client-0.98.11-hadoop2.jar:\
$HBASE_HOME/lib/hbase-common-0.98.11-hadoop2.jar:\
$HBASE_HOME/lib/protobuf-java-2.5.0.jar:\
$HBASE_HOME/lib/guava-12.0.1.jar:\
$HBASE_HOME/lib/zookeeper-3.4.6.jar:\
$HBASE_HOME/lib/hbase-protocol-0.98.11-hadoop2.jar"
NOTE: if you haven't set the $HBASE_HOME you have 2 choices.
- By export HBASE_HOME=[your hbase installation path]
- Or just replace the $HBASE_HOME with your hbase full path
HADOOP_USER_CLASSPATH_FIRST=true \
HADOOP_CLASSPATH=$($HBASE_HOME/bin/hbase mapredcp) \
hadoop jar /home/testdb.jar test
here CreateTable is my java class file
use this command
java -cp .:/home/hadoop/hbase/hbase-0.94.8/hbase-0.94.8.jar:/home/hadoop/hbase/hbase-0.94.8/lib/* CreateTable
via eclipse, I am trying to run builtin example of file system (HelloFS.java) of fuse-jna, but it gives me java.lang.NoClassDefFoundError .
My source project is in /home/syed/workspace/HelloFS
fuse-jna class files are in home/syed/Downloads/fuse-jna-master/build/classes/main/net/fusejna
In eclipse, I added class folder via buildpath and also jre path in envirnment file. I attached snapshot below.
Please help me run this example in eclipse.
error:
Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/jna/Structure
at net.fusejna.FuseFilesystem.mount(FuseFilesystem.java:545)
at net.fusejna.FuseFilesystem.mount(FuseFilesystem.java:550)
at HelloFS.main(HelloFS.java:22)
Caused by: java.lang.ClassNotFoundException: com.sun.jna.Structure
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 3 more
here is code of builtin example file system (with not red underline, which i think means that eclipse build path is entered correctly, ):
import java.io.File;
import java.nio.ByteBuffer;
import net.fusejna.DirectoryFiller;
import net.fusejna.ErrorCodes;
import net.fusejna.FuseException;
import net.fusejna.StructFuseFileInfo.FileInfoWrapper;
import net.fusejna.StructStat.StatWrapper;
import net.fusejna.types.TypeMode.NodeType;
import net.fusejna.util.FuseFilesystemAdapterFull;
public class HelloFS extends FuseFilesystemAdapterFull
{
public static void main(String args[]) throws FuseException
{
/*if (args.length != 1) {
System.err.println("Usage: HelloFS <mountpoint>");
System.exit(1);
}*/
new HelloFS().log(true).mount("./testfs1");
}
private final String filename = "/hello.txt";
private final String contents = "Hello World!\n";
#Override
public int getattr(final String path, final StatWrapper stat)
{
if (path.equals(File.separator)) { // Root directory
stat.setMode(NodeType.DIRECTORY);
return 0;
}
if (path.equals(filename)) { // hello.txt
stat.setMode(NodeType.FILE).size(contents.length());
return 0;
}
return -ErrorCodes.ENOENT();
}
#Override
public int read(final String path, final ByteBuffer buffer, final long size, final long offset, final FileInfoWrapper info)
{
// Compute substring that we are being asked to read
final String s = contents.substring((int) offset,
(int) Math.max(offset, Math.min(contents.length() - offset, offset + size)));
buffer.put(s.getBytes());
return s.getBytes().length;
}
#Override
public int readdir(final String path, final DirectoryFiller filler)
{
filler.add(filename);
return 0;
}
}
This is envirnment file contents:
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
JAVA_HOME="/usr/lib/jvm/java-6-openjdk-i386"
This is fuse-jna classes path
I added /main folder
========================================================
#Viktor K. Thanks for the help,
the above mentioned error is fixed by downloading and adding com.sun.jna ยป jna to referece library
but now it shows me new error as
Dec 28, 2013 1:18:25 PM HelloFS getName
INFO: Method succeeded. Result: null
Dec 28, 2013 1:18:25 PM HelloFS getOptions
INFO: Method succeeded. Result: null
Exception in thread "main" java.lang.NoSuchMethodError: com.sun.jna.Platform.getOSType()I
at net.fusejna.Platform.init(Platform.java:39)
at net.fusejna.Platform.fuse(Platform.java:26)
at net.fusejna.FuseJna.init(FuseJna.java:113)
at net.fusejna.FuseJna.mount(FuseJna.java:172)
at net.fusejna.FuseFilesystem.mount(FuseFilesystem.java:545)
at net.fusejna.FuseFilesystem.mount(FuseFilesystem.java:550)
at HelloFS.main(HelloFS.java:22)
=======================================================
Hmmm
The one that I downloaded was not campatable I think,
in temp folder of fuse-jna
/home/syed/Downloads/fuse-jna-master/build/tmp/expandedArchives/jna-3.5.2.jar_r4n26u14up0smlb84ivcvfnke/
there was jna3.5.2 classes, I imported that to libraray, now its working fine.
My problem solved. Thanks a lot.
The problem is not in Fuse-JNA library. Fuse-JNA library is obviously dependent on jna library (can be found in public maven repository http://mvnrepository.com/artifact/com.sun.jna/jna). You need to add this library as dependency in your project. You can see that in your project's referenced libraries there is no com.sun.jna package available.
In general - if you want to use package A (in your case Fuse-JNA) and the package A depends on package B (in your case JNA) you have to add JNA package yourself as dependency to your project. In general it is very hard to find out what are all required dependencies of the packages that you want to use - therefore many projects are using maven (or any alternative like gradle). Check this if you want to learn more : Why maven? What are the benefits?. I strongly suggest to use a tool for dependency resolution (like maven) over manual dependency resolution.
Another approach is to download a fuse jar with all dependencies - if you believe that it is the only library you'll need. However, adding jar with dependencies can lead to a big disaster if you add later other dependencies. This could lead to dependencies conflict, which is hard to find problem.