Compiling Hadoop wordcount tutorial - hadoop 2.6.3 - java

The following is my system configuration: windows8.1, vmware v12, 2 centOS VMs. I have installed hadoop on my VMs, one as master and the other as slave1. The services are correctly up and running on both nodes.
I want to compile my first mapreduce program, the wordcount tutorial:
I'm trying to compile using the following:
[hadoop#master ~]$ javac -classpath $HADOOP_HOME/share/common/hadoop-common-2.6.3.jar:$HADOOP-HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.3.jar:$HADOOP_HOME/share/common/lib/common-cli-1.2.jar -d /home/hadoop *.java
but the answer I get is this:
WordCount.java:4: error: package org.apache.hadoop.conf does not exist
import org.apache.hadoop.conf.Configuration;
^
WordCount.java:5: error: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.Path;
^
WordCount.java:6: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.IntWritable;
^
WordCount.java:7: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
^
WordCount.java:8: error: package org.apache.hadoop.mapreduce does not exist
(.
.
.) ^
symbol: class Path
location: class WordCount
WordCount.java:58: error: cannot find symbol
FileOutputFormat.setOutputPath(job, new Path(args[1]));
^
symbol: variable FileOutputFormat
location: class WordCount
40 errors
I googled and everyone said it's due to $CLASSPATH.
this is my /etc/bashrc file:
export JAVA_HOME=/opt/jdk
export JRE_HOME=/opt/jdk/jre
export PATH=$PATH:/opt/jdk/bin:/opt/jdk/jre/bin
export HADOOP_CLASSPATH=$JAVA_HOME/lib/tools.jar
export HADOOP_HOME=/opt/hadoop/
export CLASSPATH=$CLASSPATH:$HADOOP_HOME/share/common/hadoop-common-2.6.3.jar:$HADOOP-HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.3.jar:$HADOOP_HOME/share/common/lib/common-cli-1.2.jar
any idea why I get this error?? I see no problem... :(

According to the documentation you can do the following (tested on my system and works)
export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar
$HADOOP_HOME/bin/hadoop com.sun.tools.javac.Main WordCount.java
and finally create the jar using
jar cf wc.jar WordCount*.class
link to the official doc here

Related

Differing behavior in VSCode with Maven on PC and Mac

We're using Maven in VSCode to develop with JavaFX. Maven works fine with VSCode on PCs but we can't use the "Play" button or "Run Without Debugging" on Macs.
java 19.0.1
javafx 19
maven 3.8.6
VSCode 1.74.1
On a PC, we get this:
& 'C:\Program Files\Java\jdk-19\bin\java.exe' '#C:\Users\username\AppData\Local\Temp\cp_et8lcjrjl0fge41txqyx61dgy.argfile' '-m' 'activities/activities.HelloFX'
-XX:+ShowCodeDetailsInExceptionMessages -cp "C:\\Users\\username\\SoftDevII\\2225.124.MavenStarter\\target\\classes;C:\\Users\\username\\.m2\\repository\\org\\openjfx\\javafx-controls\\19\\javafx-controls-19.jar;C:\\Users\\username\\.m2\\repository\\org\\openjfx\\javafx-controls\\19\\javafx-controls-19-win.jar;C:\\Users\\username\\.m2\\repository\\org\\openjfx\\javafx-graphics\\19\\javafx-graphics-19.jar;C:\\Users\\username\\.m2\\repository\\org\\openjfx\\javafx-graphics\\19\\javafx-graphics-19-win.jar;C:\\Users\\username\\.m2\\repository\\org\\openjfx\\javafx-base\\19\\javafx-base-19.jar;C:\\Users\\username\\.m2\\repository\\org\\openjfx\\javafx-base\\19\\javafx-base-19-win.jar" -XX:+ShowCodeDetailsInExceptionMessages -cp "C:\\Users\\username\\SoftDevII\\2225.124.MavenStarter\\target\\classes;C:\\Users\\username\\.m2\\repository\\org\\openjfx\\javafx-controls\\19\\javafx-controls-19.jar;C:\\Users\\username\\.m2\\repository\\org\\openjfx\\javafx-controls\\19\\javafx-controls-19-win.jar;C:\\Users\\username\\.m2\\repository\\org\\openjfx\\javafx-graphics\\19\\javafx-graphics-19.jar;C:\\Users\\username\\.m2\\repository\\org\\openjfx\\javafx-graphics\\19\\javafx-graphics-19-win.jar;C:\\Users\\username\\.m2\\repository\\org\\openjfx\\javafx-base\\19\\javafx-base-19.jar;C:\\Users\\username\\.m2\\repository\\org\\openjfx\\javafx-base\\19\\javafx-base-19-win.jar"
And on Macs we get this, a very different invocation:
[Running] cd "/124-2225/2225.124.MavenStarter/src/main/java/activities/" && javac HelloFX.java && java HelloFX
HelloFX.java:3: error: package javafx.application does not exist
import javafx.application.Application; HelloFX.java:4: error: package javafx.stage does not exist
import javafx.stage.Stage;
^
HelloFX.java:6: error: cannot find symbol
public class HelloFX extends Application {
^
symbol: class Application
HelloFX.java:9: error: cannot find symbol
public void start(Stage primaryStage) throws Exception {
^
symbol: class Stage
location: class HelloFX
HelloFX.java:8: error: method does not override or implement a method from a supertype
#Override
^
HelloFX.java:17: error: cannot find symbol
launch(args);
^
symbol: method launch(String[])
location: class HelloFX
However, we can run it from the command line if we include what we already have set in VSCode's settings.json, "java.debug.settings.vmArgs":
java --module-path /Java/javafx-sdk-19/lib --add-modules javafx.controls,javafx.fxml,javafx.media activities/HelloFX.java
In addition, we can successfully execute the single file from Maven's javafx plugin, but we will have multiple application classes so that's not a good solution. It seems like a VSCode issue, but we have no idea where or why Macs and PCs are so different.

How to run a java class in Eclipse (part of a maven project with a pom file) from cmd

I have a maven project (folder ev) and I want to run a java class (ChargerLocation.java) from the cmd. I 've tried the link but didn't work. When I run the ChargerLocation.java from Eclipse it works but I want to run it from the cmd.
It looks like that I should be able to run by: Run Configurations->Show Command Line and paste that in my cmd, however I get the following error'-classpath' is not recognized as an internal or external command, operable program or batch file.
In order to do the: Run Configurations->Show Command Line-> paste in cmd, in which folder should I be located?
I was wondering whether the command at the cmd should be something like Maven/apache-maven-3.6.3/bin/mvn exec:java -Dexec.mainClass="" since it's a maven project as a whole? Or it doesn't matter?
By simply doing javac ChargerLocation.java I get 45 errors which seem to be like it cannot find the proper libraries to build.
ChargerLocation.java:14: error: package org.matsim.api.core.v01 does not exist
import org.matsim.api.core.v01.Coord;
^
ChargerLocation.java:15: error: package org.matsim.api.core.v01 does not exist
import org.matsim.api.core.v01.Id;
^
ChargerLocation.java:16: error: package org.matsim.api.core.v01 does not exist
import org.matsim.api.core.v01.Scenario;
^
ChargerLocation.java:17: error: package org.matsim.api.core.v01.network does not exist
import org.matsim.api.core.v01.network.Link;
^
ChargerLocation.java:18: error: package org.matsim.api.core.v01.network does not exist
import org.matsim.api.core.v01.network.Network;
^
ChargerLocation.java:19: error: package org.matsim.contrib.ev.charging does not exist
import org.matsim.contrib.ev.charging.ChargeUpToMaxSocStrategy;
^
ChargerLocation.java:20: error: package org.matsim.contrib.ev.charging does not exist
import org.matsim.contrib.ev.charging.ChargingLogic;
^
ChargerLocation.java:21: error: package org.matsim.contrib.ev.charging does not exist
import org.matsim.contrib.ev.charging.ChargingWithQueueingLogic;
^
ChargerLocation.java:22: error: package org.matsim.contrib.ev.infrastructure does not exist
import org.matsim.contrib.ev.infrastructure.Charger;
^
ChargerLocation.java:23: error: package org.matsim.contrib.ev.infrastructure does not exist
import org.matsim.contrib.ev.infrastructure.ChargerImpl;
^
ChargerLocation.java:24: error: package org.matsim.contrib.ev.infrastructure does not exist
import org.matsim.contrib.ev.infrastructure.ChargerSpecification;
^
ChargerLocation.java:25: error: package org.matsim.contrib.ev.infrastructure does not exist
import org.matsim.contrib.ev.infrastructure.ChargerWriter;
^
ChargerLocation.java:26: error: package org.matsim.contrib.ev.infrastructure does not exist
import org.matsim.contrib.ev.infrastructure.ImmutableChargerSpecification;
^
ChargerLocation.java:27: error: package org.matsim.core.config does not exist
import org.matsim.core.config.Config;
I also attach a photo of my folders in eclipse.
Finally, I want to call this java file from Python, so what I am trying to do here is: open cmd from Python (os.system()) with pre-written command to call java Class (ChargerLocation.java) and finally run the java class.
Thanks in advance
#howlger here is the command line I get from eclipse.
It's not the arguments that increase the number of characters but the different jar libraries that needs to be build first before running my main java file org.matsim.contrib.ev.example.ChargerLocation
As far as I understand the argfile is to write arguments, which in my case there aren't any.

Error when compiling java file in cmd [duplicate]

From the Definitive ANTLR4 reference I have run through the first example and it has generated the JAVA target. In the directory C:\JavaLib I have antlr-4.5-complete.jar
When I attempt to compile it with;
javac -classpath C:\JavaLib *.java
It creates the following error messages;
helloBaseListener.java:13: error: class HelloBaseListener is public, should be declared in a file named HelloBaseListener.java
public class HelloBaseListener implements HelloListener {
^
helloListener.java:9: error: class HelloListener is public, should be declared in a file named HelloListener.java
public interface HelloListener extends ParseTreeListener {
^
helloParser.java:12: error: class HelloParser is public, should be declared in a file named HelloParser.java
public class HelloParser extends Parser {
^
helloBaseListener.java:3: error: package org.antlr.v4.runtime does not exist
import org.antlr.v4.runtime.ParserRuleContext;
^
helloBaseListener.java:4: error: package org.antlr.v4.runtime.misc does not exist
import org.antlr.v4.runtime.misc.NotNull;
^
helloBaseListener.java:5: error: package org.antlr.v4.runtime.tree does not exist
import org.antlr.v4.runtime.tree.ErrorNode;
....
What am I doing wrong?
There was 2 problems. One was the file has to be named "Hello.g4" not "hello.g4" because the grammar is specified as Hello. The second was the classpath, it requires the path and name of the jar file, as well as the current directory. The following command worked;
javac -classpath .;C:\JavaLib\antlr-4.5-complete.jar *.java
With regard to the above query re the colon separator then the answer is yes. I installed via the debian packages and used the command before working out how to set CLASSPATH
javac -classpath /usr/share/java/antlr4-runtime.jar Expr*.java
Before this I got a load of compile errors. Also it seems to be worth noting on debian at the moment my .bash_profile never gets loaded so I needed to put this in .bashrc

Unable to compile a JUnit test on Windows

I'm trying to compile this very simple JUnit test on my Windows 10 box:
import static org.junit.Assert.assertTrue;
import org.junit.Test;
public class TestHelloWorld
{
#Test
public void BasicTest ()
{
assertTrue(true);
}
}
However, this seemingly innocuous task has proven impervious to three to four hours of research and trial and error.
From a Cygwin terminal I see this:
javac -cp lib/hamcrest-core-1.3.jar:/lib/junit-4.12.jar:. TestHe
TestHelloWorld.java:1: error: package org.junit does not exist
import static org.junit.Assert.assertTrue;
^
TestHelloWorld.java:1: error: static import only from classes and interfaces
import static org.junit.Assert.assertTrue;
^
TestHelloWorld.java:2: error: package org.junit does not exist
import org.junit.Test;
^
TestHelloWorld.java:6: error: cannot find symbol
#Test
^
symbol: class Test
location: class TestHelloWorld
TestHelloWorld.java:9: error: cannot find symbol
assertTrue(true);
^
symbol: method assertTrue(boolean)
location: class TestHelloWorld
5 errors
and from a CMD prompt I see this:
javac -cp C:\Users\canon\Documents\GitHub\cse1341\AssignmentTester\lib\junit.jar TestHelloWorld.java
error: error reading C:\Users\canon\Documents\GitHub\cse1341\AssignmentTester\lib\junit.jar; error in opening zip file
1 error
I honestly have no idea what I'm doing incorrectly here. I've always been on a *nix machine before, where I don't recall having any problems compiling JUnit tests when I pass in the junit and hamcrest jars to the class path using the -cp flag. Any ideas or guidance here would be much appreciated!
There are two thing:
Java in windows using ';' to separate the jar files in classpath argument so it should be
"javac -cp lib/hamcrest-core-1.3.jar;/lib/junit-4.12.jar;. TestHeTestHelloWorld.java" in cygwin
check if C:\Users\canon\Documents\GitHub\cse1341\AssignmentTester\lib\junit.jar is a valid jar. (you can use zip tool to check).
The slash before lib/junit-4.12.jar seems to be wrong.

getting error when I compile the Java code using package in commandline?

I have this directory structure:
project1/src/edu/course/firstweek/javacourse/Program1.java
Another file in one package above:
project1/src/edu/course/firstweek/program2.java
In the header of program2.java, I have
package edu.course.firstweek;
import edu.course.firstweek.javacourse.Program1;
Now to when I run the following in commandline:
Javac src/edu/course/firstweek/program2.java, I get this error:
src/edu/course/firstweek/program2.java:14:error cannot find symbol
System.out.println(program1.print("hello world"));
symbol: variable Program1
location: class program2
2 errors
I can see that the compiler is not able to find the program1,but I have the correct import package statement in program2. I need help here and after compiling, is there something that needs to be taken into account for running the program.
Thanks
Try going into one directory inside, i.e. cd src
and then compile Javac edu/course/firstweek/program2.java
for running, do java edu.course.firstweek.program2

Categories