I have installed and configured hadoop in a linux machine .Now i am trying to run a sample MR job.I have started the hadoop via the command /usr/local/hadoop/bin/start-all.sh and the output is
namenode running as process 7876. Stop it first.
localhost: datanode running as process 8083. Stop it first.
localhost: secondarynamenode running as process 8304. Stop it first.
jobtracker running as process 8398. Stop it first.
localhost: tasktracker running as process 8612. Stop it first.
so i think that my hadoop is configured successfully.But when i am tryinh to run below command it is giving
jeet#jeet-Vostro-2520:~$ hadoop fs -put gettysburg.txt /user/jeet/getty/gettysburg.txt
hadoop: command not found
i am new in hadoop.somebody please help .I am also posting the screenshot of what i am trying
As it seems from your commands history, you can replace hadoop by /usr/local/hadoop/bin/hadoop and it should help.
If you want to use hadoop command without specifying the full path to it, you can edit ~/.bashrc file and add the following line:
export PATH=$PATH:/usr/local/hadoop/bin/
Then you need to reopen your terminal.
edit PATH variable, if you want to be able to invoke hadoop without specifying full path
export PATH=$PATH:/usr/local/hadoop/bin/
if you want it for each bash profile then edit ~/.bash_profile to include this
I got the same error, and this worked for me
I configured path variable in.bashrc.
export HADOOP_HOME=/opt/hadoop
export PATH= $PATH:$HADOOP_HOME/bin
Sometime restarting your machine can resolve the issue ,only if you have configured everything correct.
cd ~
vi .bashrc
export PATH=$PATH:<hadoop installation path>
for example replace <hadoop installation path> by /usr/local/hadoop/bin/
once echo your path ,if your path hasn't set ,then go to your .bashrc file
vi ~/.bashrc
and add folliwing in that.
export PATH=$PATH:/usr/local/hadoop/bin/
Please make sure that you are logged into the particular user, whose .bashrc file has got this entry
export PATH=$PATH:/usr/local/hadoop/bin/
Assuming that your hadoop setup is lying at /usr/local
Example You have set the .bashrc file for user hadoopuser in /home/hadoopuser/.bashrc, then you should be logged in as hadoop user only and not as any other user.
Hadoop command not found?
put this 3 commands at the end of ~/.bashrc file
sudo gedit ~/.bashrc
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_HOME=/home/Work/hadoop-1.2.1
export PATH=$HADOOP_HOME/bin:$PATH*
java-8-openjdk-amd64 - put your folder name
hadoop-1.2.1 - put your folder name
save the file and use below command
source ~/.bashrc
or simply close the terminal and open it again
Related
Complete error statement :
No command 'hdfs' found, did you mean:
Command 'hfs' from package 'hfsutils-tcltk' (universe)
Command 'hdfls' from package 'hdf4-tools' (universe)
hdfs: command not found
There may be something wrong with your HDFS installation, try giving the full path to hdfs in your command.
/path/to/dir/hdfs namenode -format
the path depends on your OS and hadoop distribution you're using etc. Use locate or find for it.
If it works with full path, its probably a PATH issue, export the path in your .bashrc and then try.
Cheers.
try with bin/hdfs instead of hdfs. It worked for me.
Check if you have added your hadoop executables to PATH properly.
add the below given export commands in .bashrc if not:
export HADOOP_INSTALL=/path/where/hadoop/is/installed
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
Finally source the bashrc file
I hope this error is resolved by now, but this error occurs because the hdfs path may not be set, so first check hdfs path by using this command,
which hdfs
if path is not set, then check the environment variable in ~/.bash_profile
if it PATH is written like this,
PATH=$PATH:$HOME/bin
change it to this
PATH=$PATH:$HADOOP_HOME/bin
source ~/.bash_profile
then check the your hdfs path, if it is set by using which command(as shown above)
Then your problem might get resolved.
Hope this helps!
I'm trying to install Hadoop and run it.
And I'm sure I've installed Hadoop and formatted namenode successfully.
However, when I tried to run start-dfs.sh, I got the error below:
localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-wenruo-namenode-linux.out
localhost: /usr/local/hadoop/bin/hdfs: line 304: /usr/local/hadoop/usr/lib/jvm/java-8-oracle/bin/java: No such file or directory
My JAVA_HOME is below:
echo $JAVA_HOME
/usr/lib/jvm/java-7-openjdk-amd64
My hadoop-env.sh file:
export JAVA_HOME=${JAVA_HOME}
How could Hadoop is still looking for JDK8 as I already set JAVA_HOME to JDK7?
Thank you very much.
In general each Hadoop distribution/version should have a few basic script files that set this JAVA_HOME environment variablesuch as yarn-env.sh file if you have yarn.
Also depending on your hadoop version you might also have the path in your *-site.xml files such as hdfs-site.xml, core-site.xml, yarn-site.xml, mapred-site.xml, and a few others depending on what services you have. It is likely your update to hadoop-env.sh did not regenerate the client configuration files unless you did it through a cluster manager application then redeployed client configuration files.
Sometimes these also I find get to set use the systems bin/java executable. You can use the following command to find out what java your OS has in your bin/ path.
readlink -f /usr/bin/java
/usr/bin/java -version
Did you also update hadoop-env.sh on each node then restart all services to make sure it is picked up again?
Leave it. The problem is resolved.
In hadoop-env.sh, I changed export JAVA_HOME=${JAVA_HOME} to echo $JAVA_HOME /usr/lib/jvm/java-7-openjdk-amd64.
It looks like ${JAVA_HOME} doesn't work.
I just open file :.bash_profile
export JAVA_HOME=/usr/java/jdk1.8.0_91/bin
Now in command prompt when I fire command : echo $JAVA_HOME , It returns me blank.
Am I missing anything? I am trying to set permanent class path in linux.
I have read and follow many similar questions but still no success.
The contents of .bash_profile are only executed when you start a Login Shell. This is what happens if you, say, log in, or if you run bash --login.
The file you're probably looking for is ~/.bashrc, which is sourced by bash (i.e. executed) every time you start bash. You should put your export there.
Alternatively, on Linux you could put the line JAVA_HOME=/usr/java/jdk1.8.0_91/bin in the file /etc/environment, which should do exactly what you're trying to do (but you need to log out and log in first). This will be system-wide, though.
(for the record, the system-wide .bashrc file is /etc/bash.bashrc)
EDIT: Protip: You can check your environment variables in BASH by typing export with no arguments.
If your .bash_profile doesn't read also .bashrc, you need to add the above export also to .bashrc.
Or, you could add:
if [ -f ~/.bashrc ]; then
source ~/.bashrc
fi
And add everything in .bashrc from now on.
I'm using Jenkins to deploy the build.
We need to extract files from a .war into some directory.
We have an .xml file which includes commands to extract the files from the .war file and start the server.
The build is being created properly but the .war extraction is not happening and the destination folder is being left empty. The Jenkins Console shows the following output:
[sshexec] bash: jar: command not found
As read from other answers, I have set the PATH properly in .bashrc or in .profile but I'm still facing the same issue.
Install Java devel as follows:
yum install java-devel
To be specific, JAVA bin directory is not in your PATH variable. Add it to PATH variable. In order to search the executables, the OS need to have a list of directories to look up. So, Add the directory which contains 'jartool in thePATH` environment variable
Note : For Windows, Path separator is (;) and for Unix-like OS, Path separator is (:)
The "[sshexec]" tells me you are using some Jenkins plugin to execute a command on a remote host.
This usually means Jenkins executes this (or an equivalent thing by implementing the SSH protocol natively):
ssh user#remotehost 'jar arg1 arg2...'
What happens here is the sshd daemon on the remote host will execute the default shell of the user and ask the shell to execute the command line. The shell is typically not executed as a "login shell", meaning the shell will skip the initialization steps that login shells normally do when the user logs in interactively.
You can study man (name of your shell) to see exactly what the difference is between a shell executed with -l option and one executed without it. In the case of bash it essentially means .profile and .bashrc will not be executed, so any PATH modifications you might have there will not take place.
I think your best bet is to provide the full path to the jar command when you want to execute it.
As you are executing the command on remote host so, it will invoke the new Shell on the remote host. The PATH which you have set in the current host .bashrc file will not make any difference. Instead if you set the PATH for Java in remote host's .bashrc things should work else if you are running any script then mention the complete path for JAR like /usr/jdk64/jdk1.8.0_40/bin/jar -tf /tmp/jars/abc.jar while invoking the command on remote host.
Two ways to do:
export PATH=$PATH:/usr/jdk64/jdk1.8.0_40/bin ---> in remote host's .bashrc
/usr/jdk64/jdk1.8.0_40/bin/jar -tf /tmp/jars/abc.jar --> invoke the jar command with complete path for JAR from remote host.
Per the installation directions I run:
./bin/sqoop.sh client
and it returns error:
could not find or load main class org.apache.sqoop.shell.SqoopShell
I'm setting up Sqoop for the first time. I have the server up and running on Redhat Linux. Now I am stuck running the client on Windows Server 2012. I am running hadoop v2 and sqoop-1.99.3-bin-hadoop200. At the command line I verified java is installed by running "java -version" at the command line and seeing "1.8.0". Scoop is written in Java 1.6. Does that matter? I'm having difficulty locating an active download link to 1.6 for windows for troubleshooting.
I tried installing Cygwin and running the command. Same error.
Locate the directory traversal for the case of client in sqoop.sh
# Build class path with full path to each library
for f in $SQOOP_CLIENT_LIB/*.jar; do
CLASSPATH="${CLASSPATH}:$f"
done
Just add the following search and replace params (note changes of colon to semicolon in classpath)
# Build class path with full path to each library
PARAMSEARCH="/c/"
PARAMREPLACE="C:\\"
for f in $SQOOP_CLIENT_LIB/*.jar; do
f=${f/$PARAMSEARCH/$PARAMREPLACE}
CLASSPATH="${CLASSPATH};$f"
done
I had the same issue running on Windows 8.1. Turns out it's caused because sqoop.sh uses POSIX style directory paths. So / instead of . This results in the script being unable to set the correct CLASSPATH. This could have been remedied by using a cmd file for Windows. I don't know why they didn't do it. There are cmd files for catalina and such but not for launching sqoop!
Anyway I got around this issue by adding all the jar files in the shell\lib folder to CLASSPATH. You do have to add each & every jar file there and not just the folder path. A restart was required after this which was annoying. However the shell script will still not launch the client even after this change. The little trick I used is to just launch the client at the command prompt from the shell\lib folder with the following command
***java org.apache.sqoop.shell.SqoopShell***
Now I get the groovy prompt from where I can run all the commands provided here
I used the following steps to configure sqoop in a windows machine and now able to work with sqoop 1.99.3 successfully.
*Cygwin must be installed before continuing further steps. Also its just for using the native libraries, we can interact with sqoop directly from windows command prompt.
Save the sqoop package folder in “C:” drive name as sqoop
Copy the hadoop lib folder “hadoop” from C:\hadoop\share into C:\usr\lib\hadoop [If this directory doesn't exists, then create one]
Change the hadoop configuration properties in the file “sqoop.properties” from the location “C:\sqoop\server\conf” as below
at line:132
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=C:\hadoop\etc\hadoop
Copy the sqoop.properties and sqoop_bootstrap.properties from “C:\sqoop\server\conf” into “C:\sqoop\server\bin”.
Set the environment variables for sqoop home as well as path:
SQOOP_HOME = C:\sqoop
Starting the server:
Open Command prompt (run as Administrator) and enter into the $SQOOP_HOME location and type command as follows
C:\sqoop> cd server\bin
C:\sqoop\server\bin> catalina start
The catalina server will be start and check it in the below location
http://localhost:12000/sqoop
Client Shell Command
Open the command prompt and enter the following command
java -classpath C:\sqoop\shell\lib\sqoop-shell-2.0.0-SNAPSHOT.jar;C:\sqoop\shell\lib\sqoop-common-2.0.0-SNAPSHOT.jar;C:\sqoop\shell\lib\sqoop-client-2.0.0-SNAPSHOT.jar;C:\sqoop\shell\lib\log4j-1.2.16.jar;C:\sqoop\shell\lib\json-simple-1.1.jar;C:\sqoop\shell\lib\jline-0.9.94.jar;C:\sqoop\shell\lib\jersey-core-1.11.jar;C:\sqoop\shell\lib\jersey-client-1.11.jar;C:\sqoop\shell\lib\jansi-1.7.jar;C:\sqoop\shell\lib\hamcrest-core-1.3.jar;C:\sqoop\shell\lib\groovy-all-1.8.5.jar;C:\sqoop\shell\lib\commons-lang-2.6.jar;C:\sqoop\shell\lib\commons-cli-1.2.jar org.apache.sqoop.shell.SqoopShell
The client shell command will be displayed as :
Sqoop Shell: Type 'help' or '\h' for help.
sqoop:000>
Hope this helps
Thanks