My PHP server is hosted on Job Tracker machine and I am trying to run the map reduce job through my web page by calling the command line executing the jar command,
but I am getting no response and job is not starting.
However if I run a command to list the hdfs using same methodology it is running fine. Please guide me.
Following command is not responding me anything and job is not running:
exec("HADOOP_DIR/bin/hadoop jar /usr/local/MapReduce.jar Mapreduce [input Path] [output Path]");
But if I do this:
exec("HADOOP_DIR/bin/hadoop dfs -ls /user/hadoop");
It is running fine.
I solved this problem by changing the php server user to hduser (user which has permission to write files in hdfs). without changing this user only the commands which reads from the hdfs were working and not the one which needs to create the files or write on hdfs.
When i tried to run the command for creating the directory in hdfs through my php script, I got the following error in my php server logs (/var/log/apache2/error.log):
mkdir: org.apache.hadoop.security.AccessControlException: Permission denied: user=www-data, access=WRITE, inode="hduser":hduser:supergroup:rwxr-xr-x
And on running the Jar command to trigger MapRed program I got the following error:
Exception in thread "main" java.io.IOException: Permission denied
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:1879)
at org.apache.hadoop.util.RunJar.main(RunJar.java:115)
Then what i did is i changed the user in /etc/apache2/apache2.conf to my hadoop user and then restarted my server and every thing was working fine now.
I should reference Execute hadoop jar from PHP Server fails. Permission denied post which helped me alot in solving this problem. I hope this post helps others too.
Related
So I was trying to make a 1.17.1 minecraft server on my mac. I couldn't open my 1.17.1_server.jar with Java 8 so I download Java 16.0.2.
Unfortunately, everytime I was opening the 1.17.1_server.jar file, I got
"The Java JAR file "1.17._server.jar" could not be launched." .
I first thought that it was because the file was launch by Java 8 instead of 16.
So I went into the terminal and run :<path to java> -jar 1.17.1_server.jar
I then got this : Error: Unable to access jarfile 1.17.1_server.jar
Finally i tried to put the path of the jar file in the command...
So I've run : path to java -jar path to server
and got this :
[main/ERROR]: Failed to load properties from file: server.properties
[15:57:35] [main/WARN]: Failed to load eula.txt
[15:57:35] [main/INFO]: You need to agree to the EULA in order to run the server. Go to eula.txt for more info.
So why I have to agreed Eula if i've never launched it ? Does it think that he already been launched ?
As stated in the error message
[15:57:35] [main/INFO]: You need to agree to the EULA in order to run the server. Go to eula.txt for more info.
You have to open the file and check yes
Okay find the solution: my eula and server properties file were in my user folder dk why (never moved them so...)
I have a java program ( eclipse , Maven , windows7) to fill Hbase which exist on a VirtualBox from a service,
the program works fine when I run it from eclipse.
But when i created an executable jar using the assembly plugin in Maven and run it from the cmd or from cygwin I get this error
> [2016-05-03 14:46:44,663][DEBUG] Reading reply sessionid:0x154769ed563000a, packet:: clientPath:null serverPath:null finished:false header:: 300,4 replyHeader:: 300,3632,-101 request:: '/hbase-unsecure/meta-region-server,F response:: [2016-05-03 14:46:44,663][DEBUG] hconnection-0x6b63f5ff-0x154769ed563000a, quorum=sandbox:2181, baseZNode=/hbase-unsecure Unable to get data of znode /hbase-unsecure/meta-region-server because node does not exist (not an error) [2016-05-03 14:46:44,663][DEBUG] Looked up meta region location, connection=org.apache.hadoop.hbase.client.ZooKeeperRegistry#695e5335; servers = null [2016-05-03 14:46:44,689][DEBUG] Closing scanner id=-1
>org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:312)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:811)
at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:303)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:313)
Could any of you help me with this?
Some more information like the following may be helpful. Run these commands in the virtualbox where you have the "hbase" command
hbase zkCli -server sandbox:2181
Once you connect, you can run, in the zkcli prompt:
ls /
Please paste the output of the above command. The baseZnode of "/hbase-unsecure " does not seem to exist. In general the base znode is "/hbase".
Kindly cross check the classpath of your running program by the java code, as suggested by #RamPrasadG
System.out.println(System.getProperty("java.classpath"))
I am trying to copy the file from HDFS to Local linux file system using Hadoop FileSystem class.
I have access to create folder in the path where i am trying to copy, i checked using mkdir command.
Also i tried using shell command hadoop fs -copyToLocal hdfsFilePath localFilepath it was working.
I am running this on YARN Cluster.
I tried below approaches, but i am getting the java.io.IOException: Mkdirs failed to create file:/home/user error.
Error log:
16/01/14 01:09:36 ERROR util.FileUtil:
java.io.IOException: Mkdirs failed to create /home/user (exists=false, cwd=file:/hdfs4/yarn/nm/usercache/user/appcache/application_1452126203792_8862/container_e2457_1452126203792_8862_01_000001)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:442)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:428)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:365)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1970)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1939)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1915)
at com.batch.util.FileUtil.copyToLocalFileSystem(FileUtil.java:66)
at com.batch.dao.impl.DaoImpl.writeFile(DaoImpl.java:108)
at com.batch.JobDriver.runJob(JobDriver.java:79)
at com.batch.JobDriver.main(JobDriver.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:480)
Actually i am passing localFilePath as /home/user/test, but i am getting the error like failed to create file:/home/user
fs.copyToLocalFile(hdfsFilePath, localFilePath);
fs.copyToLocalFile(false, hdfsFilePath, localFilePath, true);
This week i faced the same thing, problem was that i was deploying the job in cluster mode, therefore the machine where the job was going to run did not have that directory created. Is it possible you are deploying the job in cluster mode? If so, try deploying it in client mode (the output directory has to exist though)
For anyone looking for this exact error, but maybe not from YARN:
I had this exact error when trying to run org.apache.hadoop.fs.FileSystem.copyToLocalFile on my local (Mac) machine, with local FS configured using the job.local.dir attribute.
This was the exception:
java.io.IOException: Mkdirs failed to create file:/User/yossiv/algo-resources/AWS/QuerySearchEngine.blacklistVersionFile (exists=false, cwd=file:/Users/yossiv/git/c2s-algo)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:456)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:441)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:928)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:806)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:368)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:341)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2066)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2035)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2011)
What fixed it was to change job.local.dir to be under the current directory , which is listed in the exception text after cwd=, In my case that's /Users/yossiv/git/c2s-algo.
Broke my head two days over this, hope this helps someone.
I have a setup up a master and slave on the same machine.
The master monitors External Jobs the slave runs a batch script and notifies the master about the same.
I followed the Windows instructions from:
https://wiki.jenkins-ci.org/display/JENKINS/Monitoring+external+jobs
and setup a buld step on the slave as follows:
set JENKINS_HOME=http://localhost:8080/jenkins/
java -jar C:\apache-tomcat-7.0.56\webapps\jenkins\WEB-INF\lib\jenkins-core-1.624.jar "POC_Main_Ext_Job" jenkins_poc_test_1
On building the slave job, I get the following error message in the console output:
Building remotely on MySlave in workspace D:\Temp\Jenkins\workspace\Slave_FreeStyle_1
[Slave_FreeStyle_1] $ cmd /c call C:\Users\ROHIT~1.BIS\AppData\Local\Temp\hudson853358493293228093.bat
D:\Temp\Jenkins\workspace\Slave_FreeStyle_1>set JENKINS_HOME=http://localhost:8080/jenkins/
D:\Temp\Jenkins\workspace\Slave_FreeStyle_1>java -jar C:\apache-tomcat-7.0.56\webapps\jenkins\WEB-INF\lib\jenkins-core-1.624.jar "POC_Main_Ext_Job" jenkins_poc_test_1
http://localhost:8080/jenkins/job/POC_Main_Ext_Job/ is not a valid external job (404 Not Found)
D:\Temp\Jenkins\workspace\Slave_FreeStyle_1>exit -1
Build step 'Execute Windows batch command' marked build as failure
Finished: FAILURE
However, the above url does work in Jenkins and takes to the corresponding job. Any Idea what I'm doing wrong?
As you can see I had missed out the user credentials while setting JENKINS_HOME.
After adding them it works smoothly.
Is there a way to simply run a python script in Apache Storm?
I'm trying to figure out how to use storm to run scripts but am having trouble. It seems like I need to create a Java program to call the script and use it as a bolt but I simply want to send a very basic python script to storm to see if it is possible.
I read that the following command is helpful in sending topologies to storm but am having trouble understanding the syntax and if I am allowed to send any python code to storm or if it needs to have specific syntax.
Can someone clarify whether or not I can submit any python script to storm and if so what the following line of code means.
storm shell resources/ python topology.py arg1 arg2
When I try to submit a basic python script using the above code i get the following output.
956 [main] INFO backtype.storm.StormSubmitter - Uploading topology jar stormshell8691441.jar to assigned location: /home/scix3/apache/storm/data/nimbus/inbox/stormjar-ae0739f9-7c93-4f00-a02b-c4eceba3b005.jar
966 [main] INFO backtype.storm.StormSubmitter - Successfully uploaded topology jar to assigned location: /home/scix3/apache/storm/data/nimbus/inbox/stormjar-ae0739f9-7c93-4f00-a02b-c4eceba3b005.jar
Exception in thread "main" java.io.IOException: Cannot run program "simple.py" (in directory "."): error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
at java.lang.Runtime.exec(Runtime.java:617)
at org.apache.commons.exec.launcher.Java13CommandLauncher.exec(Java13CommandLauncher.java:58)
at org.apache.commons.exec.DefaultExecutor.launch(DefaultExecutor.java:254)
at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:319)
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:160)
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:147)
at backtype.storm.util$exec_command_BANG_.invoke(util.clj:386)
at backtype.storm.command.shell_submission$_main.doInvoke(shell_submission.clj:29)
at clojure.lang.RestFn.applyTo(RestFn.java:139)
at backtype.storm.command.shell_submission.main(Unknown Source)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:186)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
... 10 more
The exact command I'm using (possibly incorrect) is storm shell resources/ simple.py
simple.py is merely a print 'Hello, world script.
I'm using storm version 0.9.4
Yes, you can run python on Storm. In fact you can run just about code from just about any language on a storm cluster, its just a matter of implementing the API.
However, there are some requirements for that to work, and so far as I can tell said requirements are not spelled out in the storm documentation. The fastest path to get up an running would be to take the splitsentence.py example from the storm source and run with it.
try pyleus (https://github.com/Yelp/pyleus) or streamparse (https://github.com/Parsely/streamparse), i will recommend using pyleus as it is simple.