We Keep Coding

Why java action System.out disappear when executed in Oozie?

I'm using oozie environment. After successfully completition of the job, I can't find System.out.println output in the oozie log. I googled for many hours and i found this
but without result. From oozie web console i got the job id "0000011-180801114827014-oozie-oozi-W", then i tried to get more information about the job using the following command:
oozie job -oozie http://localhost:11000/oozie/ -info 0000011-180801114827014-oozie-oozi-W
then i get the externalId from action JobCompleted "16546" and i think the job id 180801114827014. Finally i tried to get log from java action using the following command:
yarn logs -applicationId application_180801114827014_16546
Where I'm doing wrong? Any suggestion?
Edit
I check if log aggregation was enaled and seems that it is enabled
Then, where I'm a doing wrong?

I can say from experience that stdout is not removed from any YARN action, however, the encouraged way to log information in your applications is using Log4j which goes to syslog, not stdout (or stderr).
However, as your terminal says, YARN log aggregation needs enabled / completed for you to see the logs from the yarn logs command
And if that command doesn't work otherwise, go to the Oozie UI, to the job action, or directly to the YARN UI and search for the action, then find the logs link from there

Can not run program, permission denied erro13 Jenkins

I created a new job for android new project on Jenkin. Whenever I run the job I got error java.io.ioexception error 13 permission denied. Jenkin says unable to run program gradlew. I know gradlew script need execution permissions according to the error explanation . I granted these and re run the jenkin job. I still get the same error. Jenkin revert the execution permission back after build. When I create new job from existing job and configured with old projects repository , it runs fine. when I configured with new project repository it raised permission issues. I played a lot with permission but no success. I also compared old and new project script file. There was few lines difference but it shouldn't be an issue. Any one can guide me what I am doing wrong.
Thanks

It is clearly a permission issue.
java.io.IOException: Cannot run program "<http://jenkins.gradlew"
error=13, Permission denied
Caused by: java.io.IOException: error=13, Permission denied
There are two solutions to resolve the gradlew permission issue.
Go to Jenkin Job configuration
Go to Build tab and check gradlew executable.
Jenkin will change the permission when you press build now. Keep in mind that these changes are uncommitted. The process works fine if your job is not a release job. If it is release job it creates another issue that I came across.
:workspace:app:checkCommitNeeded FAILED
:app:release FAILED
* What went wrong:
Execution failed for task ':workspace:app:checkCommitNeeded'.
> You have uncommitted files:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
M gradlew
So work around is
commit gradlew with executable bit set:
git update-index --chmod=+x gradlew
git commit
Then you will no longer need the jenkins setting to set it executable, which is the workaround causing the 2nd issue.
Hopefully it would save someone time because I spent hours for the work around.

I was using Github for source code management. It was fetching code there and was updating it every single time, so changing it to +x before submitting my Jenkins job didn't help.
In Linux the project is at /.jenkins/workspace/MyProjectName/SomeSubFolder/gradlew - and it was loosing +x evereytime a new job was submitted (to 644, I think).
The solution was to select the Make gradlew executable checkbox (in Build --> Use Gradle Wrapper) - it is 755 now, and is executing gradle tasks.

Are you using a jenkinsfile for your job configuration?
If so you need to add the permission change for gradlew there.
sh 'chmod 755 ./gradlew'
It sounds like your gradlew is being replaced each time with a version that doesn't have the execute permission set so you will need to do it as part of the jenkins job either via a script or the jenkinsfile.

Jenkins Cant Connect to Master

I have a setup up a master and slave on the same machine.
The master monitors External Jobs the slave runs a batch script and notifies the master about the same.
I followed the Windows instructions from:
https://wiki.jenkins-ci.org/display/JENKINS/Monitoring+external+jobs
and setup a buld step on the slave as follows:
set JENKINS_HOME=http://localhost:8080/jenkins/
java -jar C:\apache-tomcat-7.0.56\webapps\jenkins\WEB-INF\lib\jenkins-core-1.624.jar "POC_Main_Ext_Job" jenkins_poc_test_1
On building the slave job, I get the following error message in the console output:
Building remotely on MySlave in workspace D:\Temp\Jenkins\workspace\Slave_FreeStyle_1
[Slave_FreeStyle_1] $ cmd /c call C:\Users\ROHIT~1.BIS\AppData\Local\Temp\hudson853358493293228093.bat
D:\Temp\Jenkins\workspace\Slave_FreeStyle_1>set JENKINS_HOME=http://localhost:8080/jenkins/
D:\Temp\Jenkins\workspace\Slave_FreeStyle_1>java -jar C:\apache-tomcat-7.0.56\webapps\jenkins\WEB-INF\lib\jenkins-core-1.624.jar "POC_Main_Ext_Job" jenkins_poc_test_1
http://localhost:8080/jenkins/job/POC_Main_Ext_Job/ is not a valid external job (404 Not Found)
D:\Temp\Jenkins\workspace\Slave_FreeStyle_1>exit -1
Build step 'Execute Windows batch command' marked build as failure
Finished: FAILURE
However, the above url does work in Jenkins and takes to the corresponding job. Any Idea what I'm doing wrong?

As you can see I had missed out the user credentials while setting JENKINS_HOME.
After adding them it works smoothly.

Running MapReduce job written in Java through my PHP web page

My PHP server is hosted on Job Tracker machine and I am trying to run the map reduce job through my web page by calling the command line executing the jar command,
but I am getting no response and job is not starting.
However if I run a command to list the hdfs using same methodology it is running fine. Please guide me.
Following command is not responding me anything and job is not running:
exec("HADOOP_DIR/bin/hadoop jar /usr/local/MapReduce.jar Mapreduce [input Path] [output Path]");
But if I do this:
exec("HADOOP_DIR/bin/hadoop dfs -ls /user/hadoop");
It is running fine.

I solved this problem by changing the php server user to hduser (user which has permission to write files in hdfs). without changing this user only the commands which reads from the hdfs were working and not the one which needs to create the files or write on hdfs.
When i tried to run the command for creating the directory in hdfs through my php script, I got the following error in my php server logs (/var/log/apache2/error.log):
mkdir: org.apache.hadoop.security.AccessControlException: Permission denied: user=www-data, access=WRITE, inode="hduser":hduser:supergroup:rwxr-xr-x
And on running the Jar command to trigger MapRed program I got the following error:
Exception in thread "main" java.io.IOException: Permission denied
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:1879)
at org.apache.hadoop.util.RunJar.main(RunJar.java:115)
Then what i did is i changed the user in /etc/apache2/apache2.conf to my hadoop user and then restarted my server and every thing was working fine now.
I should reference Execute hadoop jar from PHP Server fails. Permission denied post which helped me alot in solving this problem. I hope this post helps others too.

Monitoring a batch file with java

I'm calling a batch that calls another "jar" that send messages to a server and write a report in the end, the time of execution varies from day to day, and the size of the input used in the batch influences to.
I would like to monitor when the batch auto-closes so I can make my original jar to read the log...
I'm calling it like this
Process prog = Runtime.getRuntime().exec("cmd /c start C:\\chamados\\corretorRota\\VerificarNumero.bat");
and the batch:
cd C:\chamados\corretorRota
java -jar BatchDispatcher.jar brux0043 5873 gcpn-rota.txt > resultado.txt
exit
(FYI brux0043 = server, 5873 = port gcpn-rota = input file writed previously)
(the batch keeps open util the end of the called jar)

You can write a log with the timestamp whenever the batch file was executed.
put this line in your batch file
echo. |time |find "current" >> log
this will then write the timestamp when the batch file was run to the log. Also, there should be a file called log before executing the batch file.
Now your java program can monitor the log and know how many times and when the batch file was executed.

You should use
prog.waitFor()
to wait for the process to finish execution.