We Keep Coding

Stop Java from killing Bash script that started it

I've spent the past couple days working on this, and at this point I am super stuck. I have a Java program that must be run not as a service. This program must also be capable of updating itself when a new file is given for updating.
As a result, I have a script that is started with Linux that starts the Java application and then checks every 5 seconds if the application has been terminated. If the application has been terminated, it should check if there is an update and then start appropriately.
This is the code for that script:
#!/bin/bash
JAVA_HOME=/usr/lib/jvm/java-16-openjdk-amd64
WORKING_DIR=~/Data
LOG=$WORKING_DIR/logs/Bash.log
rm $LOG
echo "Script started" > $LOG
while true; do
source $WORKING_DIR/Server.pid
if ! kill -0 $AppPID; then
echo "App must be started" >> $LOG
source $WORKING_DIR/UpdateStatus
if [ "$UpdateReady" -eq "1" ]; then
echo "Moving files for update" >> $LOG
mv $WORKING_DIR/Server.jar $WORKING_DIR/old.jar
mv $WORKING_DIR/new.jar $WORKING_DIR/Server.jar
fi
nohup ${JAVA_HOME}/bin/java -jar ${WORKING_DIR}/Server.jar &
echo AppPID="$!" > $WORKING_DIR/Server.pid
echo "Server started" >> $LOG
if [ "$UpdateReady" -eq "1" ]; then
echo "Checking for safe update" >> $LOG
source $WORKING_DIR/Server.pid
echo UpdateReady="0" > $WORKING_DIR/UpdateStatus
sleep 5;
if kill -0 $AppPID; then
echo "Update successful" >> $LOG
rm $WORKING_DIR/old.jar
else
echo "Update failed, restarting old jar" >> $LOG
rm $WORKING_DIR/Server.jar
mv $WORKING_DIR/old.jar $WORKING_DIR/Server.jar
nohup ${JAVA_HOME}/bin/java -jar ${WORKING_DIR}/Server.jar &
echo AppPID="$!" > $WORKING_DIR/Server.pid
fi
fi
echo "Server start process finished, going into idle state" >> $LOG
fi
sleep 5
echo "5 seconds idle passed" >> $LOG
done
To initialize the update, I have tried a couple of different things, both with the same result. First I had set UpdateReady="1" through Java, then used exit(0);. I have also tried having Java call a Bash script which also sets UpdateReady="1" but uses kill $AppPID to shutdown the java application.
The result is that both the Java application and the Bash script stop executing causing the update and restart to fail! I have looked through a significant amount of Stack Overflow questions and answers finding things such as nohup, all to no avail.
I will once again state that the Java application cannot be run as a service. No packages other than what is included in Java or made by Apache can be used, and no programs can be installed to Linux. I would prefer to solve the problem with Bash.
Upon testing some things mentioned in comments, I may have missed something that turns out to be important. While all other runs of the startup script will be run by the startup applications manager, the initial run is not.
The install is taken care of remotely with an SSH connection sending the following command string:
cd /home/UserName; unzip -u -o Server.zip; chmod 777 install.sh; bash install.sh &; exit
install.sh is as follows:
#!/bin/bash
INSTALL_DIR=~/Data
mkdir ${INSTALL_DIR}
mkdir ${INSTALL_DIR}/logs
mkdir ${INSTALL_DIR}/data
cp Server.jar ${INSTALL_DIR}/Server.jar
cp service-start.sh ${INSTALL_DIR}/service-start.sh
chmod 777 ${INSTALL_DIR}/service-start.sh
rm Server.jar
rm service-start.sh
rm Server.zip
nohup bash $INSTALL_DIR/service-start.sh &
Upon rebooting my machine, I noticed that this problem goes away! This means that there must be a problem with the initial setup script. When the shell command is run, it does seem to be sticky and not actually let go after the bash install.sh &. I have tried putting nohup at the beginning of this, however the entire line will not run in that case for reasons I am not able to determine.
I would prefer to not have the user be forced to restart after install and can't seem to find any way to force the Startup Application manager to start an application at any time other than startup.

Well, after a lot of searching and some prompting from the comments, I found that the issue lied with how the program was initially being started.
As mentioned in the update, the first run is always started by an ssh connection. I knew there was a slight problem with this ssh connection, as it seemed to hold onto the connection no matter what I did. It turns out that this was causing the problem that resulted in the Bash instance and the Java instance remaining attached.
The solution for this problem was found here: jsch ChannelExec run a .sh script with nohup "lose" some commands
After managing to get the initial setup to start with nohup properly, the issue has gone away.

Git Hook PHP Script: No Output

I have this Script php-pull-script.phpwritten:
<?php
$output1 = shell_exec('git pull');
$output2 = shell_exec('pkill java');
$output3 = shell_exec('mvn package');
$output4 = shell_exec('java -jar ./target/compute-0.0.1-SNAPSHOT.jar');
echo "<pre>$output1</pre>";
echo "<pre>$output2</pre>";
echo "<pre>$output3</pre>";
echo "<pre>$output4</pre>";
?>
When executing in shell, I am not seeing any output in order to verify that it is running. I am not sure at all it is working. Is there a better way to do this automation script?

How to send of the java command with shell_exec and leave it running in background (Is & possible with shell exec)?
test.php
<?php
shell_exec('test.sh');
test.sh
echo "Do something"
/bin/sh -c 'sleep 10' >> /dev/null 2>&1 &
exit 0
/dev/null can be also a path to an logfile
All pathes here and in phpscript should be absolute, like: /path/tomy/test.sh
Here the test.php doesnt wait 10 seconds for the subcall.
Hope that helps a little ;)

You might not be in the correct working directory. You may need to set that manually to make the commands run. Otherwise, those commands look right. Though you really should just have this set up as a shell script. PHP's not a good language for this sort of thing, and if you are calling this from a REST endpoint there are far better solutions like Jenkins.

Unable to run nohup command from jenkins as a background process

UPDATE: Based on below discussion I have edited my answer for more accurate description.
I am trying to run a nohup command from jenkins. The full command is
nohup java -jar /home/.../jar/server-process-0.35.jar prod >> /var/../server-process-prod.log 2>&1 &
This command does not work. I can see status as success in jenkins but no java process in linux. When I do 'ps -ef | grep java'
However when I remove the last '&' , that is I change it from run in forground instead of background
It starts working. I can see the java process started.
The original command works fine If I run it on linux console.
I need to run it from jenkins in the original form that is as a backgorund process. So that it is independant of jenkins.
Any clues why is this happening?

Long story short, Jenkins kills all processes spawned by a job once that job finishes. To override this behavior, you need to set an environment variable.
The variable appears to vary from job type to job type. It used to be BUILD_ID, but for Pipeline jobs it is JENKINS_NODE_COOKIE, and there are several others mentioned in this answer.
So if you're running your command in Pipeline, it would look like this:
sh 'JENKINS_NODE_COOKIE=dontKillMe nohup java -jar /home/.../jar/server-process-0.35.jar prod >> /var/../server-process-prod.log 2>&1 &'
See the wiki on ProcessTreeKiller and this comment in the Jenkins Jira for more information.

In your jenkins shell script try:
export BUILD_ID=dontKillMe
nohup java -jar your_java_app.jar &
It worked for me!

I tried every possible combination with BUILD_ID but it didn't work.
I made it though by putting "nohup command > output.txt&" inside a shell script ran by the execute shell in jenkins, it worked perfectly!

Got the same problem, added:
BUILD_ID=dontKillMe python /var/lib/jenkins/release.py
into Execute Shell -> Command and inside release.py there is:
os.system('nohup java -jar ' + new_jars_on_server + '/' + generated_jar_by_mvn_name + '&')
and it works

Best simple solution is to use "at now" instead of "nohup"
In your job jenkins (execute shell) put :
set +e #so "at now" will run even if java -jar fails
#Run java app in background
echo "java -jar $(ls | grep *.jar | head -n 1)" | at now + 1 min

what worked for me was wrapping the nohup java -jar ... command into sh file inside execute shell command, and running that same sh file right after:
echo "starting java jar..."
cd [some location where jar is]
echo "nohup java -jar [jar_name].jar &" > start-jar-in-background.sh
sh start-jar-in-background.sh
echo "started java jar"
If I had nohup java -jar ... inline with Execute shell command, then it didn't start it from some reasons. I spent quite some time on this, hope it helps to someone ';)

Simplest way :
`nohup java -jar [jar_name].jar >log_file_you_want 2>another_file`&

set +e #so "at now" will run even if java -jar fails
#Run java app in background
echo "java -jar $(ls | grep *.jar | head -n 1)" | at now + 1 min
above command worked for him, thanks #walid, & remove at the end (+ 1 min)

HOME environment variable not set in bash script

I have a Java program that spawns a bash script that calls another script. In that second script, I'm finding that the $HOME variable is not set. Here's the gist of it:
In Java:
Process p = new ProcessBuilder("bash", "-l", "/foo/a.sh").start();
// code to actually execute p
/foo/a.sh:
#!/bin/bash
/foo/b.sh
/foo/b.sh:
#!/bin/bash
echo "HOME=$HOME"
This echoes "HOME=". The eventual problem is that $HOME/bin is supposed to be added to my PATH in ~/.profile, but since that's not happening, a bunch of custom executables aren't being made accessible.
I worked around it by doing:
if [ -d ~/bin ] ; then
PATH=~/bin:"$PATH"
fi
And that works fine. But I guess I just want to understand why $HOME wasn't being set. It seems like $HOME and ~ should be largely equivalent, no? There's probably something I'm fundamentally missing about how this environment is getting set up.
I am running Ubuntu 12.04.5, if that makes a difference.

The evidence suggests that HOME is missing from the environment in which the Java app is running. Assuming that the app doesn't explicit unset HOME, the most likely reason is that the app is being started from some context other than a login by the user the app is running as.
It's correct that ~ and $HOME are similar. If HOME is present in the environment, even if it is set to the empty string, ~ will be replaced with $HOME. However, if HOME is not present in the environment, bash will attempt to find the home directory for the currently logged in user, and use that for ~.
eg.
$ bash -c 'echo ~'
/home/rici
$ HOME='Hello, world!' bash -c 'echo ~'
Hello, world!
$ HOME= bash -c 'echo ~'
$ (unset HOME; bash -c 'echo ~';)
/home/rici
Since your workaround requires the equivalent of the last scenario, I conclude that HOME has not been set, or has been unset.

Running a process in background, linux ubuntu

I want to run a .jar file so what I do is that I put "&" at the very end of the command (actually there's no need to log the output , I only want to be able to disconnect from the remote server which hosts my java program, the program itself saves the result after being finished)
I do as it follows , but it doesn't run in the background and it keeps me waiting :
java -Xmx72G -cp //home/arian/rapidminer/lib/rapidminer.jar com.rapidminer.RapidMinerCommandLine -f //home/arian/RMRepository/testRemote.rmp &
Any idea that why it doesn't work ?
Thanks ,
Arian

I don't know why it wouldn't work. It really should, in most shells.
Anyway, if you intend to disconnect you'll usually find that just putting the job in the background is not enough: the disconnect will close the console (which will break many programs, alone) and send a SIGHUP signal (which will cause just about any program to exit).
You should considered using nohup to run the program (with the &). Alternatively, if you ever do need to come back and interact with the program later then screen or byobu might fit the bill better. Yet another alternative might be to add the task to your crontab.

What do you mean by "it keeps me waiting"? Does the RapidMinerCommandLineby any chance read from stdin or another stream?
If you want to run a process in the background and disconnect from the tty session you should use nohup, eg.:
nohup java -Xmx.... com.rapidminer.RapidMinerCommandLine &
(Do remember the & at the end!)
You may add ... 1> /dev/null before the & to disregard all stdout.
You could also consider the screen utility, which allows you to dis- and reconnect to the session, but that's more usable with (semi-)interactive sessions.
(Also, quite a hefty max heap size you're specifying?)
Cheers,

You can use JSVC, this is an utility interresting to daemonize Java applications
http://commons.apache.org/daemon/jsvc.html
It will give you a var pid file, useful to create a real start/stop script.
EDIT : Other solution, maybe could help
Here is a very old start/stop script I've done for Slackware Linux on embedded systems :
#!/bin/sh
application_start() {
cd /usr/local/YOURHOME
/usr/lib/java/bin/java \
-Xmx72G \
-classpath //home/arian/rapidminer/lib/rapidminer.jar \
com.rapidminer.RapidMinerCommandLine \
-f //home/arian/RMRepository/testRemote.rmp &
echo -n "Starting App daemon: $CMDLINE"
ps -Ao pid,command | grep java | grep com.rapidminer.RapidMinerCommandLine | awk '{print $1}' > /var/run/app.pid
echo
}
application_stop() {
echo -n "Stopping DataBaseSynchronizerClient daemon..."
kill `cat /var/run/DataBaseSynchronizerClient.pid`
echo
sleep 1
rm -f /var/run/DataBaseSynchronizerClient.pid
killall DataBaseSynchronizerClient 2> /dev/null
}
application_restart() {
application_stop
sleep 1
application_start
}
case "$1" in
'start')
application_start
;;
'stop')
application_stop
;;
'restart')
application_restart
;;
*)
echo "usage $0 start|stop|restart"
esac

I agree it should work as it is, but I had problems with Java running in background too. My solution was to use the screen utility (which normally is installed in most Linux distributions) where you can open a shell from which you can detach. If I remember well the commands are something like this (but there is a good manpage too)
screen -S myCustomName # runs a new shell called myCustomName
CTRL + D # detach from the current screen instance
screen -ls # list active screen instances
screen -r myCustomName # reattach to the screen instance.
Hope it will solve your problem.