We Keep Coding

Stop Java from killing Bash script that started it

I've spent the past couple days working on this, and at this point I am super stuck. I have a Java program that must be run not as a service. This program must also be capable of updating itself when a new file is given for updating.
As a result, I have a script that is started with Linux that starts the Java application and then checks every 5 seconds if the application has been terminated. If the application has been terminated, it should check if there is an update and then start appropriately.
This is the code for that script:
#!/bin/bash
JAVA_HOME=/usr/lib/jvm/java-16-openjdk-amd64
WORKING_DIR=~/Data
LOG=$WORKING_DIR/logs/Bash.log
rm $LOG
echo "Script started" > $LOG
while true; do
source $WORKING_DIR/Server.pid
if ! kill -0 $AppPID; then
echo "App must be started" >> $LOG
source $WORKING_DIR/UpdateStatus
if [ "$UpdateReady" -eq "1" ]; then
echo "Moving files for update" >> $LOG
mv $WORKING_DIR/Server.jar $WORKING_DIR/old.jar
mv $WORKING_DIR/new.jar $WORKING_DIR/Server.jar
fi
nohup ${JAVA_HOME}/bin/java -jar ${WORKING_DIR}/Server.jar &
echo AppPID="$!" > $WORKING_DIR/Server.pid
echo "Server started" >> $LOG
if [ "$UpdateReady" -eq "1" ]; then
echo "Checking for safe update" >> $LOG
source $WORKING_DIR/Server.pid
echo UpdateReady="0" > $WORKING_DIR/UpdateStatus
sleep 5;
if kill -0 $AppPID; then
echo "Update successful" >> $LOG
rm $WORKING_DIR/old.jar
else
echo "Update failed, restarting old jar" >> $LOG
rm $WORKING_DIR/Server.jar
mv $WORKING_DIR/old.jar $WORKING_DIR/Server.jar
nohup ${JAVA_HOME}/bin/java -jar ${WORKING_DIR}/Server.jar &
echo AppPID="$!" > $WORKING_DIR/Server.pid
fi
fi
echo "Server start process finished, going into idle state" >> $LOG
fi
sleep 5
echo "5 seconds idle passed" >> $LOG
done
To initialize the update, I have tried a couple of different things, both with the same result. First I had set UpdateReady="1" through Java, then used exit(0);. I have also tried having Java call a Bash script which also sets UpdateReady="1" but uses kill $AppPID to shutdown the java application.
The result is that both the Java application and the Bash script stop executing causing the update and restart to fail! I have looked through a significant amount of Stack Overflow questions and answers finding things such as nohup, all to no avail.
I will once again state that the Java application cannot be run as a service. No packages other than what is included in Java or made by Apache can be used, and no programs can be installed to Linux. I would prefer to solve the problem with Bash.
Upon testing some things mentioned in comments, I may have missed something that turns out to be important. While all other runs of the startup script will be run by the startup applications manager, the initial run is not.
The install is taken care of remotely with an SSH connection sending the following command string:
cd /home/UserName; unzip -u -o Server.zip; chmod 777 install.sh; bash install.sh &; exit
install.sh is as follows:
#!/bin/bash
INSTALL_DIR=~/Data
mkdir ${INSTALL_DIR}
mkdir ${INSTALL_DIR}/logs
mkdir ${INSTALL_DIR}/data
cp Server.jar ${INSTALL_DIR}/Server.jar
cp service-start.sh ${INSTALL_DIR}/service-start.sh
chmod 777 ${INSTALL_DIR}/service-start.sh
rm Server.jar
rm service-start.sh
rm Server.zip
nohup bash $INSTALL_DIR/service-start.sh &
Upon rebooting my machine, I noticed that this problem goes away! This means that there must be a problem with the initial setup script. When the shell command is run, it does seem to be sticky and not actually let go after the bash install.sh &. I have tried putting nohup at the beginning of this, however the entire line will not run in that case for reasons I am not able to determine.
I would prefer to not have the user be forced to restart after install and can't seem to find any way to force the Startup Application manager to start an application at any time other than startup.

Well, after a lot of searching and some prompting from the comments, I found that the issue lied with how the program was initially being started.
As mentioned in the update, the first run is always started by an ssh connection. I knew there was a slight problem with this ssh connection, as it seemed to hold onto the connection no matter what I did. It turns out that this was causing the problem that resulted in the Bash instance and the Java instance remaining attached.
The solution for this problem was found here: jsch ChannelExec run a .sh script with nohup "lose" some commands
After managing to get the initial setup to start with nohup properly, the issue has gone away.

Unable to run nohup command from jenkins as a background process

UPDATE: Based on below discussion I have edited my answer for more accurate description.
I am trying to run a nohup command from jenkins. The full command is
nohup java -jar /home/.../jar/server-process-0.35.jar prod >> /var/../server-process-prod.log 2>&1 &
This command does not work. I can see status as success in jenkins but no java process in linux. When I do 'ps -ef | grep java'
However when I remove the last '&' , that is I change it from run in forground instead of background
It starts working. I can see the java process started.
The original command works fine If I run it on linux console.
I need to run it from jenkins in the original form that is as a backgorund process. So that it is independant of jenkins.
Any clues why is this happening?

Long story short, Jenkins kills all processes spawned by a job once that job finishes. To override this behavior, you need to set an environment variable.
The variable appears to vary from job type to job type. It used to be BUILD_ID, but for Pipeline jobs it is JENKINS_NODE_COOKIE, and there are several others mentioned in this answer.
So if you're running your command in Pipeline, it would look like this:
sh 'JENKINS_NODE_COOKIE=dontKillMe nohup java -jar /home/.../jar/server-process-0.35.jar prod >> /var/../server-process-prod.log 2>&1 &'
See the wiki on ProcessTreeKiller and this comment in the Jenkins Jira for more information.

In your jenkins shell script try:
export BUILD_ID=dontKillMe
nohup java -jar your_java_app.jar &
It worked for me!

I tried every possible combination with BUILD_ID but it didn't work.
I made it though by putting "nohup command > output.txt&" inside a shell script ran by the execute shell in jenkins, it worked perfectly!

Got the same problem, added:
BUILD_ID=dontKillMe python /var/lib/jenkins/release.py
into Execute Shell -> Command and inside release.py there is:
os.system('nohup java -jar ' + new_jars_on_server + '/' + generated_jar_by_mvn_name + '&')
and it works

Best simple solution is to use "at now" instead of "nohup"
In your job jenkins (execute shell) put :
set +e #so "at now" will run even if java -jar fails
#Run java app in background
echo "java -jar $(ls | grep *.jar | head -n 1)" | at now + 1 min

what worked for me was wrapping the nohup java -jar ... command into sh file inside execute shell command, and running that same sh file right after:
echo "starting java jar..."
cd [some location where jar is]
echo "nohup java -jar [jar_name].jar &" > start-jar-in-background.sh
sh start-jar-in-background.sh
echo "started java jar"
If I had nohup java -jar ... inline with Execute shell command, then it didn't start it from some reasons. I spent quite some time on this, hope it helps to someone ';)

Simplest way :
`nohup java -jar [jar_name].jar >log_file_you_want 2>another_file`&

set +e #so "at now" will run even if java -jar fails
#Run java app in background
echo "java -jar $(ls | grep *.jar | head -n 1)" | at now + 1 min
above command worked for him, thanks #walid, & remove at the end (+ 1 min)

Running Scrapy from bash (shell script)

I have developed a web app in Java which uses Scrapy to get some data. To reach that, I invoke a shell script from Java:
Process p = Runtime.getRuntime().exec("sh myPath/myScript.sh");
p.waitFor();
which contains
#!/bin/bash
cd mySpiderPath
echo "We are going tu run scrapy"
scrapy crawl mySpider
echo "done!"
After running it, both "echo" are printed but scrapy does nothing. If I run myScript.sh from shell it works perfectly... I'm confused!
What can I do to try to debug this strange behavior?
EDIT
I have changed myScript.sh to run python version instead of scrapy command, and it doesn't work... so, the conclusion is that is not an "scrapy problem" but it is a bash script problem when it's invoked from Java...any ideas? (if I execute myScript.sh from shell it works fine)
#!/bin/bash
cd mySpiderPath
echo "We are going tu run scrapy"
python --version
echo "done!"

Try changing:
Process p = Runtime.getRuntime().exec("sh myPath/myScript.sh");
to:
Process p = Runtime.getRuntime().exec("bash myPath/myScript.sh");
This will probably run the script with /bin/bash instead of /bin/sh, that often points to a simpler shell.

Executing java program from bash script does not work

I am trying to run a java program (weka) from a bash script. The script takes as arguments an inputfile, an outputfile and the content of file containing the command to run the java program (environment variable $CMD). The script does not work as I wish and informs me that I use an unknown option for java. I tried to echo the command that the program sends to the shell, and the output is exactly the right command. So I assume that the echo output and the command sent to the shell are not the same.
So please tell me: What did I do wrong?
What is the difference between the output I get...
echo "java $(cat $CMD) $in > $out"
...and the command the computer gets?
java "$(cat $CMD)" $in > $out
If more information is needed, please comment!
Edit:
For those familiar with weka (or familiar with java), this is what I want to get, and what is printed to me by echo:
java -cp /usr/share/java/weka.jar weka.filters.supervised.attribute.AttributeSelection -E "weka.attributeSelection.ClassifierSubsetEval -B weka.classifiers.bayes.NaiveBayes -T" -S "weka.attributeSelection.BestFirst -D 1 -N 14" -i /home/aldorado/weka_examples/iris.arff > /home/aldorado/weka_examples/irisselected131113.txt

Add set -x in before the line which causes trouble.
That will make the computer print the command again as it understood it. You will see something like
+ 'java' '-classpath weka.jar name.of.the.main.Class' 'inputFile' > 'outputFile'
Note that quotes which the shell uses to tell you "this was one word / argument for me". It's very useful to notice problems with white space and quoting.
Note that it is very hard to get something like java "$(cat $CMD)" $in > $out working. I suggest to move the java into $CMD. That will allow you to say:
bash "./$CMD" $in > $out
or, if you make the file executable:
"./$CMD" "$in" > $out
Use "$1" in the file $CMD to get a property quoted reference to "$in":
cp="weka.jar"
cp="$cp;other.jar"
cp="$cp;more.jar"
cp="$cp;foo.jar"
java -classpath "$cp" name.of.the.main.Class "$1"

Running a process in background, linux ubuntu

I want to run a .jar file so what I do is that I put "&" at the very end of the command (actually there's no need to log the output , I only want to be able to disconnect from the remote server which hosts my java program, the program itself saves the result after being finished)
I do as it follows , but it doesn't run in the background and it keeps me waiting :
java -Xmx72G -cp //home/arian/rapidminer/lib/rapidminer.jar com.rapidminer.RapidMinerCommandLine -f //home/arian/RMRepository/testRemote.rmp &
Any idea that why it doesn't work ?
Thanks ,
Arian

I don't know why it wouldn't work. It really should, in most shells.
Anyway, if you intend to disconnect you'll usually find that just putting the job in the background is not enough: the disconnect will close the console (which will break many programs, alone) and send a SIGHUP signal (which will cause just about any program to exit).
You should considered using nohup to run the program (with the &). Alternatively, if you ever do need to come back and interact with the program later then screen or byobu might fit the bill better. Yet another alternative might be to add the task to your crontab.

What do you mean by "it keeps me waiting"? Does the RapidMinerCommandLineby any chance read from stdin or another stream?
If you want to run a process in the background and disconnect from the tty session you should use nohup, eg.:
nohup java -Xmx.... com.rapidminer.RapidMinerCommandLine &
(Do remember the & at the end!)
You may add ... 1> /dev/null before the & to disregard all stdout.
You could also consider the screen utility, which allows you to dis- and reconnect to the session, but that's more usable with (semi-)interactive sessions.
(Also, quite a hefty max heap size you're specifying?)
Cheers,

You can use JSVC, this is an utility interresting to daemonize Java applications
http://commons.apache.org/daemon/jsvc.html
It will give you a var pid file, useful to create a real start/stop script.
EDIT : Other solution, maybe could help
Here is a very old start/stop script I've done for Slackware Linux on embedded systems :
#!/bin/sh
application_start() {
cd /usr/local/YOURHOME
/usr/lib/java/bin/java \
-Xmx72G \
-classpath //home/arian/rapidminer/lib/rapidminer.jar \
com.rapidminer.RapidMinerCommandLine \
-f //home/arian/RMRepository/testRemote.rmp &
echo -n "Starting App daemon: $CMDLINE"
ps -Ao pid,command | grep java | grep com.rapidminer.RapidMinerCommandLine | awk '{print $1}' > /var/run/app.pid
echo
}
application_stop() {
echo -n "Stopping DataBaseSynchronizerClient daemon..."
kill `cat /var/run/DataBaseSynchronizerClient.pid`
echo
sleep 1
rm -f /var/run/DataBaseSynchronizerClient.pid
killall DataBaseSynchronizerClient 2> /dev/null
}
application_restart() {
application_stop
sleep 1
application_start
}
case "$1" in
'start')
application_start
;;
'stop')
application_stop
;;
'restart')
application_restart
;;
*)
echo "usage $0 start|stop|restart"
esac

I agree it should work as it is, but I had problems with Java running in background too. My solution was to use the screen utility (which normally is installed in most Linux distributions) where you can open a shell from which you can detach. If I remember well the commands are something like this (but there is a good manpage too)
screen -S myCustomName # runs a new shell called myCustomName
CTRL + D # detach from the current screen instance
screen -ls # list active screen instances
screen -r myCustomName # reattach to the screen instance.
Hope it will solve your problem.