We Keep Coding

Stop Java from killing Bash script that started it

I've spent the past couple days working on this, and at this point I am super stuck. I have a Java program that must be run not as a service. This program must also be capable of updating itself when a new file is given for updating.
As a result, I have a script that is started with Linux that starts the Java application and then checks every 5 seconds if the application has been terminated. If the application has been terminated, it should check if there is an update and then start appropriately.
This is the code for that script:
#!/bin/bash
JAVA_HOME=/usr/lib/jvm/java-16-openjdk-amd64
WORKING_DIR=~/Data
LOG=$WORKING_DIR/logs/Bash.log
rm $LOG
echo "Script started" > $LOG
while true; do
source $WORKING_DIR/Server.pid
if ! kill -0 $AppPID; then
echo "App must be started" >> $LOG
source $WORKING_DIR/UpdateStatus
if [ "$UpdateReady" -eq "1" ]; then
echo "Moving files for update" >> $LOG
mv $WORKING_DIR/Server.jar $WORKING_DIR/old.jar
mv $WORKING_DIR/new.jar $WORKING_DIR/Server.jar
fi
nohup ${JAVA_HOME}/bin/java -jar ${WORKING_DIR}/Server.jar &
echo AppPID="$!" > $WORKING_DIR/Server.pid
echo "Server started" >> $LOG
if [ "$UpdateReady" -eq "1" ]; then
echo "Checking for safe update" >> $LOG
source $WORKING_DIR/Server.pid
echo UpdateReady="0" > $WORKING_DIR/UpdateStatus
sleep 5;
if kill -0 $AppPID; then
echo "Update successful" >> $LOG
rm $WORKING_DIR/old.jar
else
echo "Update failed, restarting old jar" >> $LOG
rm $WORKING_DIR/Server.jar
mv $WORKING_DIR/old.jar $WORKING_DIR/Server.jar
nohup ${JAVA_HOME}/bin/java -jar ${WORKING_DIR}/Server.jar &
echo AppPID="$!" > $WORKING_DIR/Server.pid
fi
fi
echo "Server start process finished, going into idle state" >> $LOG
fi
sleep 5
echo "5 seconds idle passed" >> $LOG
done
To initialize the update, I have tried a couple of different things, both with the same result. First I had set UpdateReady="1" through Java, then used exit(0);. I have also tried having Java call a Bash script which also sets UpdateReady="1" but uses kill $AppPID to shutdown the java application.
The result is that both the Java application and the Bash script stop executing causing the update and restart to fail! I have looked through a significant amount of Stack Overflow questions and answers finding things such as nohup, all to no avail.
I will once again state that the Java application cannot be run as a service. No packages other than what is included in Java or made by Apache can be used, and no programs can be installed to Linux. I would prefer to solve the problem with Bash.
Upon testing some things mentioned in comments, I may have missed something that turns out to be important. While all other runs of the startup script will be run by the startup applications manager, the initial run is not.
The install is taken care of remotely with an SSH connection sending the following command string:
cd /home/UserName; unzip -u -o Server.zip; chmod 777 install.sh; bash install.sh &; exit
install.sh is as follows:
#!/bin/bash
INSTALL_DIR=~/Data
mkdir ${INSTALL_DIR}
mkdir ${INSTALL_DIR}/logs
mkdir ${INSTALL_DIR}/data
cp Server.jar ${INSTALL_DIR}/Server.jar
cp service-start.sh ${INSTALL_DIR}/service-start.sh
chmod 777 ${INSTALL_DIR}/service-start.sh
rm Server.jar
rm service-start.sh
rm Server.zip
nohup bash $INSTALL_DIR/service-start.sh &
Upon rebooting my machine, I noticed that this problem goes away! This means that there must be a problem with the initial setup script. When the shell command is run, it does seem to be sticky and not actually let go after the bash install.sh &. I have tried putting nohup at the beginning of this, however the entire line will not run in that case for reasons I am not able to determine.
I would prefer to not have the user be forced to restart after install and can't seem to find any way to force the Startup Application manager to start an application at any time other than startup.

Well, after a lot of searching and some prompting from the comments, I found that the issue lied with how the program was initially being started.
As mentioned in the update, the first run is always started by an ssh connection. I knew there was a slight problem with this ssh connection, as it seemed to hold onto the connection no matter what I did. It turns out that this was causing the problem that resulted in the Bash instance and the Java instance remaining attached.
The solution for this problem was found here: jsch ChannelExec run a .sh script with nohup "lose" some commands
After managing to get the initial setup to start with nohup properly, the issue has gone away.

Serial port access fails when Java app is Dockerized

My Spring Boot app uses the jSerialComm library (v2.6.0) to do serial comms over a USB port. The jSerialComm documentation notes the importance of adding the user to a number of groups:
Note for Linux users: Serial port access is limited to certain users and groups in Linux. To enable user access, you must open a terminal and enter the following commands before jSerialComm will be able to access the ports on your system. Don't worry if some of the commands fail. All of these groups may not exist on every Linux distro. (Note, this process must only be done once for each user):
sudo usermod -a -G uucp username
sudo usermod -a -G dialout username
sudo usermod -a -G lock username
sudo usermod -a -G tty username
So I wrote the Dockerfile as follows:
FROM adoptopenjdk:11-jre-hotspot
# Run application as non-root user to help to mitigate some risks
RUN groupadd -r spring && useradd -r spring -g spring && \
usermod -a -G uucp spring && \
usermod -a -G dialout spring && \
usermod -a -G tty spring
# `lock` group doesn't seem to exist, hence commented-out:
# usermod -a -G lock spring
USER spring:spring
COPY /Java/tempctrl/build/libs/*.jar app.jar
EXPOSE 80
ENTRYPOINT ["java", "-jar", "/app.jar"]
... and I include --device=/dev/ttyACM0:/dev/ttyACM0 (and, temporarily clutching at straws, --privileged) in the docker run command.
When the app starts, logging confirms that /dev/ttyACM0 is found OK. But when the app tries to read from the serial port it receives continuous zeros. (Note: saw this a few times before moving the app to Docker and it was symptomatic of the USB port being already in use.)
If I comment-out USER spring:spring (i.e. allow the contained app to run as root) everything is fine.
How can I make this work without root privileges?

According to the Docker docs, devices shared with --device seem to be root-owned (see image in link).
Perhaps you can try chowning it?

Can't get Java app to listen on port for remote debugging purposes?

I have a Spring Boot Java app running on Ubuntu 14.x using Oracle Java 1.8.0 that I want to debug remotely with IntelliJ. I have tried to get it to listen on a port for debug purposes but with no success. Note, the ports I tried are all well above the port 1024, to make sure it's not a permission problem. I am not root but I do have sudo access to the box.
I tried adding this to the java command line:
-agentlib:jdwp=transport=dt_socket,address=localhost:9009,server=y,suspend=y
A technique I got from this document:
http://javahowto.blogspot.com/2010/09/java-agentlibjdwp-for-attaching.html
However when I run this command:
sudo netstat -an | grep LISTEN
I don't see port 9009. Also, the app does not wait for debugger attachment as indicated by the "suspend=y" parameter, because I see the app initialization messages stream by as normal as the app starts up. Why isn't this working?
Here is the shell script that launches the app. Note, this shell script is launched by supervisord. I point this out in case that might be causing any trouble:
# !/bin/bash
# Shell script to launch Spring Boot app
# Kill subprocess when parent bash process is terminated by supervisor or when CTRL+C is received
trap 'kill -TERM $PID' TERM INT
java \
-Dnetworkaddress.cache.ttl=5 \
-Dnetworkaddress.cache.negative.ttl=5 \
\
-jar spbootapp.jar \
-agentlib:jdwp=transport=dt_socket,address=localhost:9009,server=y,suspend=y
--spring.application.name=spbootapp-awsdev \
--spring.profiles.active=cluster \
--spring.cloud.config.enabled=false \
--endpoints.configprops.enabled=false \
--endpoints.health.sensitive=false \
&

The debug parameters -agentlib:jdwp=transport=dt_socket,address=localhost:9009,server=y,suspend=y need to go before the -jar in the command.

Ansible terminates my app with "nohup"

I have a task in Ansible playbook that executes a .jar file in the background, but after finishing that task, the (java) app terminates.
- name: Run Java app in the background
shell: nohup java -jar app.jar &
I need the app running for tasks further down in the playbook. Any ideas??
NOTE: When I run it in Putty ssh session it runs smoothly and the app stays in background.

The most likely reason is attached IO. Try:
- name: Run Java app in the background
shell: nohup java -jar app.jar </dev/null >/dev/null 2>&1 &

Ok So I had this issue today and my script is a tad different but essentially doing the same thing so this is what I have, running on Oracle, Centos and RHEL.
in script start_service.sh I have this, this script is executed not sourced.
#!/bin/sh
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)"
nohup java -jar ${DIR}/$1 --spring.config.location=${DIR}/application.yml --logging.config=${DIR}/logging.xml > ${DIR}/log/output.log 2>&1 &
In my playbook I have:
- name: run service
shell: "{{ service_install_location }}/application_name/start-service.sh application_name-{{ application_version }}.jar"
args:
chdir: "{{ service_install_location }}/application_name"
executable: /bin/bash
Now, the BASH_SOURCE argument is specific to bash so if you are running plain old sh then you need $0. nohup is required so when the ssh connection terminates your background task does not get killed (turn on -vvv for ansible to see connections being made to understand how this happens on a remote machine).
I should point out we are doing this temporarily as this should really be installed as a service to be under the control of service/init.d but for now, this answers your question

Exception in thread "main" java.io.StreamCorruptedException: invalid stream header: 0BDAACED

I am trying to execute jenkins cli command from master.
ssh user#192.168.1.2 -C "/usr/bin/java -jar /home/user/slave.jar"
Getting following error:
<===[JENKINS REMOTING CAPACITY]===>Exception in thread "main" java.io.StreamCorruptedException: invalid stream header: 0BDAACED
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:808)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:301)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
at hudson.remoting.ChannelBuilder.makeTransport(ChannelBuilder.java:430)
at hudson.remoting.ChannelBuilder.negotiate(ChannelBuilder.java:389)
at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:310)
at hudson.remoting.Launcher.main(Launcher.java:528)
at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:468)
at hudson.remoting.Launcher.run(Launcher.java:242)
at hudson.remoting.Launcher.main(Launcher.java:195)
ERROR: Unexpected error in launching an agent. This is probably a bug in Jenkins
hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected EOF
at hudson.remoting.Request.abort(Request.java:303)
at hudson.remoting.Channel.terminate(Channel.java:847)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:92)
at ......remote call to ubuntu-slave(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
at hudson.remoting.Request.call(Request.java:172)
at hudson.remoting.Channel.call(Channel.java:780)
at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:508)
at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:381)
at hudson.slaves.CommandLauncher.launch(CommandLauncher.java:131)
at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:253)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Unexpected EOF
at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:99)
at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
ERROR: Connection terminated
java.io.IOException: Unexpected EOF
at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:99)
at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
ERROR: Process terminated with exit code 1
Please tell me what could be the problem?

The short answer
Jenkins' slave.jar communicates with the Jenkins server through the slave's stdin/stdout. You rely on ssh to transfer stdin/stdout to/from the slave.
Something that you are not aware of tampers with the stdin before it reaches the slave, and the resulting communication protocol violation causes the exception that you see.
The longer explanation and a fix
I suppose that you have the ssh command from your question, which starts the slave, in a shell script on the jenkins server, and that this shell script also has other commands in it that precede this command. Because this is how it was in my case, inspired by the jenkins remoting documetation that suggests you could do things like copying the correct slave.jar to the slave.
So we are both using the "Launch agent via execution of command on the server" option to start the slave agent. In my case I do this to be able to use an ssh jump host to reach the slave, but this is not of relevance to this answer.
I will now give examples of non-working and working shell scripts to start a slave.jar on a remote node, and then come up with reasoning of what might cause the observed behaviour. I use /bin/bash as the shell, feel free to use others.
1) These versions both work if you already have slave.jar on the slave
#!/bin/bash
ssh user#host "java -jar /home/user/slave.jar"
.
#!/bin/bash
exec ssh user#host "java -jar /home/user/slave.jar"
Both versions have only a single command in them. Nothing else is in here that tampers with stdin or stdout. In the first version, the shell lingers around and forwards stdin/stdout to/from the ssh command. In the second version, the shell is replaced by the ssh process and directly inherits its stdin/stdout. Both versions work ok, and the additional shell process in the first version should not matter on any system.
2) This version copies slave.jar from some location on the server to the slave before executing it on the slave
#!/bin/bash
scp -q /some/location/slave.jar user#host:.
exec ssh user#host "java -jar /home/user/slave.jar"
Of course, this only works if you have /some/location/slave.jar on the server.
3) This version tries to do some additional cleanup on the slave before starting the slave agent.
#!/bin/bash
ssh user#host "rm -rf /home/user/tmp/jenkins"
exec ssh user#host "java -jar /home/user/slave.jar"
The /home/user/tmp/jenkins location on the slave is just an example.
This version fails after a 4 minute timeout. With the exact error message from the question. The failure is not caused by anything important missing from /home/user/tmp/jenkins, as you will see in the next example:
4) Different ways to make example 3 work
#!/bin/bash
ssh user#host "rm -rf /home/user/tmp/jenkins" </dev/null
exec ssh user#host "java -jar /home/user/slave.jar"
.
#!/bin/bash
exec ssh user#host "rm -rf /home/user/tmp/jenkins && java -jar /home/user/slave.jar"
In the first fix, we make sure stdin to communicate with the slave is not forwarded to the remote rm command. rm does not read its stdin, but apparently ssh does not know that and buffers some bytes from its own stdin and forwards them to the remote command just in case it is needed? This is fixed by forwarding /dev/null as stdin to the rm command instead of the stdin destined for the slave communication.
In the second fix, only one ssh command is used, the rm command again does not read anything from stdin, and the slave.jar receives the untampered stream.