Recently we migrated to Teamcity deployment from manual process. Ours is java application on linux server.
Whenever deployment is done through Teamcity, logs are stuck i.e. after shutdown log messages nothing else is printed in the logs. Then we run manual stop and start scripts on the server to get the logs running.
Looks like somehow Teamcity locks log file and doesn't release it.
How to overcome it?
In Teamcity, deploy step is defined as below:
REMOTE_PATH="/opt/app/$ARTIFACT/releases/teamcity"
cd $TEAMCITY_REPO_HOME/$ARTIFACT/build/libs
echo "Uploading artifact"
ssh $UAT_HOST -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no "mkdir -p $REMOTE_PATH"
scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no $ARTIFACT.jar $UAT_HOST:$REMOTE_PATH
echo "Stopping service"
ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no $UAT_HOST "sh /opt/app/$ARTIFACT/stop.sh"
sleep 3s
echo "Copying new artifact"
ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no $UAT_HOST "cp $REMOTE_PATH/$ARTIFACT.jar /opt/app/$ARTIFACT"
sleep 6s
echo "Starting service"
ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no $UAT_HOST "sh /opt/app/$ARTIFACT/start.sh"
Taking ideas from above comments of Andy Dufresne, I created one deploy script as below:
This single script stops server, takes backup of current artifact, copies new build and then executes start script to start the server.
#!/bin/bash
#check for correct number of arguments
if [ "$#" -ne 1 ]; then
echo "Incorrect number of arguments, exiting..."
exit 1
fi
ARTIFACT=$1
cd "/opt/app/$ARTIFACT"
echo $(pwd)
echo "Stopping service"
bash stop.sh
sleep 3s
echo "Tagging artifact with release"
cp "releases/teamcity/$ARTIFACT.jar" "releases/teamcity/$ARTIFACT`date +'_%y_%m_%d'`.jar"
echo "Deleting old releases"
cd "releases/teamcity" && \
ls | grep -v '/$' | head -n -6 | xargs -d '\n' -r rm -- && \
cd "/opt/app/$ARTIFACT" && \
ls -l "releases/teamcity"
echo "Copying new artifact"
cp "releases/teamcity/$ARTIFACT.jar" .
echo "Starting service"
bash start.sh
sleep 2s
This script is called from team-city in its build steps(previously stopping server, copying artifact and starting server all were done by teamcity explicitly as outlined in the question.)
Now teamcity build step looks concise as under and WORKS as expected(that important !!!):
REMOTE_PATH="/opt/app/$ARTIFACT/releases/teamcity"
cd $TEAMCITY_REPO_HOME/$ARTIFACT/build/libs
echo "Uploading artifact"
ssh -o "StrictHostKeyChecking=no" "$UAT_HOST" "mkdir -p $REMOTE_PATH"
scp -o "StrictHostKeyChecking=no" "$ARTIFACT.jar" "$UAT_HOST:$REMOTE_PATH"
echo "Deploying artifact"
ssh -o "StrictHostKeyChecking=no" "$UAT_HOST" "sh /opt/app/$ARTIFACT/deploy.sh $ARTIFACT"
I am using CentOS 6.3 64bit.
I am using starting my server as a service.
#!/bin/sh
# chkconfig: 2345 90 10
#description: URL Server
# processname: urlovedserver
start(){
java -jar /root/BookkServer-0.0.1-jar-with-dependencies.jar
}
# See how we were called.
case "$1" in
start)
start
;;
*)
echo "Usage: $0 {start}"
exit 1
esac
What should I add to the script so that it responds to "service urlovedserver restart"? I can't get how to stop a java service.
Write PID in file. When you want you can read PID-file for kill java process.
start(){
#run your process in background
java -jar /root/BookkServer-0.0.1-jar-with-dependencies.jar &
# write PID
echo $! > program.pid
}
Now you can kill process:
kill -9 `cat program.pid` #kill java process
One idea would be to get the pid of the executed command (use the $! shell variable) save it in a file and then add a stop command in your shell script that just kills the java process that it reads from the previous file. Something like
kill -9 `cat /var/run/myjavaservice.pid`.
Then restart will be just stop() and start()
I have written a Java server application that runs on a standard virtual hosted Linux solution. The application runs all the time listening for socket connections and creating new handlers for them. It is a server side implementation to a client-server application.
The way I start it is by including it in the start up rc.local script of the server. However once started I do not know how to access it to stop it and if I want to install an update, so I have to restart the server in order to restart the application.
On a windows PC, for this type of application I might create a windows service and then I can stop and start it as I want. Is there anything like that on a Linux box so that if I start this application I can stop it and restart it without doing a complete restart of the server.
My application is called WebServer.exe. It is started on server startup by including it in my rc.local as such:
java -jar /var/www/vhosts/myweb.com/phpserv/WebServer.jar &
I am a bit of a noob at Linux so any example would be appreciated with any posts. However I do have SSH, and full FTP access to the box to install any updates as well as access to a Plesk panel.
I wrote another simple wrapper here:
#!/bin/sh
SERVICE_NAME=MyService
PATH_TO_JAR=/usr/local/MyProject/MyJar.jar
PID_PATH_NAME=/tmp/MyService-pid
case $1 in
start)
echo "Starting $SERVICE_NAME ..."
if [ ! -f $PID_PATH_NAME ]; then
nohup java -jar $PATH_TO_JAR /tmp 2>> /dev/null >> /dev/null &
echo $! > $PID_PATH_NAME
echo "$SERVICE_NAME started ..."
else
echo "$SERVICE_NAME is already running ..."
fi
;;
stop)
if [ -f $PID_PATH_NAME ]; then
PID=$(cat $PID_PATH_NAME);
echo "$SERVICE_NAME stoping ..."
kill $PID;
echo "$SERVICE_NAME stopped ..."
rm $PID_PATH_NAME
else
echo "$SERVICE_NAME is not running ..."
fi
;;
restart)
if [ -f $PID_PATH_NAME ]; then
PID=$(cat $PID_PATH_NAME);
echo "$SERVICE_NAME stopping ...";
kill $PID;
echo "$SERVICE_NAME stopped ...";
rm $PID_PATH_NAME
echo "$SERVICE_NAME starting ..."
nohup java -jar $PATH_TO_JAR /tmp 2>> /dev/null >> /dev/null &
echo $! > $PID_PATH_NAME
echo "$SERVICE_NAME started ..."
else
echo "$SERVICE_NAME is not running ..."
fi
;;
esac
You can follow a full tutorial for init.d here and for systemd (ubuntu 16+) here
If you need the output log replace the 2
nohup java -jar $PATH_TO_JAR /tmp 2>> /dev/null >> /dev/null &
lines for
nohup java -jar $PATH_TO_JAR >> myService.out 2>&1&
A simple solution is to create a script start.sh that runs Java through nohup and then stores the PID to a file:
nohup java -jar myapplication.jar > log.txt 2> errors.txt < /dev/null &
PID=$!
echo $PID > pid.txt
Then your stop script stop.sh would read the PID from the file and kill the application:
PID=$(cat pid.txt)
kill $PID
Of course I've left out some details, like checking whether the process exists and removing pid.txt if you're done.
Linux service init script are stored into /etc/init.d. You can copy and customize /etc/init.d/skeleton file, and then call
service [yourservice] start|stop|restart
see http://www.ralfebert.de/blog/java/debian_daemon/. Its for Debian (so, Ubuntu as well) but fit more distribution.
Maybe not the best dev-ops solution, but good for the general use of a server for a lan party or similar.
Use screen to run your server in and then detach before logging out, this will keep the process running, you can then re-attach at any point.
Workflow:
Start a screen: screen
Start your server: java -jar minecraft-server.jar
Detach by pressing: Ctl-a, d
Re-attach: screen -r
More info here: https://www.gnu.org/software/screen/manual/screen.html
Another alternative, which is also quite popular is the Java Service Wrapper. This is also quite popular around the OSS community.
Referring to Spring Boot application as a Service as well, I would go for the systemd version, since it's the easiest, least verbose, and best integrated into modern distros (and even the not-so-modern ones like CentOS 7.x).
The easiest way is to use supervisord. Please see full details here: http://supervisord.org/
More info:
https://askubuntu.com/questions/779830/running-an-executable-jar-file-when-the-system-starts/852485#852485
https://www.digitalocean.com/community/tutorials/how-to-install-and-manage-supervisor-on-ubuntu-and-debian-vps
Here is a sample shell script (make sure you replace the MATH name with the name of the your application):
#!/bin/bash
### BEGIN INIT INFO
# Provides: MATH
# Required-Start: $java
# Required-Stop: $java
# Short-Description: Start and stop MATH service.
# Description: -
# Date-Creation: -
# Date-Last-Modification: -
# Author: -
### END INIT INFO
# Variables
PGREP=/usr/bin/pgrep
JAVA=/usr/bin/java
ZERO=0
# Start the MATH
start() {
echo "Starting MATH..."
#Verify if the service is running
$PGREP -f MATH > /dev/null
VERIFIER=$?
if [ $ZERO = $VERIFIER ]
then
echo "The service is already running"
else
#Run the jar file MATH service
$JAVA -jar /opt/MATH/MATH.jar > /dev/null 2>&1 &
#sleep time before the service verification
sleep 10
#Verify if the service is running
$PGREP -f MATH > /dev/null
VERIFIER=$?
if [ $ZERO = $VERIFIER ]
then
echo "Service was successfully started"
else
echo "Failed to start service"
fi
fi
echo
}
# Stop the MATH
stop() {
echo "Stopping MATH..."
#Verify if the service is running
$PGREP -f MATH > /dev/null
VERIFIER=$?
if [ $ZERO = $VERIFIER ]
then
#Kill the pid of java with the service name
kill -9 $($PGREP -f MATH)
#Sleep time before the service verification
sleep 10
#Verify if the service is running
$PGREP -f MATH > /dev/null
VERIFIER=$?
if [ $ZERO = $VERIFIER ]
then
echo "Failed to stop service"
else
echo "Service was successfully stopped"
fi
else
echo "The service is already stopped"
fi
echo
}
# Verify the status of MATH
status() {
echo "Checking status of MATH..."
#Verify if the service is running
$PGREP -f MATH > /dev/null
VERIFIER=$?
if [ $ZERO = $VERIFIER ]
then
echo "Service is running"
else
echo "Service is stopped"
fi
echo
}
# Main logic
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status
;;
restart|reload)
stop
start
;;
*)
echo $"Usage: $0 {start|stop|status|restart|reload}"
exit 1
esac
exit 0
From Spring Boot application as a Service, I can recommend the Python-based supervisord application. See that stack overflow question for more information. It's really straightforward to set up.
Other answers do a good job giving custom scripts and setups depending on your platform. In addition to those, here are the mature, special purpose programs that I know of:
JSW from TanukiSoftware
YAJSW is an open source clone from the above. It is written in Java, and it is a nanny process that manages the child process (your code) according to configurations. Works on windows / linux.
JSVC is a native application. Its also a nanny process, but it invokes your child application through the JNI, rather than as a subprocess.
You can use Thrift server or JMX to communicate with your Java service.
From Spring Boot Reference Guide
Installation as an init.d service (System V)
Simply symlink the jar to init.d to support the standard start, stop, restart and status commands.
Assuming that you have a Spring Boot application installed in /var/myapp, to install a Spring Boot application as an init.d service simply create a symlink:
$ sudo ln -s /var/myapp/myapp.jar /etc/init.d/myapp
Once installed, you can start and stop the service in the usual way. For example, on a Debian based system:
$ service myapp start
If your application fails to start, check the log file written to /var/log/<appname>.log for errors.
Continue reading to know how to secure a deployed service.
After doing as written I've discovered that my service fails to start with this error message in logs: start-stop-daemon: unrecognized option --no-close. And I've managed to fix it by creating a config file /var/myapp/myapp.conf with the following content
USE_START_STOP_DAEMON=false
It is possible to run the war as a Linux service, and you may want to force in your pom.xml file before packaging, as some distros may not recognize in auto mode. To do it, add the following property inside of spring-boot-maven-plugin plugin.
<embeddedLaunchScriptProperties>
<mode>service</mode>
</embeddedLaunchScriptProperties>
Next, setup your init.d with:
ln -s myapp.war /etc/init.d/myapp
and you will be able to run
service myapp start|stop|restart
There are many other options that you can find in Spring Boot documentation, including Windows service.
Im having Netty java application and I want to run it as a service with systemd. Unfortunately application stops no matter of what Type I'm using. At the end I've wrapped java start in screen. Here are the config files:
service
[Unit]
Description=Netty service
After=network.target
[Service]
User=user
Type=forking
WorkingDirectory=/home/user/app
ExecStart=/home/user/app/start.sh
TimeoutStopSec=10
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
start
#!/bin/sh
/usr/bin/screen -L -dmS netty_app java -cp app.jar classPath
from that point you can use systemctl [start|stop|status] service.
To run Java code as daemon (service) you can write JNI based stub.
http://jnicookbook.owsiak.org/recipe-no-022/
for a sample code that is based on JNI. In this case you daemonize the code that was started as Java and main loop is executed in C. But it is also possible to put main, daemon's, service loop inside Java.
https://github.com/mkowsiak/jnicookbook/tree/master/recipes/recipeNo029
Have fun with JNI!
However once started I don't know how to access it to stop it
You can write a simple stop script that greps for your java process, extracts the PID and calls kill on it. It's not fancy, but it's straight forward.
Something like that may be of help as a start:
#!/bin/bash
PID = ps ax | grep "name of your app" | cut -d ' ' -f 1
kill $PID
I have used the following scripting for start and stop a jar file.
**start.sh**
#!/bin/bash
nohup nice java -jar Server.jar > ./Server.out 2>&1 &
**stop.sh**
#!/bin/bash
kill `ps -ef | grep Server.jar | grep -v grep | awk '{ print $2 }'`
Now I want to merge both scripts and create a new restart script. I also want this script output in a terminal instead of a text file(Server.out).
Would appreciate any kind of input/help.
You can either put the commands of the two sripts after each other (kill first, java second) or just call the two scipts in the appropriate order.
The idea is that restart is basically equivalent to killing the current running version and starting a new one.
To avoid the output to a file, remove the > ./Server.out part.
Edit: removed note about removing the redirection part as I misread the grep part of the kill script
Update: Missed the nohup part of the script: with nohup you need to redirect output to a file, because the process is detached from the terminal (see documentation). If you do want to see the output in the terminal, remove nohup as well as the redirection to the file
I need to be able to start up a screen without connecting to it, but it also needs to run my start.sh script which contains the java line to start Minecraft.
screen -d -m new3 -c start.sh
Is what I've been trying to use, but it never runs start.sh
In a snippet of code I found on line it seems to do what I want but I need some help
mc_start() {
cd $MCPATH
as_user "cd $MCPATH && screen -dmS $SCREEN $INVOCATION"
#
# Waiting for the server to start
#
seconds=0
until ps ax | grep -v grep | grep -v -i SCREEN | grep $SERVICE > /dev/null
do
sleep 1
seconds=$seconds+1
if [[ $seconds -eq 5 ]]
then
echo "Still not running, waiting a while longer..."
fi
if [[ $seconds -ge 120 ]]
then
echo "Failed to start, aborting."
exit 1
fi
done
echo "$SERVICE is running."
}
I think this is because your command is wrong. I'm assuming that you want to create a new session named new3 and detach from that
screen -d -m -S new3 ~/start.sh
Afterwards you can run the following command to connect back to your session.
screen -R new3