I am trying to run a simple script on a Linux HPC, where I am looping over a number of RNA-seq files and want to call fastQC, a Java based program, on each of the files.
In the script I'm calling a wrapper script that runs the Java fastQC-program.
The script I wrote is as follows:
#!/bin/bash
#SBATCH -t 00:30:00
#SBATCH -n 1
#SBATCH --mail-type=end
#SBATCH --clusters=serial
#SBATCH --partition=serial_std
#SBATCH --mem=6gb
echo "Starting to run fastQC..."
FILENAMES_TXT=*filenames_path*/small_test_sample_filenames.txt
DIR_DATA=*data_directory*
IPPG=*roote_directory on the cluster dss*
FILENAMES_PATH=$IPPG/$FILENAMES_TXT
DATA_PATH=$IPPG/$DIR_DATA
LINES=$(cat $FILENAMES_PATH)
for LINE in $LINES
do
FILE=$DATA_PATH/$LINE
echo $FILE
*home_directory_path*/FastQC/fastqc $FILE
done
echo "fastQC is done!"
When I submit this job using sbatch, the SLURM error message is:
Can't exec "java": No such file or directory
Running the script on a single sample on the HPC-Login-Node worked fine, so I think the script published by fastQC is fine, and the problem only arises, when the script is submitted to the SLURM system with sbatch.
Apparently, the fastQC-script cannot find JAVA in the SLURM system, after it was called in my Bash-script.
Can anybody with HPC / SLURM experience tell me, how to import Java into the environment when submitting a job (I am not sure if that question is phrased correctly, but I hope somebody can understand)?
Solution was found through the system administrator:
module load openjdk
Related
update：
I have followed spring boot deployment for Installation as an init.d Service (System V). I start my application successfully.But, after one day.My application closed again......Is there any ways to help me ?
I hava one spring boot application which names my.jar. I put it into my ubuntu server(20.04) and use the command of nohup java -jar my.jar &.In the first few hours, my application is in good condition.But,after one or two days,it will shut down automatically.
I have seen the log of my application which don't recorded any error and saved the last correct log before exiting
The current situation is my application is very simple and have only a small number of visits.
According to my guess,it is seems that ubuntu kill my process for inactivity?
My scripts are as follow,
stop.sh:
#!/bin/bash
PID=$(ps -ef | grep centre-0.0.1-SNAPSHOT.jar | grep -v grep | awk '{ print $2 }')
if [ -z "$PID" ]
then
echo Application is already stopped
else
echo kill $PID
kill $PID
fi
run.sh:
#!/bin/bash
echo stop application
source stop.sh
echo start application
source start.sh
start.sh:
#!/bin/bash
nohup java -jar centre-0.0.1-SNAPSHOT.jar &
I need my application will run all the time.
Is there any ways to resolve the issue?
nohup is more suited for running processes that are expected to end after a while. For example running a lengthy batch script.
As of why it is killed, there may be a number of reasons: memory leak, server security policy .... The server probably decided your program wasn't behaving correctly. Logs of the server, like dmesg or /var/log/ contents may have some clues about it.
What you said about your app receiving visits feels like it is more a service rather than a script.
You may want to
daemonize your program
. this will make your program tied to the server availability.
Here is an explanation of the differences between nohup and daemons.
Also check this link at baeldung for help on setting up the daemon
After I check the memory usage, so I realized that my spring boot have used a lot of memory abnormally.
I solved it in the following way:
I updated my start.sh with the command of "nohup java -Xms100m -Xmx300m -jar ./target/centre-0.0.1-SNAPSHOT.jar &"
But, I did not figure out why did my spring boot application use a lot of memory?
Is there anyone can explain this phenomenon?
I'm trying to optimize some experiments with a Java application. The same application is on many machines. I want to run all of them via a bash script with ssh.
I have a bash script that has a while loop to run the application. Like this
while [ $COUNTER -lt $WORKERS ]
do
ssh ubuntu#host "java java-app.jar" > /data/some-logs.log
((COUNTER++))
((IP_BEGINS++))
done
However when I run the script I have to wait a moment and press Ctrl+C for every machine. How can I run every aplication on background?
prefix with nohup and append a & to the command, that will run it in the background.
while [ $COUNTER -lt $WORKERS ]
do
ssh ubuntu#host "nohup java -jar java-app.jar > /data/some-logs.log 2>&1 &"
((COUNTER++))
((IP_BEGINS++))
done
You might need to muck around with the quotes and placements of the & to make sure the remote ssh command gets backgrounded and not your local ssh
EDIT - I fixed the answer based on your comment. Also added the stderr redirect to the same log file, that might help when things go wrong
i usually use this bash script to check if my java application is running and start it again if not. I use crontab to check it
#!/bin/bash
if [ "$(pidof java)" ]
then
# process was found
echo "application running"
else
# process not found
cd /home/assist/emanager
setsid java -jar emanager-1.0.0.jar </dev/zero &>/dev/null &
fi
Now the problem is that there are other java applications running on the server, so the script does not start my app because the if condition is true. Is there a way to check if a specific java application is running?
Thanks
Your question is very similar to this one:
Start a Java application using Bat file if not already started
The main difference is you are asking for bash while the other one is in Windows. Hence replace jcmd with jps, the rest is still applicable.
The following process normally works for my startup scripts. However, when I introduce a command to execute a JAR file, it does not work. This script works while I am logged in. However, it does not work as a startup script.
In /etc/init.d I create a bash script (test.sh) with the following contents:
#!/bin/bash
pw=$(curl http://169.254.169.254/latest/meta-data/instance-id)
pwh=$(/usr/bin/java -jar PWH.jar $pw &)
echo $pwh > test.txt
Make script executable
In /etc/rc.local, I add the following line:
sh /etc/init.d/test.sh
Notes:
I make a reference to the script in /etc/rc.local, because this script needs to run last after all services have started.
Please do not ask me to change the process (i.e., create script in /etc/init.d/ and reference it from /etc/rc.local), because it works for my other startup scripts.
I have tried adding nohup in front of java command, and it still did not work.
Thanks
As written, there is insufficient information to say what is going wrong. There are too many possibilities to enumerate.
Try running those commands one at a time in an interactive shell. The java command is probably writing something to standard error, and that will give you some clues.
So, I have a php application, running through cgi, with nginx.
I have a .jar program used to print barcodes on, guess what, a barcode printer (Zebra)!.
My php application runs the .jar program using the exec() command. On my computer (php, cgi, nginx, debian testing), it works.
Now, I'm trying the same thing, but on a raspberry pi. Debian (wheezy, raspberry version), php, cgi, nginx.
The exec() command does not fail (in the way I dont get a huge fatal error), does not return anything, and does not succeed neither.
The java command works fine when I try it in the shell, using the www-data user (the one running nginx/php-cgi). It works when I put it in a php-cli (cli, not cgi) script, and run it with the php command in the shell, with the www-data user :
<?php exec('java -cp /path/ Methode options');?>
$ php print.php
Cups configuration seems ok (I (the pi user) can print, the www-data user can print through command line (shell), root can print, everyone can).
I'm confused and can't find where the problem is.
Try adding 2>&1 at the end of your exec(), in example: exec('java -cp /path/ Method options 2>&1');
Edit:
I would not advice you to do this if it was a real server; but since its a PI you can edit the sudoers file and add the following
run sudo visudo
and add:
www-data ALL=NOPASSWD: ALL
now modify your exec string with:
exec('sudo java -cp /path/ Method options');