We Keep Coding

Will this NGS data annotation code pipeline work?

threads=36
task() {
NEWNAME="${1/%.norm.vcf.gz/.snv.indel.vcf.gz}"
java -Xmx8g -jar /home/ubuntu/snpEff/SnpSift.jar annotate /home/ubuntu/data/spliceai_scores.raw.snv.hg19.vcf.gz $1 | java -Xmx8g -jar /home/ubuntu/snpEff/SnpSift.jar annotate /home/ubuntu/data/spliceai_scores.raw.indel.hg19.vcf.gz > $NEWNAME1
}
for file in $(ls *norm.vcf.gz)
do
if [ $(jobs -r | wc -l) -ge $threads ]; then
wait $(jobs -r -p | head -1)
fi
task "$file" &
done
wait
I am now learning NGS pipeline and I would like to annotate two DBs(snv,indel) using SnpSift. Currently I am using 36 threads CPU. Since there are two commands in task() function, annotate snv and indel, will my code wait $(jobs -r -p | head -1) work fine?
Are there any other recommendations to use CPU efficiently?

How do i display linux command result in html [duplicate]

This question already has an answer here:
Formatting issues when unix file is sent to mail
(1 answer)
Closed 1 year ago.
I want to run linux command(ps aux --sort -rss | head -n 10) and display the result in table format in mail.
can somebody please suggest.
here is my script.
#!/usr/bin/ksh
current_dir=$(pwd)
script_dir=$(dirname $0)
if [ $script_dir = '.' ]; then
script_dir="$current_dir"
fi
source_dir=$PWD
cd $script_dir
FWK_TIMESTAMP=`date +%Y%m%d_%H:%M:%S`
REPORT_TS=`date -d '1 day ago' +'%m/%d/%Y'`
TILL_DATE=`date +%m/%d/%Y`
#javac -classpath mail1.4.4.jar:activation.jar SendEmail.java
if [ $? -ne 0 ];then
echo javac failed $?
exit 1
fi
outfile=test.txt
rm test.txt
echo "<h1> Top 10 Memory Consumed Processes from $REPORT_TS to $TILL_DATE</h1>" >> $outfile
ps aux --sort -rss | head -n 10 >> $outfile
to='abc#abc.com'
mail_server='-mail.smtp.host abc.com'
param="-to $to $mail_server -from abc#abc.com"
#-mail.debug
java -cp mail1.4.4.jar:activation.jar:. SendEmail $param -subject "Top Memory Consumed Processes" -file test.txt

As I am able to see in your code,
you are saving the results of
ps aux --sort -rss | head -n 10 >> $outfile
in the $outfile.
You can use <iframe src="your_filename_goes_here" width=200 height=400 frameborder=0 ></iframe>

"Too late for "-C" option" error With Perl and Shell scripts

I have a jar application that has several functions, one of which is to convert from HTML to XML. When I try to run a simple command such as:
java -jar lt4el-cmd.jar send -l en "l2:https://en.wikipedia.org/wiki/Personal_computer"
I get the following errors:
ERROR [Thread-1]: html2base/html2base-wrapper.sh: Too late for "-C" option at html2base/html2xml.pl line 1.
/tmp/lpc.30872.html: failed
cat: /tmp/lpc.30872.xml: No such file or directory
(LpcControl.java:229)
ERROR [Thread-1]: ana2ont/ana2ont.sh ${lang}: -:1: parser error : Document is empty
-:1: parser error : Start tag expected, '<' not found
Tokenization/tagging failed
^
-:1: parser error : Document is empty
unable to parse -
-:1: parser error : Document is empty
unable to parse -
(LpcControl.java:229)
ERROR [Thread-1]: Error in conversion: Error running conversion script (ana2ont/ana2ont.sh ${lang}): 6 (AppInterface.java:159)
This is the html2base-wrapper.sh script which seems to be where the first error occurs.
#!/bin/bash
if [ "$1" == "check" ]; then
. common.sh
check_binary perl || exit 1
check_perl_module HTML::TreeBuilder || exit 1
check_perl_module XML::LibXML || exit 1
check_binary tidy || exit 1
check_binary xmllint || exit 1
check_binary xsltproc || exit 1
exit
fi
cat >"$TMPDIR/lpc.$$.html"
html2base/html2base.sh -d html2base/LT4ELBase.dtd -x html2base/LT4ELBase.xslt -t "$TMPDIR/lpc.$$.html" >&2
cat "$TMPDIR/lpc.$$.xml";
rm -f "$TMPDIR"/lpc.$$.{ht,x}ml
And the html2base.sh script:
#!/bin/bash
#
# Sample script for automated HTML -> XML conversion
#
# Miroslav Spousta <spousta#ufal.mff.cuni.cz>
# $Id: html2base.sh 462 2008-03-17 08:37:14Z qiq $
basedir=`dirname $0`;
# constants
HTML2XML_BIN=${basedir}/html2xml.pl
ICONV_BIN=iconv
TIDY_BIN=tidy
XMLLINT_BIN=xmllint
XSLTPROC_BIN=xsltproc
DTDPARSE_BIN=dtdparse
TMPDIR=/tmp
# default values
VERBOSE=0
ENCODING=
TIDY=0
VALIDATE=0
DTD=${basedir}/LT4ELBase.dtd
XSLT=${basedir}/LT4ELBase.xslt
usage()
{
echo "usage: html2base.sh [options] file(s)"
echo "XML -> HTML conversion script."
echo
echo " -e, --encoding=charset Convert input files from encoding to UTF-8 (none)"
echo " -d, --dtd=file DTD to be used for conversion and validation ($DTD)"
echo " -x, --xslt=file XSLT to be applied after conversion ($XSLT)"
echo " -t, --tidy Run HTMLTidy on input HTML files"
echo " -a, --validate Validate output XML files"
echo " -v, --verbose Be verbose"
echo " -h, --help Print this usage"
exit 1;
}
OPTIONS=`getopt -o e:d:x:tahv -l encoding:,dtd:,xlst,tidy,validate,verbose,help -n 'convert.sh' -- "$#"`
if [ $? != 0 ]; then
usage;
fi
eval set -- "$OPTIONS"
while true ; do
case "$1" in
-e | --encoding) ENCODING=$2; shift 2 ;;
-d | --dtd) DTD=$2; shift 2 ;;
-x | --xslt) XSLT=$2; shift 2 ;;
-t | --tidy) TIDY=1; shift 1;;
-a | --validate) VALIDATE=1; shift 1;;
-v | --verbose) VERBOSE=1; shift 1 ;;
-h | --help) usage; shift 1 ;;
--) shift ; break ;;
*) echo "Internal error!" ; echo $1; exit 1 ;;
esac
done
if [ $# -eq 0 ]; then
usage;
fi
DTD_XML=`echo "$DTD"|sed -e 's/\.dtd/.xml/'`
if [ "$VERBOSE" -eq 1 ]; then
VERBOSE=--verbose
else
VERBOSE=
fi
# create $DTD_XML if necessary
if [ ! -f "$DTD_XML" ]; then
if ! $DTDPARSE_BIN $DTD -o $DTD_XML 2>/dev/null; then
echo "cannot run dtdparse, cannot create $DTD_XML";
exit 1;
fi;
fi
# process file by file
total=0
nok=0
while [ -n "$1" ]; do
file=$1;
if [ -n "$VERBOSE" ]; then
echo "Processing $file..."
fi
f="$file";
result=0;
if [ -n "$ENCODING" ]; then
$ICONV_BIN -f "$ENCODING" -t utf-8 "$f" -o "$file.xtmp"
result=$?
error="encoding error"
f=$file.xtmp
fi
if [ "$result" -eq 0 ]; then
if [ "$TIDY" = '1' ]; then
$TIDY_BIN --force-output 1 -q -utf8 >"$file.xtmp2" "$f" 2>/dev/null
f=$file.xtmp2
fi
out=`echo $file|sed -e 's/\.x\?html\?$/.xml/'`
if [ "$out" = "$file" ]; then
out="$out.xml"
fi
$HTML2XML_BIN --simplify-ws $VERBOSE $DTD_XML -o "$out" "$f"
result=$?
error="failed"
fi
if [ "$result" -eq 0 ]; then
$XSLTPROC_BIN --path `dirname $DTD` $XSLT "$out" |$XMLLINT_BIN --noblanks --format -o "$out.tmp1" -
result=$?
error="failed"
mv "$out.tmp1" "$out"
if [ "$result" -eq 0 -a "$VALIDATE" = '1' ]; then
tmp=`dirname $file`/$DTD
delete=0
if [ ! -f $tmp ]; then
cp $DTD $tmp
delete=1
fi
$XMLLINT_BIN --path `dirname $DTD` --valid --noout "$out"
result=$?
error="validation error"
if [ "$delete" -eq 1 ]; then
rm -f $tmp
fi
fi
fi
if [ "$result" -eq 0 ]; then
if [ -n "$VERBOSE" ]; then
echo "OK"
fi
else
echo "$file: $error "
nok=`expr $nok + 1`
fi
total=`expr $total + 1`
rm -f $file.xtmp $file.xtmp2
shift;
done
if [ -n "$VERBOSE" ]; then
echo
echo "Total: $total, failed: $nok"
fi
And the beginning part of the html2xml.pl file:
#!/usr/bin/perl -W -C
# Simple HTML to XML (subset of XHTML) conversion tool. Should always produce a
# valid XML file according to the output DTD file specified.
#
# Miroslav Spousta <spousta#ufal.mff.cuni.cz>
# $Id: html2xml.pl 461 2008-03-09 09:49:42Z qiq $
use HTML::TreeBuilder;
use HTML::Element;
use HTML::Entities;
use XML::LibXML;
use Getopt::Long;
use Data::Dumper;
use strict;
I can't seem to figure where the problem is. And what exactly does ERROR [Thread-1] mean?
Thanks

The error comes from having -C on the shebang (#!) line of a Perl script, but not passing the -C to perl. This type of error happens when someone does
perl html2base/html2xml.pl ...
instead of
html2base/html2xml.pl ...

The error was from the the html2xml.pl script as other users rightly mentioned. I'm running ubuntu 16.04.2 system which comes with a default perl 5.22 version. And as this post mentions, using the -C option (as from perl 5.10.1) on the #! line requires you to also specify it on the command line at execution time, which I wasn't sure how to do because I was running a jar file. I installed perlbrew, instead, which I used to get an earlier version of perl and modified my perl script to:
#!/usr/bin/path/to/perlbrew/perl -W -C
# Simple HTML to XML (subset of XHTML) conversion tool. Should always produce a
# valid XML file according to the output DTD file specified.
#
# Miroslav Spousta <spousta#ufal.mff.cuni.cz>
# $Id: html2xml.pl 461 2008-03-09 09:49:42Z qiq $
This might also come in handy in setting up shell scripts when using perlbrew.
Thanks for the efforts in contribution.

Spring XD: "Stream closed between payloads" exception for shell processor

The simple shell processor takes input data and echos it back:
nano /tmp/echo.sh
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
#!/bin/bash
[ $# -ge 1 -a -f "$1" ] && input="$1" || input="-"
cat $input
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
It works fine when deployed to Spring XD by:
stream create test --definition "file --mode=contents --outputType=text/plain | shell --command='bash /tmp/echo.sh' | b:file --binary=true --dirExpression='''/tmp/out''' --nameExpression=headers[file_name]" --deploy
But when i try to extend the shell script to call a webservice, e.g. by
#!/bin/bash
[ $# -ge 1 -a -f "$1" ] && input="$1" || input="-"
#cat $input
curl http://some.address/webservice -d 'output=xml' -d "text=$(cat $input)"
i always get an "Stream closed between payloads" exception from Spring XD. Even if i try to return an (existing) file by
#!/bin/bash
[ $# -ge 1 -a -f "$1" ] && input="$1" || input="-"
#cat $input
cat /tmp/output.xml
i get the same exception.

How to find unused properties in a pom

After inheriting a maven project, I would like to check for unused properties and remove them.
One way I dont want to take is to remove them one by one and see the build fail. Another way would be to count the occurences in the whole codebase (to make sure properties for filters and resources are not wrongly seen as unused) with a custom script. Before I do that, I would like to make sure I'm not reinventing the wheel.
There is a way to do that for the dependencies explained here.
Is there something similar for properties that I have missed? Or a better way?
Thank you
I am using maven 3.0.3

It's a bit complicated but here's a bash script that will parse the properties element from a pom.xml, check if each property is used in the pom, and optionally checks all project files (a deep search).
#!/bin/bash
cmd=$(basename $0)
read_dom () {
local IFS=\>
read -d \< entity content
local retval=$?
tag=${entity%% *}
attr=${entity#* }
return $retval
}
parse_dom () {
# uncomment this line to access element attributes as variables
#eval local $attr
if [[ $tag = "!--" ]]; then # !-- is a comment
return
elif [[ $tag = "properties" ]]; then
in=true
elif [[ $tag = "/properties" ]]; then
in=
elif [[ "$in" && $tag != /* ]]; then #does not start with slash */
echo $tag
fi
}
unused_terms () {
file=$1
while read p; do
grep -m 1 -qe "\${$p}" $file
if [[ $? == 1 ]]; then echo $p; fi
done
}
unused_terms_dir () {
dir=$1
while read p; do
unused_term_find $dir $p
done
}
unused_term_find () {
dir=$1
p=$2
echo -n "$p..."
find $dir -type f | xargs grep -m 1 -qe "\${$p}" 2> /dev/null
if [[ $? == 0 ]]; then
echo -ne "\r$(tput el)"
else
echo -e "\b\b\b "
fi
}
if [[ -z $1 ]]; then
echo "Usage: $cmd [-d] <pom-file>"
exit
fi
if [[ $1 == "-d" ]]; then
deep=true
shift
fi
file=$1
dir=$(dirname $1)
if [ $deep ]; then
while read_dom; do
parse_dom
done < $file | unused_terms $file | unused_terms_dir $dir
else
while read_dom; do
parse_dom
done < $file | unused_terms $file
fi
The script uses XML parsing code from this thread.
This script won't find any properties that are used by maven or maven plugins directly. It simply looks for the pattern ${property} within project files.
It works on my mac; your milage may vary.

Unfortunately there is no better way instead of doing it manually ...

If you are using IntelliJ, you could just search for usages of the property. Just put your cursor in the tag, and right click -> Find Usages (keyboard shortcut in my case: Alt+F7).
Haven't found a better way yet.

dependency_cleaner https://github.com/junaidbs/dependency_cleaner
This jar will help to identify and then remove the unwanted dependency from pom .
It will automate the process of Removing a dependency and run then check whether dependency needful
It's a better way than doing manualy