The simple shell processor takes input data and echos it back:
nano /tmp/echo.sh
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
#!/bin/bash
[ $# -ge 1 -a -f "$1" ] && input="$1" || input="-"
cat $input
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
It works fine when deployed to Spring XD by:
stream create test --definition "file --mode=contents --outputType=text/plain | shell --command='bash /tmp/echo.sh' | b:file --binary=true --dirExpression='''/tmp/out''' --nameExpression=headers[file_name]" --deploy
But when i try to extend the shell script to call a webservice, e.g. by
#!/bin/bash
[ $# -ge 1 -a -f "$1" ] && input="$1" || input="-"
#cat $input
curl http://some.address/webservice -d 'output=xml' -d "text=$(cat $input)"
i always get an "Stream closed between payloads" exception from Spring XD. Even if i try to return an (existing) file by
#!/bin/bash
[ $# -ge 1 -a -f "$1" ] && input="$1" || input="-"
#cat $input
cat /tmp/output.xml
i get the same exception.
Related
threads=36
task() {
NEWNAME="${1/%.norm.vcf.gz/.snv.indel.vcf.gz}"
java -Xmx8g -jar /home/ubuntu/snpEff/SnpSift.jar annotate /home/ubuntu/data/spliceai_scores.raw.snv.hg19.vcf.gz $1 | java -Xmx8g -jar /home/ubuntu/snpEff/SnpSift.jar annotate /home/ubuntu/data/spliceai_scores.raw.indel.hg19.vcf.gz > $NEWNAME1
}
for file in $(ls *norm.vcf.gz)
do
if [ $(jobs -r | wc -l) -ge $threads ]; then
wait $(jobs -r -p | head -1)
fi
task "$file" &
done
wait
I am now learning NGS pipeline and I would like to annotate two DBs(snv,indel) using SnpSift. Currently I am using 36 threads CPU. Since there are two commands in task() function, annotate snv and indel, will my code wait $(jobs -r -p | head -1) work fine?
Are there any other recommendations to use CPU efficiently?
This question already has an answer here:
Formatting issues when unix file is sent to mail
(1 answer)
Closed 1 year ago.
I want to run linux command(ps aux --sort -rss | head -n 10) and display the result in table format in mail.
can somebody please suggest.
here is my script.
#!/usr/bin/ksh
current_dir=$(pwd)
script_dir=$(dirname $0)
if [ $script_dir = '.' ]; then
script_dir="$current_dir"
fi
source_dir=$PWD
cd $script_dir
FWK_TIMESTAMP=`date +%Y%m%d_%H:%M:%S`
REPORT_TS=`date -d '1 day ago' +'%m/%d/%Y'`
TILL_DATE=`date +%m/%d/%Y`
#javac -classpath mail1.4.4.jar:activation.jar SendEmail.java
if [ $? -ne 0 ];then
echo javac failed $?
exit 1
fi
outfile=test.txt
rm test.txt
echo "<h1> Top 10 Memory Consumed Processes from $REPORT_TS to $TILL_DATE</h1>" >> $outfile
ps aux --sort -rss | head -n 10 >> $outfile
to='abc#abc.com'
mail_server='-mail.smtp.host abc.com'
param="-to $to $mail_server -from abc#abc.com"
#-mail.debug
java -cp mail1.4.4.jar:activation.jar:. SendEmail $param -subject "Top Memory Consumed Processes" -file test.txt
As I am able to see in your code,
you are saving the results of
ps aux --sort -rss | head -n 10 >> $outfile
in the $outfile.
You can use <iframe src="your_filename_goes_here" width=200 height=400 frameborder=0 ></iframe>
I have a jar application that has several functions, one of which is to convert from HTML to XML. When I try to run a simple command such as:
java -jar lt4el-cmd.jar send -l en "l2:https://en.wikipedia.org/wiki/Personal_computer"
I get the following errors:
ERROR [Thread-1]: html2base/html2base-wrapper.sh: Too late for "-C" option at html2base/html2xml.pl line 1.
/tmp/lpc.30872.html: failed
cat: /tmp/lpc.30872.xml: No such file or directory
(LpcControl.java:229)
ERROR [Thread-1]: ana2ont/ana2ont.sh ${lang}: -:1: parser error : Document is empty
-:1: parser error : Start tag expected, '<' not found
Tokenization/tagging failed
^
-:1: parser error : Document is empty
unable to parse -
-:1: parser error : Document is empty
unable to parse -
(LpcControl.java:229)
ERROR [Thread-1]: Error in conversion: Error running conversion script (ana2ont/ana2ont.sh ${lang}): 6 (AppInterface.java:159)
This is the html2base-wrapper.sh script which seems to be where the first error occurs.
#!/bin/bash
if [ "$1" == "check" ]; then
. common.sh
check_binary perl || exit 1
check_perl_module HTML::TreeBuilder || exit 1
check_perl_module XML::LibXML || exit 1
check_binary tidy || exit 1
check_binary xmllint || exit 1
check_binary xsltproc || exit 1
exit
fi
cat >"$TMPDIR/lpc.$$.html"
html2base/html2base.sh -d html2base/LT4ELBase.dtd -x html2base/LT4ELBase.xslt -t "$TMPDIR/lpc.$$.html" >&2
cat "$TMPDIR/lpc.$$.xml";
rm -f "$TMPDIR"/lpc.$$.{ht,x}ml
And the html2base.sh script:
#!/bin/bash
#
# Sample script for automated HTML -> XML conversion
#
# Miroslav Spousta <spousta#ufal.mff.cuni.cz>
# $Id: html2base.sh 462 2008-03-17 08:37:14Z qiq $
basedir=`dirname $0`;
# constants
HTML2XML_BIN=${basedir}/html2xml.pl
ICONV_BIN=iconv
TIDY_BIN=tidy
XMLLINT_BIN=xmllint
XSLTPROC_BIN=xsltproc
DTDPARSE_BIN=dtdparse
TMPDIR=/tmp
# default values
VERBOSE=0
ENCODING=
TIDY=0
VALIDATE=0
DTD=${basedir}/LT4ELBase.dtd
XSLT=${basedir}/LT4ELBase.xslt
usage()
{
echo "usage: html2base.sh [options] file(s)"
echo "XML -> HTML conversion script."
echo
echo " -e, --encoding=charset Convert input files from encoding to UTF-8 (none)"
echo " -d, --dtd=file DTD to be used for conversion and validation ($DTD)"
echo " -x, --xslt=file XSLT to be applied after conversion ($XSLT)"
echo " -t, --tidy Run HTMLTidy on input HTML files"
echo " -a, --validate Validate output XML files"
echo " -v, --verbose Be verbose"
echo " -h, --help Print this usage"
exit 1;
}
OPTIONS=`getopt -o e:d:x:tahv -l encoding:,dtd:,xlst,tidy,validate,verbose,help -n 'convert.sh' -- "$#"`
if [ $? != 0 ]; then
usage;
fi
eval set -- "$OPTIONS"
while true ; do
case "$1" in
-e | --encoding) ENCODING=$2; shift 2 ;;
-d | --dtd) DTD=$2; shift 2 ;;
-x | --xslt) XSLT=$2; shift 2 ;;
-t | --tidy) TIDY=1; shift 1;;
-a | --validate) VALIDATE=1; shift 1;;
-v | --verbose) VERBOSE=1; shift 1 ;;
-h | --help) usage; shift 1 ;;
--) shift ; break ;;
*) echo "Internal error!" ; echo $1; exit 1 ;;
esac
done
if [ $# -eq 0 ]; then
usage;
fi
DTD_XML=`echo "$DTD"|sed -e 's/\.dtd/.xml/'`
if [ "$VERBOSE" -eq 1 ]; then
VERBOSE=--verbose
else
VERBOSE=
fi
# create $DTD_XML if necessary
if [ ! -f "$DTD_XML" ]; then
if ! $DTDPARSE_BIN $DTD -o $DTD_XML 2>/dev/null; then
echo "cannot run dtdparse, cannot create $DTD_XML";
exit 1;
fi;
fi
# process file by file
total=0
nok=0
while [ -n "$1" ]; do
file=$1;
if [ -n "$VERBOSE" ]; then
echo "Processing $file..."
fi
f="$file";
result=0;
if [ -n "$ENCODING" ]; then
$ICONV_BIN -f "$ENCODING" -t utf-8 "$f" -o "$file.xtmp"
result=$?
error="encoding error"
f=$file.xtmp
fi
if [ "$result" -eq 0 ]; then
if [ "$TIDY" = '1' ]; then
$TIDY_BIN --force-output 1 -q -utf8 >"$file.xtmp2" "$f" 2>/dev/null
f=$file.xtmp2
fi
out=`echo $file|sed -e 's/\.x\?html\?$/.xml/'`
if [ "$out" = "$file" ]; then
out="$out.xml"
fi
$HTML2XML_BIN --simplify-ws $VERBOSE $DTD_XML -o "$out" "$f"
result=$?
error="failed"
fi
if [ "$result" -eq 0 ]; then
$XSLTPROC_BIN --path `dirname $DTD` $XSLT "$out" |$XMLLINT_BIN --noblanks --format -o "$out.tmp1" -
result=$?
error="failed"
mv "$out.tmp1" "$out"
if [ "$result" -eq 0 -a "$VALIDATE" = '1' ]; then
tmp=`dirname $file`/$DTD
delete=0
if [ ! -f $tmp ]; then
cp $DTD $tmp
delete=1
fi
$XMLLINT_BIN --path `dirname $DTD` --valid --noout "$out"
result=$?
error="validation error"
if [ "$delete" -eq 1 ]; then
rm -f $tmp
fi
fi
fi
if [ "$result" -eq 0 ]; then
if [ -n "$VERBOSE" ]; then
echo "OK"
fi
else
echo "$file: $error "
nok=`expr $nok + 1`
fi
total=`expr $total + 1`
rm -f $file.xtmp $file.xtmp2
shift;
done
if [ -n "$VERBOSE" ]; then
echo
echo "Total: $total, failed: $nok"
fi
And the beginning part of the html2xml.pl file:
#!/usr/bin/perl -W -C
# Simple HTML to XML (subset of XHTML) conversion tool. Should always produce a
# valid XML file according to the output DTD file specified.
#
# Miroslav Spousta <spousta#ufal.mff.cuni.cz>
# $Id: html2xml.pl 461 2008-03-09 09:49:42Z qiq $
use HTML::TreeBuilder;
use HTML::Element;
use HTML::Entities;
use XML::LibXML;
use Getopt::Long;
use Data::Dumper;
use strict;
I can't seem to figure where the problem is. And what exactly does ERROR [Thread-1] mean?
Thanks
The error comes from having -C on the shebang (#!) line of a Perl script, but not passing the -C to perl. This type of error happens when someone does
perl html2base/html2xml.pl ...
instead of
html2base/html2xml.pl ...
The error was from the the html2xml.pl script as other users rightly mentioned. I'm running ubuntu 16.04.2 system which comes with a default perl 5.22 version. And as this post mentions, using the -C option (as from perl 5.10.1) on the #! line requires you to also specify it on the command line at execution time, which I wasn't sure how to do because I was running a jar file. I installed perlbrew, instead, which I used to get an earlier version of perl and modified my perl script to:
#!/usr/bin/path/to/perlbrew/perl -W -C
# Simple HTML to XML (subset of XHTML) conversion tool. Should always produce a
# valid XML file according to the output DTD file specified.
#
# Miroslav Spousta <spousta#ufal.mff.cuni.cz>
# $Id: html2xml.pl 461 2008-03-09 09:49:42Z qiq $
This might also come in handy in setting up shell scripts when using perlbrew.
Thanks for the efforts in contribution.
I define array #ID at the beginning of the code. I do different things with this list of ids, and therefore I define my $id; at the beginning only (I use strict; warnings).
Maybe I simplified too much to ask the question, so it end up being unclear.
Both $cv and $smo are $id dependent and they are all perl strings. Like this:
....
for $id (#ID) {
$cv = $htotal{$id};
$smo = $hsmote{$id};
system('java -Djava.util.Arrays.useLegacyMergeSort=true weka.classifiers.meta.FilteredClassifier -t Projects/proteins/$id_MSA/$id_.arff -x $cv -s 0 -p 1,2 -distribution \
-F "weka.filters.MultiFilter -F \" weka.filters.unsupervised.attribute.Remove -R 7,9\" -F \" weka.filters.unsupervised.attribute.RemoveType -T string\" \
-F \"weka.filters.supervised.instance.SMOTE -C 0 -K 5 -P $smo -S 1\"" -W weka.classifiers.functions.MultilayerPerceptron -- -L 0.3 -M 0.2 -N 500 -V 0 -S 0 -E 20 -H 0 \
> Projects/rbfs/$id.rbf');
}
...
To make sure there were no Weka errors, I tried, e.g. cv = 2, smo =100, id=P12345; and it worked fine, so it was an interpolation issue, as some of you mentioned.
From the solutions you've mentioned I tried double quotes + {} as #nlu suggested:
system('java [...] -t Projects/proteins/"${id}"_MSA/"${id}"_.arff -x "${cv}" -s 0 -p 1,2 -distribution [...] -F \"weka.filters.supervised.instance.SMOTE -C 0 -K 5 -P "${smo}" -S 1\"" [...] > Projects/rbfs/"${id}".rbf');
But didn't worked, did I write it wrong?
Same with this (replacing system() for `` ) didn't work:
java [...] -t Projects/150400_GSupdate/proteins/${id}_MSA/${id}.arff -x ${cv} -s 0 -p 1,2 [...] -F \"weka.filters.supervised.instance.SMOTE -C 0 -K 5 -P ${smo} -S [...] > Projects/150400_GSupdate/rbfs/${id}.rbf;
What finally has worked for me is string concatenation (as #Matt suggested) but I'm sure that all other options should be fine too, although don't know why not.
It's not clear if the other variables are shell variables other than $id? If you want just $id to be substituted the easiest way is to just use string concatenation:
for $id (#ID) {
system('java -Djava.util.Arrays.useLegacyMergeSort=true weka.classifiers.meta.FilteredClassifier -t $PATH/proteins/$id_MSA/$id_.arff -x $cv -s 0 -p 1,2 -distribution \
-F "weka.filters.MultiFilter -F \" weka.filters.unsupervised.attribute.Remove -R 7,9\" -F \" weka.filters.unsupervised.attribute.RemoveType -T string\" \
-F \"weka.filters.supervised.instance.SMOTE -C 0 -K 5 -P $smo -S 1\"" -W weka.classifiers.functions.MultilayerPerceptron -- -L 0.3 -M 0.2 -N 500 -V 0 -S 0 -E 20 -H 0 \
> $PATH/rbfs/' . $id . '.rbf');
}
If you wanted all the $something to be treated as if they were perl variables probably the easiest way given you need to use " and \" inside the system string is to replace system() with the `` operator which basically does the same thing as system.
for $id (#ID) {
`java -Djava.util.Arrays.useLegacyMergeSort=true weka.classifiers.meta.FilteredClassifier -t $PATH/proteins/$id_MSA/$id_.arff -x $cv -s 0 -p 1,2 -distribution \
-F "weka.filters.MultiFilter -F \" weka.filters.unsupervised.attribute.Remove -R 7,9\" -F \" weka.filters.unsupervised.attribute.RemoveType -T string\" \
-F \"weka.filters.supervised.instance.SMOTE -C 0 -K 5 -P $smo -S 1\"" -W weka.classifiers.functions.MultilayerPerceptron -- -L 0.3 -M 0.2 -N 500 -V 0 -S 0 -E 20 -H 0 \
> $PATH/rbfs/$id.rbf`;
}
When you call a function in Perl, like system in your case, you can use interpolation to have variables inside the strings substituted.
You need to pass the string in double quotes for this to happen, so first you have to replace your surrounding single quotes with double quotes.
Besides, for disambiguation of any variable names appearing in the string, you should write them like this:
"${id}"
to avoid confusing variable names like i.e. id and idsomething.
Documented here:
http://perldoc.perl.org/perldata.html#Scalar-value-constructors
Despite my luddite tendencies, I now have a phone with Java support - but no Flash support. I also have a copy of Macromedia Flash MX2004, though I'm unlikely to upgrade any time soon.
What I'd like to be able to do is develop some content (including vector animations) in Flash, then use those resources in a Java Micro Edition application. I don't need all features of Flash - in particular, I don't care about ActionScript support. But I do want to be able to load a SWF file (or, perhaps better, an alternative file format that can be generated using a converter tool), and to be able to display animations and use other resources (particularly play sounds) from in that file.
Is there a good library and toolkit to support this kind of thing? Obviously (from the MX2004) it doesn't need to be completely up to date.
On knowledge level - I've been a programmer for decades, and my everyday language these days is C++. However, I have a very limited knowledge of Java, and virtually no knowledge (yet) of Micro Edition and its libraries.
I've already heard of Flash to J2ME converters, but so far as I can see they generate complete applications in one step, rather than treating the SWF file as a source of resources to be controlled from separately written Java code.
EDIT
I get the feeling that this is (with slight modifications) probably quite easy. Java Mobile Edition supports SVG vector graphics. SVG supports animations. There are (I'm pretty certain) ways to convert flash animations to SVG - probably a simple export-to-SVG in the application, though I've not checked.
This in itself doesn't give me a convenient bundle-of-media resources file format, but that's a relatively simple problem to solve, so long as there's a way to "load" SVG and other media files from some kind of non-file stream class that gets its data in turn from the bundle-of-media file.
I've never used Java ME before, so I won't be able to help on that side, but I use actionscript/flash on a daily basis.
The 'easiest' thing I can think of is a 2 step process:
Export your animation as vector sequence via File > Export > Export Movie and choosing the right format (e.g. .ai/.eps/.dxf).
Convert the vector sequence to svg. Inkscape has a few handy SVG conversion tools.
A long winded way would be to write a JSFL script in Flash MX 2004.
You would traverse the shapes for each frame, then write the path data to SVG.
Another slightly different way would be to export the vector sequence as explained above (unfortunately there is no JSFL functionality to automate that), then from JSFL read and loop through each file, parse it and write an SVG.
The only advantage this would give you though is not having to install Inkscape and you wouldn't need to switch to another application.
I wouldn't recommend this though because:
You would need to write a parser (dxf/eps might be the simplest)
You will need to make an SVG and you only have Strings at your disposal (E4X XML support was added in Flash CS3)
I'm not saying it's impossible, it just seems impractical.
Found this thread on the Inkscape forum sharing a bash script that
extracts SWF objects to an SVG file
using SWFTools, but haven't tried that yet. For reference, here is hadi's script:
#!/bin/bash
#USAGE ./swf2svg.sh /path/to/file.swf > output.svg
FILE=$1;
DUMP="dump.txt"
echo '<?xml version="1.0" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg width="100%" height="100%" version="1.1"
xmlns="http://www.w3.org/2000/svg">
';
swfdump -s $FILE > $DUMP
fillCols=();
lineCols=();
lineWidth=();
FILLREGEX="[0-9]+(\s*)SOLID [0-f]{8}";
LINEREGEX="[0-9]+(\s*)[0-9]\.[0-9]{0,2} [0-f]{8}";
lastStartPoint="";
pathClosedTag="";
firstGroup="TRUE";
firstPath="TRUE";
cat $DUMP | while read line
do
#Remove ( and )
line=`echo $line | sed "s/[()]//g"`
#tmp=`echo $line | egrep -o "DEFINE(SHAPE|SPRITE)"`;
tmp=`echo $line | egrep -o "DEFINE(SHAPE|SPRITE)[0-9]? defines id [0-9]+"`;
if [ "$tmp" != "" ]
then
if [ "$firstGroup" == "TRUE" ]
then
firstGroup="FALSE";
else
if [ "$firstPath" == "FALSE" ]
then
if [ "$lastStartPoint" != "" ]
then
if [ "$lastStartPoint" == "$curPoint" ]
then
pathClosedTag="Z";
fi
fi
lastStartPoint=$curPoint;
echo $pathClosedTag'" />';
fi;
firstPath="TRUE";
echo '</g>';
fi
id=`echo $tmp | awk {'print $4'}`
echo '<g id="'$id'">';
fillCols=();
lineCols=();
lineWidth=();
fi
tmp=`echo $line | egrep -o "($FILLREGEX)?((\s*)$LINEREGEX)?"`;
if [ "$tmp" != "" ]
then
fillInx=`echo $tmp | egrep -o "$FILLREGEX" | awk {'print $1'}`;
fillCol=`echo $tmp | egrep -o "$FILLREGEX" | awk {'print $3'}`;
if [ "$fillCol" != "" ]
then
fillCols[$fillInx]=$fillCol;
fi
lineInx=`echo $tmp | egrep -o "$LINEREGEX" | awk {'print $1'}`;
lineWth=`echo $tmp | egrep -o "$LINEREGEX" | awk {'print $2'}`;
lineCol=`echo $tmp | egrep -o "$LINEREGEX" | awk {'print $3'}`;
if [ "$lineCol" != "" ]
then
lineCols[$lineInx]=$lineCol;
lineWidth[$lineInx]=$lineWth;
fi
fi
tmp=`echo $line | awk {'print $6'}`;
if [ "$tmp" == "lineTo" ]
then
echo $line | awk {'print "L"$7" "$8'}
fi
if [ "$tmp" == "moveTo" ]
then
curPoint=`echo $line | awk {'print $9" "$10'}`;
if [ "$lastStartPoint" != "" ]
then
if [ "$lastStartPoint" == "$curPoint" ]
then
pathClosedTag="Z";
fi
fi
lastStartPoint=$curPoint;
if [ "$firstPath" == "TRUE" ]
then
firstPath="FALSE";
else
echo $pathClosedTag'" />';
fi;
#Remove : and /
line=`echo $line | sed "s/[:/]/ /g"`
fInx=`echo $line | awk '{printf "%d", $4}'`;
lInx=`echo $line | awk '{printf "%d", $6}'`;
stl="";
val=${fillCols[$fInx]:0:6};
if [ $fInx -gt 0 -a "$val" != "" ]
then
stl="fill:#$val;";
fi
val=${lineCols[$lInx]:0:6};
if [ $lInx -gt 0 -a "$val" != "" ]
then
stl=$stl"stroke:#$val;";
val=${lineWidth[$lInx]};
if [ "$val" != "" ]
then
stl=$stl"stroke-width:$val;";
fi
fi
echo '<path style="'$stl'" d="';
echo $line | awk {'print "M"$9" "$10'}
fi
if [ "$tmp" == "splineTo" ]
then
echo $line | awk {'print "Q"$7" "$8" "$9" "$10'}
fi
done
echo 'Z" />';
echo '</g>';
echo '</svg>';
If anybody else using a more recent version of Flash (like CS4 or CS5) reads this, there is a Flash 2 SVG extension available.