We Keep Coding

"Too late for "-C" option" error With Perl and Shell scripts

I have a jar application that has several functions, one of which is to convert from HTML to XML. When I try to run a simple command such as:
java -jar lt4el-cmd.jar send -l en "l2:https://en.wikipedia.org/wiki/Personal_computer"
I get the following errors:
ERROR [Thread-1]: html2base/html2base-wrapper.sh: Too late for "-C" option at html2base/html2xml.pl line 1.
/tmp/lpc.30872.html: failed
cat: /tmp/lpc.30872.xml: No such file or directory
(LpcControl.java:229)
ERROR [Thread-1]: ana2ont/ana2ont.sh ${lang}: -:1: parser error : Document is empty
-:1: parser error : Start tag expected, '<' not found
Tokenization/tagging failed
^
-:1: parser error : Document is empty
unable to parse -
-:1: parser error : Document is empty
unable to parse -
(LpcControl.java:229)
ERROR [Thread-1]: Error in conversion: Error running conversion script (ana2ont/ana2ont.sh ${lang}): 6 (AppInterface.java:159)
This is the html2base-wrapper.sh script which seems to be where the first error occurs.
#!/bin/bash
if [ "$1" == "check" ]; then
. common.sh
check_binary perl || exit 1
check_perl_module HTML::TreeBuilder || exit 1
check_perl_module XML::LibXML || exit 1
check_binary tidy || exit 1
check_binary xmllint || exit 1
check_binary xsltproc || exit 1
exit
fi
cat >"$TMPDIR/lpc.$$.html"
html2base/html2base.sh -d html2base/LT4ELBase.dtd -x html2base/LT4ELBase.xslt -t "$TMPDIR/lpc.$$.html" >&2
cat "$TMPDIR/lpc.$$.xml";
rm -f "$TMPDIR"/lpc.$$.{ht,x}ml
And the html2base.sh script:
#!/bin/bash
#
# Sample script for automated HTML -> XML conversion
#
# Miroslav Spousta <spousta#ufal.mff.cuni.cz>
# $Id: html2base.sh 462 2008-03-17 08:37:14Z qiq $
basedir=`dirname $0`;
# constants
HTML2XML_BIN=${basedir}/html2xml.pl
ICONV_BIN=iconv
TIDY_BIN=tidy
XMLLINT_BIN=xmllint
XSLTPROC_BIN=xsltproc
DTDPARSE_BIN=dtdparse
TMPDIR=/tmp
# default values
VERBOSE=0
ENCODING=
TIDY=0
VALIDATE=0
DTD=${basedir}/LT4ELBase.dtd
XSLT=${basedir}/LT4ELBase.xslt
usage()
{
echo "usage: html2base.sh [options] file(s)"
echo "XML -> HTML conversion script."
echo
echo " -e, --encoding=charset Convert input files from encoding to UTF-8 (none)"
echo " -d, --dtd=file DTD to be used for conversion and validation ($DTD)"
echo " -x, --xslt=file XSLT to be applied after conversion ($XSLT)"
echo " -t, --tidy Run HTMLTidy on input HTML files"
echo " -a, --validate Validate output XML files"
echo " -v, --verbose Be verbose"
echo " -h, --help Print this usage"
exit 1;
}
OPTIONS=`getopt -o e:d:x:tahv -l encoding:,dtd:,xlst,tidy,validate,verbose,help -n 'convert.sh' -- "$#"`
if [ $? != 0 ]; then
usage;
fi
eval set -- "$OPTIONS"
while true ; do
case "$1" in
-e | --encoding) ENCODING=$2; shift 2 ;;
-d | --dtd) DTD=$2; shift 2 ;;
-x | --xslt) XSLT=$2; shift 2 ;;
-t | --tidy) TIDY=1; shift 1;;
-a | --validate) VALIDATE=1; shift 1;;
-v | --verbose) VERBOSE=1; shift 1 ;;
-h | --help) usage; shift 1 ;;
--) shift ; break ;;
*) echo "Internal error!" ; echo $1; exit 1 ;;
esac
done
if [ $# -eq 0 ]; then
usage;
fi
DTD_XML=`echo "$DTD"|sed -e 's/\.dtd/.xml/'`
if [ "$VERBOSE" -eq 1 ]; then
VERBOSE=--verbose
else
VERBOSE=
fi
# create $DTD_XML if necessary
if [ ! -f "$DTD_XML" ]; then
if ! $DTDPARSE_BIN $DTD -o $DTD_XML 2>/dev/null; then
echo "cannot run dtdparse, cannot create $DTD_XML";
exit 1;
fi;
fi
# process file by file
total=0
nok=0
while [ -n "$1" ]; do
file=$1;
if [ -n "$VERBOSE" ]; then
echo "Processing $file..."
fi
f="$file";
result=0;
if [ -n "$ENCODING" ]; then
$ICONV_BIN -f "$ENCODING" -t utf-8 "$f" -o "$file.xtmp"
result=$?
error="encoding error"
f=$file.xtmp
fi
if [ "$result" -eq 0 ]; then
if [ "$TIDY" = '1' ]; then
$TIDY_BIN --force-output 1 -q -utf8 >"$file.xtmp2" "$f" 2>/dev/null
f=$file.xtmp2
fi
out=`echo $file|sed -e 's/\.x\?html\?$/.xml/'`
if [ "$out" = "$file" ]; then
out="$out.xml"
fi
$HTML2XML_BIN --simplify-ws $VERBOSE $DTD_XML -o "$out" "$f"
result=$?
error="failed"
fi
if [ "$result" -eq 0 ]; then
$XSLTPROC_BIN --path `dirname $DTD` $XSLT "$out" |$XMLLINT_BIN --noblanks --format -o "$out.tmp1" -
result=$?
error="failed"
mv "$out.tmp1" "$out"
if [ "$result" -eq 0 -a "$VALIDATE" = '1' ]; then
tmp=`dirname $file`/$DTD
delete=0
if [ ! -f $tmp ]; then
cp $DTD $tmp
delete=1
fi
$XMLLINT_BIN --path `dirname $DTD` --valid --noout "$out"
result=$?
error="validation error"
if [ "$delete" -eq 1 ]; then
rm -f $tmp
fi
fi
fi
if [ "$result" -eq 0 ]; then
if [ -n "$VERBOSE" ]; then
echo "OK"
fi
else
echo "$file: $error "
nok=`expr $nok + 1`
fi
total=`expr $total + 1`
rm -f $file.xtmp $file.xtmp2
shift;
done
if [ -n "$VERBOSE" ]; then
echo
echo "Total: $total, failed: $nok"
fi
And the beginning part of the html2xml.pl file:
#!/usr/bin/perl -W -C
# Simple HTML to XML (subset of XHTML) conversion tool. Should always produce a
# valid XML file according to the output DTD file specified.
#
# Miroslav Spousta <spousta#ufal.mff.cuni.cz>
# $Id: html2xml.pl 461 2008-03-09 09:49:42Z qiq $
use HTML::TreeBuilder;
use HTML::Element;
use HTML::Entities;
use XML::LibXML;
use Getopt::Long;
use Data::Dumper;
use strict;
I can't seem to figure where the problem is. And what exactly does ERROR [Thread-1] mean?
Thanks

The error comes from having -C on the shebang (#!) line of a Perl script, but not passing the -C to perl. This type of error happens when someone does
perl html2base/html2xml.pl ...
instead of
html2base/html2xml.pl ...

The error was from the the html2xml.pl script as other users rightly mentioned. I'm running ubuntu 16.04.2 system which comes with a default perl 5.22 version. And as this post mentions, using the -C option (as from perl 5.10.1) on the #! line requires you to also specify it on the command line at execution time, which I wasn't sure how to do because I was running a jar file. I installed perlbrew, instead, which I used to get an earlier version of perl and modified my perl script to:
#!/usr/bin/path/to/perlbrew/perl -W -C
# Simple HTML to XML (subset of XHTML) conversion tool. Should always produce a
# valid XML file according to the output DTD file specified.
#
# Miroslav Spousta <spousta#ufal.mff.cuni.cz>
# $Id: html2xml.pl 461 2008-03-09 09:49:42Z qiq $
This might also come in handy in setting up shell scripts when using perlbrew.
Thanks for the efforts in contribution.

powershell for each file build a string to execute

I need to schedule a windows job to loop thru thousand of files and execute a command with the command options, the file name, and extension.
These files have an extension of *.xls in directory c:\proj. For example, one of the file is myImportantFile0001.xls, I would execute the following in powershell, manually:
PS> java.exe -classpath "C:\PROGRA~2\my1.0.2\lib/*" com.my.madsci -v --collector-url https://api.fun.my.com/ --oauth-url https://api.my.com/oauth/access --type /fileName:string/discovery:string --key /fileName:"myImportantFile0001/discovery:discovery" "C:\proj\myImportantFile0001_xls.txt"
couple of problems I've got are:
the command itself has double quotes,
the file name and extensions need to be found and converted
execute a java within powershell
So how can I loop thru the thousand of files to build a string to execute the command using powershell?

does below script helps you? i think it just strings combination.
$Commands = ls C:\Temp\*.xlsx | %{
"java.exe -classpath `"C:\PROGRA~2\my1.0.2\lib/*`" com.my.madsci -v --collector-url https://api.fun.my.com/ --oauth-url https://api.my.com/oauth/access --type /fileName:string/discovery:string --key /fileName:`"$($_.BaseName)/discovery:discovery`" `"C:\proj\$($_.BaseName)_xls.txt`""
}
$Commands
exit # remove the line if you want to execute
$Commands | %{
& $_
}

What about
PS> ls C:\Temp\*.xlsx | foreach-object {
"java.exe -classpath `"C:\PROGRA~2\my1.0.2\lib/*`" com.my.madsci -v --collector-url https://api.fun.my.com/ --oauth-url https://api.my.com/oauth/access --type /fileName:string/discovery:string --key /fileName:`"$($_.name)/discovery:discovery`" `"C:\proj\$($_.name)_xls.txt`""
}

You can do it like this:
$files = Get-ChildItem "C:\temp" -Filter "*.xls"
foreach ($file in $files)
{
$command = "java.exe -classpath ""C:\PROGRA~2\my1.0.2\lib/*"" com.my.madsci "
$command = $command + "-v --collector-url https://api.fun.my.com/ "
$command = $command + "--oauth-url https://api.my.com/oauth/access "
$command = $command + "--type /fileName:string/discovery:string "
$command = $command + ("--key /fileName:""{0}/discovery:discovery"" " -f $file.BaseName)
$command = $command + ("""C:\proj\{0}_xls.txt""" -f $file.BaseName)
Invoke-Expression $command
}

Additionally, you can also schedule it from powershell itself (Works with PS 3.0 and above)
$trigger = New-JobTrigger -Daily -At "9:00 AM"
$scriptPath = "C:\FilePath\myScript.ps1"
Register-ScheduledJob -Name "MyJob" -FilePath $scriptPath -Trigger $trigger

Spring XD: "Stream closed between payloads" exception for shell processor

The simple shell processor takes input data and echos it back:
nano /tmp/echo.sh
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
#!/bin/bash
[ $# -ge 1 -a -f "$1" ] && input="$1" || input="-"
cat $input
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
It works fine when deployed to Spring XD by:
stream create test --definition "file --mode=contents --outputType=text/plain | shell --command='bash /tmp/echo.sh' | b:file --binary=true --dirExpression='''/tmp/out''' --nameExpression=headers[file_name]" --deploy
But when i try to extend the shell script to call a webservice, e.g. by
#!/bin/bash
[ $# -ge 1 -a -f "$1" ] && input="$1" || input="-"
#cat $input
curl http://some.address/webservice -d 'output=xml' -d "text=$(cat $input)"
i always get an "Stream closed between payloads" exception from Spring XD. Even if i try to return an (existing) file by
#!/bin/bash
[ $# -ge 1 -a -f "$1" ] && input="$1" || input="-"
#cat $input
cat /tmp/output.xml
i get the same exception.

Perl strings inside system call to Java

I define array #ID at the beginning of the code. I do different things with this list of ids, and therefore I define my $id; at the beginning only (I use strict; warnings).
Maybe I simplified too much to ask the question, so it end up being unclear.
Both $cv and $smo are $id dependent and they are all perl strings. Like this:
....
for $id (#ID) {
$cv = $htotal{$id};
$smo = $hsmote{$id};
system('java -Djava.util.Arrays.useLegacyMergeSort=true weka.classifiers.meta.FilteredClassifier -t Projects/proteins/$id_MSA/$id_.arff -x $cv -s 0 -p 1,2 -distribution \
-F "weka.filters.MultiFilter -F \" weka.filters.unsupervised.attribute.Remove -R 7,9\" -F \" weka.filters.unsupervised.attribute.RemoveType -T string\" \
-F \"weka.filters.supervised.instance.SMOTE -C 0 -K 5 -P $smo -S 1\"" -W weka.classifiers.functions.MultilayerPerceptron -- -L 0.3 -M 0.2 -N 500 -V 0 -S 0 -E 20 -H 0 \
> Projects/rbfs/$id.rbf');
}
...
To make sure there were no Weka errors, I tried, e.g. cv = 2, smo =100, id=P12345; and it worked fine, so it was an interpolation issue, as some of you mentioned.
From the solutions you've mentioned I tried double quotes + {} as #nlu suggested:
system('java [...] -t Projects/proteins/"${id}"_MSA/"${id}"_.arff -x "${cv}" -s 0 -p 1,2 -distribution [...] -F \"weka.filters.supervised.instance.SMOTE -C 0 -K 5 -P "${smo}" -S 1\"" [...] > Projects/rbfs/"${id}".rbf');
But didn't worked, did I write it wrong?
Same with this (replacing system() for `` ) didn't work:
java [...] -t Projects/150400_GSupdate/proteins/${id}_MSA/${id}.arff -x ${cv} -s 0 -p 1,2 [...] -F \"weka.filters.supervised.instance.SMOTE -C 0 -K 5 -P ${smo} -S [...] > Projects/150400_GSupdate/rbfs/${id}.rbf;
What finally has worked for me is string concatenation (as #Matt suggested) but I'm sure that all other options should be fine too, although don't know why not.

It's not clear if the other variables are shell variables other than $id? If you want just $id to be substituted the easiest way is to just use string concatenation:
for $id (#ID) {
system('java -Djava.util.Arrays.useLegacyMergeSort=true weka.classifiers.meta.FilteredClassifier -t $PATH/proteins/$id_MSA/$id_.arff -x $cv -s 0 -p 1,2 -distribution \
-F "weka.filters.MultiFilter -F \" weka.filters.unsupervised.attribute.Remove -R 7,9\" -F \" weka.filters.unsupervised.attribute.RemoveType -T string\" \
-F \"weka.filters.supervised.instance.SMOTE -C 0 -K 5 -P $smo -S 1\"" -W weka.classifiers.functions.MultilayerPerceptron -- -L 0.3 -M 0.2 -N 500 -V 0 -S 0 -E 20 -H 0 \
> $PATH/rbfs/' . $id . '.rbf');
}
If you wanted all the $something to be treated as if they were perl variables probably the easiest way given you need to use " and \" inside the system string is to replace system() with the `` operator which basically does the same thing as system.
for $id (#ID) {
`java -Djava.util.Arrays.useLegacyMergeSort=true weka.classifiers.meta.FilteredClassifier -t $PATH/proteins/$id_MSA/$id_.arff -x $cv -s 0 -p 1,2 -distribution \
-F "weka.filters.MultiFilter -F \" weka.filters.unsupervised.attribute.Remove -R 7,9\" -F \" weka.filters.unsupervised.attribute.RemoveType -T string\" \
-F \"weka.filters.supervised.instance.SMOTE -C 0 -K 5 -P $smo -S 1\"" -W weka.classifiers.functions.MultilayerPerceptron -- -L 0.3 -M 0.2 -N 500 -V 0 -S 0 -E 20 -H 0 \
> $PATH/rbfs/$id.rbf`;
}

When you call a function in Perl, like system in your case, you can use interpolation to have variables inside the strings substituted.
You need to pass the string in double quotes for this to happen, so first you have to replace your surrounding single quotes with double quotes.
Besides, for disambiguation of any variable names appearing in the string, you should write them like this:
"${id}"
to avoid confusing variable names like i.e. id and idsomething.
Documented here:
http://perldoc.perl.org/perldata.html#Scalar-value-constructors

Uninstalling Java 6 and Reinstalling Java 7 using Powershell

I am using the following Powershell script. The first half (uninstall) works flawlessly. The second half (install) only works if I allow user input. Can anyone provide some assistance? Here is the script: (sorry for poor formatting)
#uninstall
$java = Get-WmiObject -Class win32_product | where { $_.Name -like "*Java*"}
$msiexec = "C:\Windows\system32\msiexec.exe";
$msiexecargs = '/x "$($app.IdentifyingNumber)" /qn /norestart'
if ($java -ne $null)
{
foreach ($app in $java)
{
write-host $app.LocalPackage
write-host $app.IdentifyingNumber
C:\Windows\system32\cmd.exe /c "C:\Windows\system32\msiexec.exe /x $($app.IdentifyingNumber) /qn"
Start-Process -FilePath $msiexec -Arg $msiexecargs -Wait -Passthru
[Diagnostics.Process]::Start($msiexec, $msiexecargs);
}
}
if ($java -ne $null)
{
foreach ($app in $java)
{
write-host $app.LocalPackage
write-host $app.IdentifyingNumber
C:\Windows\system32\cmd.exe /c
"C:\Windows\system32\msiexec.exe /x $($app.IdentifyingNumber) /qn"
Start-Process -FilePath $msiexec -Arg $msiexecargs -Wait -Passthru
[Diagnostics.Process]::Start($msiexec, $msiexecargs);
}
}
function Get-ScriptDirectory{
$Invocation = (Get-Variable MyInvocation -Scope 1).Value
try {
Split-Path $Invocation.MyCommand.Path -ea 0
}
catch {
Write-Warning 'You need to call this function from within a saved script.'
}
}
function Get-Architecture{
return $(gwmi win32_operatingsystem).OSArchitecture
}
$Path = Get-ScriptDirectory
#Close all instances of IE, Firefox, & Chrome
Get-Process | where {$_.ProcessName -match "iexplore"} | Stop-Process -Force
Get-Process | where {$_.ProcessName -match "chrome"} | Stop-Process -Force
Get-Process | where {$_.ProcessName -match "firefox"} | Stop-Process -Force
#Install
Start-Process -FilePath "msiexec.exe" -ArgumentList "/i "C:\temp\jre1.7.0_17.msi" ""/log "c:\temp\javainst.log " -Credential $cred -wait
#Also Install the 64-bit JRE if on a 64 workstation
if(Get-Architecture -match "64")
{
$cred = Get-Credential
Start-Process -FilePath "msiexec.exe" -ArgumentList "/i "C:\temp\jre1.7.0_17 (x64).msi" ""/log c:\temp\javainst.log " -Credential $cred -wait
}
#Import reg keys to disable auto updating
reg import "C:\temp\JavaUpdate.reg"{
}

#uninstall everything Java
Get-WmiObject -Class win32_product | ? {$_.Name -like "*Java*"} | % {msiexec /x "$($_.IdentifyingNumber)" /qn | Out-Null}
#The Out-Null waits for the command to finish
#If you have made a Java MSI use this
msiexec /i $pathtomsi /qn
#If you only have the exe you'll need to look up the Command Line Interface (CLI) for Java
$cmd = "$pathtoexe /s"
cmd /c $cmd
As for your script change the #Install line to:
Start-Process -FilePath 'msiexec.exe' -ArgumentList '/i "C:\temp\jre1.7.0_17.msi" /log "c:\temp\javainst.log" /qn' -Credential $cred -wait
Best Practice use mostly single quotes