We Keep Coding

Apache Commons Exec Problems Handling Regex Argument?

I'm trying to use Apache Commons Exec to run a git command which uses a regex.
When I form my CommandLine and print it out it looks like this:
[git, --no-pager, grep, --line-number, --untracked, --extended-regexp, "^\s*public void\s+(testFindByAdAccount).*", --, *Test.java]
However when I execute this, git returns no results, resulting in an exit code 1.
When I run this command manually though, it returns plenty of results and succeeds. Changing the --extended-regexp argument to just a string like testFindByAdAccount does yield results when run via Exec, so I think Apache Commons is doing something to the regexp argument making it invalid. Any ideas what is going on?
EDIT: Adding a reproducible example
Clone https://github.com/ragurney/min-example
Run gradlew shadowJar to produce jar file for project
Run the app with java -jar app/build/libs/app-all.jar
Note the output which shows the command printed fails with an exit code 1 (because there are no results returned by the git command)
$ java -jar app/build/libs/app-all.jar
HELLOOOOOO
WD::: null
[git, --no-pager, grep, --line-number, --untracked, --extended-regexp, "^\s*public void\s+(testAppHasAGreeting)\(\).*", --, *Test.java]
WD::: /Users/rgurney/Src/personal/min-example
Exception in thread "main" java.lang.RuntimeException: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
at min.example.App.lambda$runCommand$1(App.java:74)
at io.vavr.control.Try.getOrElseThrow(Try.java:748)
Running the command manually does produce expected results:
$ git --no-pager grep --line-number --untracked --extended-regexp "^\s*public void\s+(testAppHasAGreeting)\(\).*" -- "*Test.java"
app/src/test/java/min/example/AppTest.java:11: public void testAppHasAGreeting() {

I got a clue as to what's going on here when the sample you provided worked just fine on my Windows laptop but failed on my Linux desktop.
Once I made sure the git version wasn't the culprit (tested several versions between 2.17 and 2.39 on both machines), I figured the difference must be in the way different shells handle quoting. Specifically, the only argument here that has any potential quoting issues is the regex ("^\s*public void\s+(testFindByAdAccount).*"), which is added to the command line by commandLine.addArgument(regex);.
addArgument may look innocuous, but under the hood, it allows the CommandLine to handle the quoting itself (i.e., addArgument(String argument) calls addArgument(String argument, true). Since you've handled the quoting yourself, you should not allow the CommandLine to handle the quoting, and should explicitly call it with the second argument false. i.e.:
public static List<String> grep(String regex, String filePattern, String wd) {
CommandLine commandLine = CommandLine.parse("git");
commandLine.addArgument("--no-pager");
commandLine.addArgument("grep");
commandLine.addArgument("--line-number");
commandLine.addArgument("--untracked");
commandLine.addArgument("--extended-regexp");
commandLine.addArgument(regex, false);
// Here -----------------------^
commandLine.addArgument("--");
commandLine.addArgument(filePattern);
System.out.println(commandLine);
return List.of(runCommand(commandLine, wd).split("\n"));
}
This takes the quote-handling logic away and ensures the same code runs smoothly both on Windows and Linux (at least those I've tested).

Using SnpSift, only 0.52% of VCF is annotated by dbsnp database

I generated a coordinate sorted vcf file from a cram using the following commands:
samtools sort -# 10 -o /output/sorted.cram
samtools index -# 10 /output/sorted.cram
bcftools mpileup -f reference.fa -r chrz:zzzz-zzzzx -a INFO/AD,FORMAT/DP --threads 10 -O v -o /output/mpileup.vcf /input/sorted.cram
I am trying to annotate the coordinate sorted vcf file (ref genome Hg38) with snpsift. I am using the following command:
java -jar SnpSift.jar annotate -v /dbsnp/file.vcf.gz /input/mpileup.vcf > /output/annotated.vcf
I have downloaded the dbsnp vcf file and tab index here: ftp://ftp.ncbi.nih.gov/snp/organisms/human_9606_b151_GRCh38p7/VCF/GATK/
However, only 0.52% of the vcf is being annotated... This seems strange. Additionally, when I try to use the ensemble web interface (https://useast.ensembl.org/Multi/Tools/VEP?db=core) to annotate my vcf I get the error "invalid input". This leads me believe something is wrong with my vcf file? I am only trying to annotate one gene, is it normal for only 0.52% of a gene to be annotated by dbsnp? Thank you in advance for any assistance!
Update! If use bcftools mpileup | bcftools call --variants-only then the ensembl tool works. Additionally, this artificially increases the % of SNPs annotated.

Trimmomatic not acknowleding commands over Linux cluster

I am trying to use the program Trimmomatic to removed adapter sequences from an Illumina paired-end read over a computer cluster. While I can get the program to open, it will either not acknowledge the commands I enter or will return an error message. I have tried all kinds of permutations of the input commands without success. Examples of input code and error messages are below
Code:
java -classpath /*filepath*/Trimmomatic-0.32/trimmomatic-0.32.jar org.usadellab.trimmomatic.TrimmomaticPE \
-phred33 -trimlog /Results/log.txt \
~/*filepath*/data_R1.fq ~/*filepath*/data_R2.fq \
ILLUMINACLIP:/*filepath*/Trimmomatic-0.32/adapters/TruSeq3-PE-2.fa:2:30:10:3:"true"
Results: (the o/s seems to find and execute the software, but is not feeding in the command; I get the same result if I use the java -jar option for executing Trimmomatic)
TrimmomaticPE [-threads <threads>] [-phred33|-phred64] [-trimlog <trimLogFile>] [-basein <inputBase> | <inputFile1> <inputFile2>] [-baseout <outputBase> | <outputFile1P> <outputFile1U> <outputFile2P> <outputFile2U>] <trimmer1>...
Code: (If I add in the command PE immediately before all other commands, the program executes and can find the fasta file containing the adapter sequences, but then searches for and cannot fund a file called 'PE')
java -classpath /*filepath*/trimmomatic-0.32.jar org.usadellab.trimmomatic.TrimmomaticPE \
PE -phred33 -trimlog /Results/log.txt \
~/*filepath*/data_R1.fq ~/*filepath*/data_R2.fq \
ILLUMINACLIP:/*filepath*/Trimmomatic-0.32/adapters/TruSeq3-PE-2.fa:2:30:10:3:"true"
Results: (Programs rus and finds the fasta file of adapter sequences, but then fails to execute. Why is it looking for a PE file?)
TrimmomaticPE: Started with arguments: PE -phred33 -trimlog /Results/log.txt /*filepath*/data_R1.fq /*filepath*/data_R2.fq ILLUMINACLIP:/*filepath*/Trimmomatic-0.32/adapters/TruSeq3-PE-2.fa:2:30:10:3:true
Multiple cores found: Using 12 threads
Using PrefixPair: 'TACACTCTTTCCCTACACGACGCTCTTCCGATCT' and 'GTGACTGGAGTTCAGACGTGTGCTCTTCCGATCT'
Using Long Clipping Sequence: 'AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC'
Using Long Clipping Sequence: 'TACACTCTTTCCCTACACGACGCTCTTCCGATCT'
Using Long Clipping Sequence: 'GTGACTGGAGTTCAGACGTGTGCTCTTCCGATCT'
Using Long Clipping Sequence: 'AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTA'
ILLUMINACLIP: Using 1 prefix pairs, 4 forward/reverse sequences, 0 forward only sequences, 0 reverse only sequences
Exception in thread "main" java.io.FileNotFoundException: PE (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at org.usadellab.trimmomatic.fastq.FastqParser.parse(FastqParser.java:127)
at org.usadellab.trimmomatic.TrimmomaticPE.process(TrimmomaticPE.java:251)
at org.usadellab.trimmomatic.TrimmomaticPE.run(TrimmomaticPE.java:498)
at org.usadellab.trimmomatic.TrimmomaticPE.main(TrimmomaticPE.java:506)

I've never used trimmomatic but it looks like you are passing in the incorrect parameters.
the trimmomatic webpage lists the usage from version 0.27+ as:
java -jar <path to trimmomatic.jar> PE [-threads <threads] [-phred33 | -phred64] [-trimlog <logFile>] <input 1> <input 2> <paired output 1> <unpaired output 1> <paired output 2> <unpaired output 2> <step 1> ...
or using the "old way"
java -classpath <path to trimmomatic jar> org.usadellab.trimmomatic.TrimmomaticPE [-threads <threads>] [-phred33 | -phred64] [-trimlog <logFile>] <input 1> <input 2> <paired output 1> <unpaired output 1> <paired output 2> <unpaired output 2> <step 1> ...
Where the only difference is the new way is specifying "PE" as the main class instead of a fully qualified path.
First, addressing your 2nd problem:
You look like you are doing both: specifying a fully qualified class name as well as the PE part. This makes trimmomatic think you have a fastq file named "PE" which doesn't exist.
If you get rid of the "PE" OR the qualfited class name; it will call the correct class. Which is what you do first in your first problem.
1st problem
I don't think you have the correct number of arguments listed in your first invocation so trimmomatic displays the usage to tell you what parameters are required. It would be nice if it told you what was wrong but it doesn't.
Solution
It looks like you are only providing 2 fastq files but trimmmoatic needs 6 file paths. You are missing the output paired and unpaired files paths for the read 1 and read 2 data which I assume get created by the program when it runs.
I guess your 2nd attempt got further along in the program since it saw enough parameters that you potentially had enough file paths specified (however, it turns out you had optional step parameters)

Following the advice of dkatzel below and user blakeoft on SeqAnswers (http://seqanswers.com/forums/showthread.php?t=45094), I dropped the PE flag and added individual file names for each output file and the program executed properly.
java -classpath /*filepath*/Trimmomatic-0.32/trimmomatic-0.32.jar org.usadellab.trimmomatic.TrimmomaticPE \
-phred33 \
~/refs/lec12/data_R1.fq ~/refs/lec12/data_R2.fq \
lane1_forward_paired.fq lane1_forward_unpaired.fq lane1_reverse_paired.fq lane1_reverse_unpaired.fq \
ILLUMINACLIP:/*filepath*/Trimmomatic-0.32/adapters/TruSeq3-PE-2.fa:2:30:10:3:true
NB: I also tried using the -baseout flag rather than a list of four files, and the program would open but not execute any commands
NB: The a log file could be generated using the flag -trimlog filename, but only if I first made a blank text file with the same name as the intended log file.

error while running CRF++ from a java project

I want to call CRF++ toolkit from a java program. I type the following:
Process process = runtime.exec("/home/toshiba/Bureau/CRF++-0.54/.libs/lt-crf_learn /home/toshiba/Bureau/CRF++-0.54/example/atb/template /home/toshiba/Bureau/CRF++-0.54/example/atb/tr_java.data");
process.waitFor();
But, I have the the following error:
CRF++: Yet Another CRF Tool Kit
Copyright (C) 2005-2009 Taku Kudo, All rights reserved.
Usage: /home/toshiba/Bureau/CRF++-0.54/.libs/lt-crf_learn [options] files
-f, --freq=INT use features that occuer no less than INT(default 1)
-m, --maxiter=INT set INT for max iterations in LBFGS routine(default 10k)
-c, --cost=FLOAT set FLOAT for cost parameter(default 1.0)
-e, --eta=FLOAT set FLOAT for termination criterion(default 0.0001)
-C, --convert convert text model to binary model
-t, --textmodel build also text model file for debugging
-a, --algorithm=(CRF|MIRA) select training algorithm
-p, --thread=INT number of threads(default 1)
-H, --shrinking-size=INT set INT for number of iterations variable needs to be optimal before considered for shrinking. (default 20)
-v, --version show the version and exit
-h, --help show this help and exit
I 'm wondering if any one could help me?

I don't think that's a bug in CRF++, since you are able to run it from command line. So the actual question is how to pass arguments properly when starting a process using Runtime.exec(). I would suggest trying the following:
String[] cmd = {"/home/toshiba/Bureau/CRF++-0.54/.libs/lt-crf_learn",
"/home/toshiba/Bureau/CRF++-0.54/example/atb/template",
"/home/toshiba/Bureau/CRF++-0.54/example/atb/tr_java.data"};
Process p = Runtime.getRuntime().exec(cmd);
This may help since Runtime.exec() sometimes splits the command line into arguments in a rather strange fashion.
Another potential problem is mentioned here: Java Runtime.exec()

There's a simple solution for this. Just write your command into a temporary file and execute that file as Runtime.getRuntime.exec("sh <temp-filename>"). Later you can delete this file. I will explain reason behind this if this solution works for you.

Calling Haskell from Java with C in between

This probably sounds like a nightmare, but I'd really like to get this working. I am using this example for the most part: Calling C from Haskell and am trying to get this working on ubuntu.
I am running this in java:
package test;
public class JniTest {
public native int fib(int x);
}
this in c after creating the .h file with javah: (test_JniTest.c)
#include "test_JniTest.h"
#include "Safe_stub.h"
JNIEXPORT jint JNICALL Java_test_JniTest_fib(JNIEnv * e, jobject o, jint f)
{
return fibonacci_hs(f);
}
and then for reference in haskell (before stub): (Safe.hs)
module Safe where
import Foreign.C.Types
fibonacci :: Int -> Int
fibonacci n = fibs !! n
where fibs = 0 : 1 : zipWith (+) fibs (tail fibs)
fibonacci_hs :: CInt -> CInt
fibonacci_hs = fromIntegral . fibonacci . fromIntegral
foreign export ccall fibonacci_hs :: CInt -> CInt
and this is what i'm trying to compile it with:
ghc -c -O Safe.hs
followed by:
ghc -shared -o libTest.jnilib -optc-O test_JniTest.c
-I/usr/lib/jvm/java-6-sun-1.6.0.26/include -I/usr/lib/jvm/java-6-sun-1.6.0.26/include/linux
and I am getting this error:
/usr/bin/ld: test_JniTest.o: relocation R_X86_64_PC32 against
undefined symbol `fibonacci_hs' can not be used when making a shared
object; recompile with -fPIC /usr/bin/ld: final link failed: Bad value
collect2: ld returned 1 exit status
I am not a c expert by any means and have no idea what to do about this. I tried compiling various ways with -fPIC, but I kept on getting the same error. Any idea what I might be doing wrong?
Thanks!

Although I've pretty much answered this question here: Communication between Java and Haskell, since this issue is more about the error itself, I will be adding the details for that here. The issue stems from Haskell not supporting shared libraries very well, while Java requires them.
Buildings plugins as Haskell shared libs gives us this insight and workaround:
In principle you can use -shared without -dynamic in the link step. That would mean to statically link the rts all the base libraries into your new shared library. This would make a very big, but standalone shared library. However that would require all the static libraries to have been built with -fPIC so that the code is suitable to include into a shared library and we don't do that at the moment.
If we use ldd again to look at the libfoo.so that we've made we will notice that it is missing a dependency on the rts library. This is problem that we've yet to sort out, so for the moment we can just add the dependency ourselves:
$ ghc --make -dynamic -shared -fPIC Foo.hs -o libfoo.so \
-lHSrts-ghc6.11 -optl-Wl,-rpath,/opt/ghc/lib/ghc-6.11/
This is a workaround because it requires us to know the version of the rts library at build time.

If your goal is to actually get something done (as opposed to just playing around with JNI) I suggest tackling this as a garden variety RPC problem and utilizing one of the many framework/protocols for it:
Protocol Buffers from Google
Thrift from Facebook
Avro (well this is mostly a wire protocol)
From what you are trying to do, Thrift might be your best bet since it describes a full client/server RPC stack but I'm pretty sure any of them would pretty much work over a simple socket.