We Keep Coding

How to use ProcessBuilder with bash -c, if the command has double quotes

I have to join two huge files based on multiple columns. Pipe(|) symbol is the delimiter in both files.
Right now, I am generating unix join command, writing it to a shell script and then executing the shell script using ProcessBuilder to get the desired output.
The command looks something like this,
join -a 1 -a 2 -t \| -1 1 -2 1 -o 1.3,1.4,1.5,1.6,1.7,2.3,2.4,2.5,2.6,2.7 <(<"/home/input file 1" awk -F'|' 'NR>1{print $3"~&~"$4"|"$0}' | sort -k1 -t\| ) <(<"/home/input file 2" awk -F'|' 'NR>1{print $3~&~$4"|"$0}' | sort -k1 -t\| ) > "/home/output file"
This is working as expected.
I am trying to omit the step of writing the command to a shell script by using bash -c. But I am running into issues mainly because of the double quotes(") and dollar($) in the awk command. I tried to escape them using backward slash, but was of no use.
the java code I am using currently is
long pid = -1;
try {
StringBuilder completeCommand = new StringBuilder();
for(String s: commands){
completeCommand.append(s);
completeCommand.append(" ");
}
completeCommand.append(" > \"");
completeCommand.append(outputDir + File.separator + outputFileName);
completeCommand.append("\"");
File fileOutputDir = new File(outputDir);
fileOutputDir.mkdirs();
Files.writeString(Path.of(outputDir + File.separator + scriptName),
completeCommand.toString(),
new OpenOption[]{StandardOpenOption.TRUNCATE_EXISTING, StandardOpenOption.WRITE,
StandardOpenOption.CREATE});
ProcessBuilder processBuilder = new ProcessBuilder("bash", "\"" + outputDir + File.separator + scriptName + "\"");
Process p = processBuilder.start();
pid = p.pid();
p.waitFor();
return pid;
} catch (IOException | InterruptedException e) {
log.error("Error in running join command" , e);
return pid;
}
When I tried to use bash -c, I just changed the ProcessBuilder statement like below,
ProcessBuilder processBuilder = new ProcessBuilder("bash", "-c", completeCommand.toString());
This doesn't throw any error, but it generates an empty file.
Is there any way I can solve this issue ?
Any help or suggestions would be great.
Thanks

Why do my Bash scripts stop?

Bellow is a Bash script that I create and execute through java. It utilises several different scripts to produce statistics for RNA-Seq data. My problem is that when the script is executed it reaches the second stage of the script before the processes stops (the program does not error out, the required programs just stop processing). I have tried running the created script separately from the command line and it completes with no errors. An suggestions would be appreciated.
String Trans_ref =
"#!/bin/bash \n" +
"mkdir -p "+Output+"/"+Sample+"_RSEM \n" +
"cd "+Output+"/"+Sample+"_RSEM \n" +
"PATH=$PATH:"+RSEMprep+" \n" +
"export PATH=$PATH \n" +
""+RSEMprep+"/rsem-prepare-reference --no-polyA --bowtie "+Output+"/Trans_CDHIT.fast Trans_CDHIT.RSEM \n" +
""+RSEMprep+"/rsem-calculate-expression --paired-end -p "+CPU+" "+Output+"/SRR617145_1.fastq "+Output+"/SRR617145_2.fastq Trans_CDHIT.RSEM Trans_CDHIT.genes.results \n"+
""+Trinprep+"/util/misc/count_features_given_MIN_FPKM_threshold.pl "+Output+"/"+Sample+"_RSEM/RSEM.genes.results > "+Output+"/"+Sample+"_RSEM/cumul_counts.txt \n"+
""+Trinprep+"/util/filter_fasta_by_rsem_values.pl --rsem_output= "+Output+"/"+Sample+"_RSEM/RSEM.isoforms.results --fasta="+Output+"/Trans_CDHIT.fasta -t 100 --output="+Output+"/"+Sample+"_RSEM/Trans_RSEMfilters.fasta \n" +
""+Trinprep+"/util/bowtie_PE_separate_then_join.pl --seqType fq --left "+Output+"/"+Sample+"_1.fasta --right "+Output+"/"+Sample+"_2.fasta --target "+Output+"/Trans_CDHIT.fasta --aligner bowtie --SS_lib_type FR -- -p 4 --all --best --strata -m 300 \n";
System.out.println(Trans_ref);
FileUtils.writeStringToFile(new File(Output+"/TranRSEM"), Trans_ref);
StringBuffer Trim = new StringBuffer();
String cmd = (Output+"/TranRSEM");
Process p;
try{
p = Runtime.getRuntime().exec(new String[]{"/bin/sh","-c", cmd});
p.waitFor();
BufferedReader reader1 =
new BufferedReader(new InputStreamReader(p.getInputStream()));
System.out.println("Merg Finished");
} catch (Exception e) {
e.printStackTrace();
}

To All
Thankyou very much for your comments
I managed to solve the problem. I discovered it was a java memory limitation. I solved this by using the Xmx20000m command in the project properties in my IDE (netbeans).
this appears to have solved my problem.

How do I Pipe process output to a file on Windows and JDK 6u45

I have the following windows batch file (run.bat):
#echo off
echo hello batch file to sysout
And the following java code, which runs the batch files and redirects output to a file:
public static void main(String[] args) throws IOException {
System.out.println("Current java version is: " + System.getProperty("java.version"));
ProcessBuilder pb =
new ProcessBuilder("cmd.exe", "/c",
"run.bat"
,">>", "stdout.txt","2>>", "stderr.txt"
);
System.out.println("Command is: " + pb.command());
Process proc = pb.start();
InputStream in = proc.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
String line = null;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
int exitValue = proc.exitValue();
System.out.println("Exit value: " + exitValue);
}
On JDKs up to and including JDK6u43 I get the following output:
Current java version is: 1.6.0_29
Command is: [cmd.exe, /c, run.bat, >>, stdout.txt, 2>>, stderr.txt]
Exit value: 0
and the script output is written to the file.
As of JDK 6u45 and 7, I get the following output:
Current java version is: 1.6.0_45
Command is: [cmd.exe, /c, run.bat, >>, stdout.txt, 2>>, stderr.txt]
hello batch file to sysout
Exit value: 0
And nothing is written to the output file.
This may or may not be related to the changes made in Runtime.exec() , described at: http://www.oracle.com/technetwork/java/javase/6u45-relnotes-1932876.html
What is the correct way of starting a process on Windows with output redirected to files?
Note: In a real world scenario, the command to execute may include parameters with spaces, as in:
ProcessBuilder pb = new ProcessBuilder("cmd.exe", "/c",
"run.bat", "Some Input With Spaces",
">>", "stdout.txt","2>>", "stderr.txt");

This is the simplest method i found on http://tamanmohamed.blogspot.in/2012/06/jdk7-processbuilder-and-how-redirecting.html
File output = new File("C:/PBExample/ProcessLog.txt");
ProcessBuilder pb = new ProcessBuilder("cmd");
pb.redirectOutput(output);

Several suggestions here:
Does the input with the spaces need to be treated as single String (with spaces),or id it in actual several inputs? If the first Option is the case I would suggest to quote it for the windows runtime:
ProcessBuilder pb = new ProcessBuilder("cmd.exe", "/c",
"run.bat", "\"Some Input With Spaces\"",
">>", "stdout.txt","2>>", "stderr.txt");
Instead of redirecting the input to stdout.txt and stderr.txt using the shell, why not do it using Java using getOutputStream() and getErrorStream()? Here is an example using Guava's IO package. Of course you may want to have those in separate threads, you need proper exception handling, etc.
InputStream stdout = new BufferedInputStream(proc.getInputStream());
FileOutputStream stdoutFile = new FileOutputStream("stdout.txt");
ByteStreams.copy(stdout, stdoutFile);
InputStream stderr = new BufferedInputStream(proc.getErrorStream());
FileOutputStream stderrFile = new FileOutputStream("stderr.txt");
ByteStreams.copy(stderr, stderrFile);
stdout.close();
stderr.close();
stdoutFile.close();
stderrFile.close();
Another option, why not create a run.bat wrapper that will make the redirections?
#echo off
cmd.exe /c run.bat "%1" >> "%2" 2>> "%3"

Use getOutputStream() on the process, instead of using System.out.println(). Sometimes the semantics change between Java implementations.
This seems to be a bugfix actually - the newer implementation makes sense.

Runtime.exec command not working

I have a java application that downloads a file from a web service using wget. When executing the command through java it returns with: "wget: not an http or ftp url:"
When i execute the command directly it runs without problems. Here is my code:
try {
Debug.println("Starting copy of "+srcFile+" to "+destFile);
String command = "wget -O " + destFile + " \""+ srcFile +"\"";
Process p = Runtime.getRuntime().exec(command);
int exitCode = p.waitFor();
if(Debug.isDebugMode())
{
Debug.println(command);
BufferedReader stdInput = new BufferedReader(new InputStreamReader(p.getErrorStream()));
String s;
while((s = stdInput.readLine()) != null)
{
Debug.println(s);
}
}
Debug.println("Finished with code: " + String.valueOf(exitCode));
}
catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return false;
And this is the output:
24/04/2013 10:11:05 Starting copy of stoppenmetroken.webcolors.local/service/track?track=3b1ac68a288345c183a08c714901a398&mac=089000A09090 to /opt/byato/data/song/3b1ac68a288345c183a08c714901a398
24/04/2013 10:11:05 wget -O /opt/byato/data/song/3b1ac68a288345c183a08c714901a398 "stoppenmetroken.webcolors.local/service/track?track=3b1ac68a288345c183a08c714901a398&mac=089000A09090"
24/04/2013 10:11:05 wget: not an http or ftp url: "http://stoppenmetroken.webcolors.local/service/track?track=3b1ac68a288345c183a08c714901a398&mac=089000A09090"
24/04/2013 10:11:05 Finished with code: 1
ps: i removed the http:// part of the output because i dont have enough reputation points -.-
What am i missing?

Can you try to execute the command like this :
Process p = Runtime.getRuntime().exec("/bin/bash -c "+command); //for linux
or
Process p = Runtime.getRuntime().exec("cmd.exe /c "+command); //for Windows
Sometimes we need to explicitly invoke Linux shell or command prompt.
Hope this will work.

I suspect this:
String command = "wget -O " + destFile + " \""+ srcFile +"\"";
is the problem. When you run in a shell, the quotes around the URL will be removed. However when you run via Java you're not running via a shell and your URL starts with "http... (look closely at the error message).
If you don't want Runtime.exec() to parse and split your arguments then you might consider the variant that takes individual arguments. A more efficient solution still would be to download using HttpComponents.

Java can't get full error when running an external command

I'm trying to compile typescript files in java.
Here is a ".ts" file which has errors:
alert("hello, typescript");
errrrrrrrrrrrrrrrrrrrrrrrrrror
When I compile in windows shell(cmd):
tsc hello.ts
It will report error with message:
E:/WORKSPACE/test/typescripts/hello.ts(2,0): The name 'errrrrrrrrrrrrrrrrrrrrrrrrror'
does not exist in the current scope
But when I do it in java:
String cmd = "cmd /C tsc hello.ts";
Process p = Runtime.getRuntime().exec(cmd);
String out = IOUtils.toString(p.getInputStream());
String error = IOUtils.toString(p.getErrorStream());
System.out.println("### out: " + out);
System.out.println("### err: " + error);
It prints:
### out:
### err: E:/WORKSPACE/test/typescripts/hello.ts(2,0):
You can see the detail errors is not captured. Where is wrong with my code?
update
I just made sure that the tsc.exe provided by MS has no such problem, and the one I run in this question is the tsc.cmd installed from npm npm install typescript

Have you tried using a raw Process/ProcessBuilder combination?
ProcessBuilder pb = new ProcessBuilder("cmd /C tsc hello.ts");
//merge error output with the standard output
pb.redirectErrorStream(true);
Process p = pb.start();
try (BufferedReader reader = new BufferedReader(new InputStreamReader(p.getInputStream(), Charset.forName("UTF-8")))) {
String line;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
}

I just spent a couple hours chasing the same problem.
In the end I worked around it by adding "2> errorfile.txt" to my command line. This redirects the stderr to a file and then I read and print that file.