We Keep Coding

How to use ProcessBuilder with bash -c, if the command has double quotes

I have to join two huge files based on multiple columns. Pipe(|) symbol is the delimiter in both files.
Right now, I am generating unix join command, writing it to a shell script and then executing the shell script using ProcessBuilder to get the desired output.
The command looks something like this,
join -a 1 -a 2 -t \| -1 1 -2 1 -o 1.3,1.4,1.5,1.6,1.7,2.3,2.4,2.5,2.6,2.7 <(<"/home/input file 1" awk -F'|' 'NR>1{print $3"~&~"$4"|"$0}' | sort -k1 -t\| ) <(<"/home/input file 2" awk -F'|' 'NR>1{print $3~&~$4"|"$0}' | sort -k1 -t\| ) > "/home/output file"
This is working as expected.
I am trying to omit the step of writing the command to a shell script by using bash -c. But I am running into issues mainly because of the double quotes(") and dollar($) in the awk command. I tried to escape them using backward slash, but was of no use.
the java code I am using currently is
long pid = -1;
try {
StringBuilder completeCommand = new StringBuilder();
for(String s: commands){
completeCommand.append(s);
completeCommand.append(" ");
}
completeCommand.append(" > \"");
completeCommand.append(outputDir + File.separator + outputFileName);
completeCommand.append("\"");
File fileOutputDir = new File(outputDir);
fileOutputDir.mkdirs();
Files.writeString(Path.of(outputDir + File.separator + scriptName),
completeCommand.toString(),
new OpenOption[]{StandardOpenOption.TRUNCATE_EXISTING, StandardOpenOption.WRITE,
StandardOpenOption.CREATE});
ProcessBuilder processBuilder = new ProcessBuilder("bash", "\"" + outputDir + File.separator + scriptName + "\"");
Process p = processBuilder.start();
pid = p.pid();
p.waitFor();
return pid;
} catch (IOException | InterruptedException e) {
log.error("Error in running join command" , e);
return pid;
}
When I tried to use bash -c, I just changed the ProcessBuilder statement like below,
ProcessBuilder processBuilder = new ProcessBuilder("bash", "-c", completeCommand.toString());
This doesn't throw any error, but it generates an empty file.
Is there any way I can solve this issue ?
Any help or suggestions would be great.
Thanks

Java - ProcessBuilder hanging while executing a shell script that returns nothing

I know there are plenty of questions answered about processbuilder hanging while executing a command. I went through most of the answers, but nothing works for me so I am either missing something obvious or something else is wrong in my code. So here goes.
I have a java method that does this
final File tmp = File.createTempFile("out", null);
System.out.println("Tmp File path: "+tmp.getAbsolutePath());
tmp.deleteOnExit();
System.out.println("Executing script: "+ app + " " + report + " " + oldFileName + " " + newFileName);
ProcessBuilder procBuilder = new ProcessBuilder(app, report, oldFileName, newFileName).redirectErrorStream(true).redirectOutput(tmp);
procBuilder.redirectError(tmp);
Process proc = procBuilder.start();
// throw exception if the process fails
try {
proc.waitFor();
} catch (InterruptedException e) {
e.printStackTrace();
}
int changes = proc.exitValue();
if (changes != 0) {
throw new FileCompareException("File Comparison found changes: " + newFileName);
}
The command resolves to call a shell script like this
src/main/resources/scripts/./vimdiff.sh src/main/resources/auth/report/report.html src/main/resources/tests/goldenFile/golden_online src/main/resources/genFile/file_20170715234458224.log
The shell script itself is calling vimdiff to diff between two files and create an output.
VAR="vimdiff -c TOhtml -c 'w $1 | qall!' $2 $3 &> /dev/null"
eval $VAR
Now when I run the program from eclipse, it just hangs at proc.waitfor method. But when I copy the out of the generated command and run it from the command line it runs without any issues.
In the shell script, I am directing any output to /dev/null, so no output stream issue should arise. However, I am directing the output to a tmp file using redirectErrorStream(true).redirectOutput(tmp) in my code anyway. So I am at a loss. Why is my program hanging?

Issues while Invoking Python process using java process builder

I am using java process builder to start python process with one flag and with one argument as shown below. But i don't see any exception nor process starts up.
Command i want to run is
python oc_db5.py -c input.json
location of file oc_db5.py is
/opt/jvision/grpc/gui
My code is shown below
processBuilder = new ProcessBuilder(
Arrays.asList(
"python",
"oc_db5.py",
"-c",
"input.json"));
processBuilder.directory(new File("/opt/jvision/grpc/gui"));
processBuilder.start();
logger.info("Process started ..." + new Date());
int count = 0;
BufferedReader br = new BufferedReader(new InputStreamReader(
process.getInputStream()));
while ((lineData = br.readLine()) != null) {
System.out.println("line: " + ++count + " " + lineData);
logger.info("line: " + ++count + " " + lineData);
}
process.waitFor();
process.getErrorStream();
process.waitFor();
process.exitValue();
I can see that log file contains entry "process start..." but i don't really see that process is started. Wondering what i am missing.

Can you check if python is in your PATH? I have similar problem with custom command long long time ago. You can use absolute path to try it :).
You can also check you enviromental variables via
Map env = System.getenv();
If you are using Linux you can start process like "sleep 1000" then check it is present in system process table via "ps aux | grep sleep" or something like it :)

Piping smbpasswd via Java

I tried to piping password to smbpassword via Java application, here how I pipe via terminal:
(echo newPassword; echo confirmPassword) | smbpasswd -a -s client1
and the output show the command is nicely done:
Added user client1
However, I cannot accomplish this via Java application, here the codes that I use:
public void run(String command, String[] prompt) {
try {
String[] args = new String[] {"/bin/bash","-c","echo " + rootPassword + "| sudo -S " + command};
Process proc = new ProcessBuilder(args).start();
if (prompt != null && prompt.length > 0) {
for (int i = 0; i < prompt.length; i++) {
proc.getOutputStream().write((prompt[i] + "\r\n").getBytes());
proc.getOutputStream().flush();
}
}
proc.waitFor();
String output = "";
String line;
BufferedReader input = new BufferedReader(new InputStreamReader(proc.getInputStream()));
while ((line = input.readLine()) != null) {
System.out.println(line);
output += line + "\n";
}
System.out.println(output);
input.close();
}
catch (Exception ex) {
System.out.println(null, ex.getMessage());
}
}
and I tried to use OutputStream it like:
String[] pipe = { password, password };
run("smbpasswd -s -a " + username, pipe);
or pipe it like:
run("(echo " + password + "; echo " + password + ") | smbpasswd -s -a " + username, null);
but both doesn't work and I got no output and the user seems not created after I check via pdbedit -L. However, I able to execute another command with pipe such as echo username:password | chpasswd via that function.
Any idea?
Thanks in advance.

The main conceptual problem is that shell code such as you are trying to run:
(echo newPassword; echo confirmPassword) | smbpasswd -a -s client1
is not a "command" in the sense that sudo requires. In fact, it's not a "command" even in shell terminology; rather, it's a pipeline, consisting of a compound command and a simple command. sudo expects you to provide a simple command.
But that's actually a bit of an aside. Your problem isn't really specific to sudo; rather, it's a shell problem. In the failure case, your run() method invokes bash to execute the following shell code:
echo rootPassword | sudo -S (echo newPassword; echo confirmPassword) | smbpasswd -a -s client1
That's problematic for at least two reasons:
A subshell invocation cannot appear in a simple command's (i.e. sudo's) argument list like that. You could put a command substitution there ($(echo ...)), but that will not serve your purpose because the internal line break will not then be preserved.
Even if you solved (1), your second pipe is at the wrong level: it would pipe the output of sudo into smbpasswd. That's fine data-wise, but it leaves smbpasswd running without privilege -- the sudo does not apply to it.
You could probably solve the problem by instructing sudo to run the code via a shell. You could do that by embedding the ultimate command inside an inner bash -c command, but it would probably be easier to use sudo's -s option (but beware that -s can choose a different shell than bash if you let it do). Something along those lines is probably the most general solution if you want to encode the whole operation in a single String in your Java source:
String[] args = new String[] { "/bin/bash", "-c",
"echo " + rootPassword + " | sudo -S -s '" + command + "'" };
String[] args = new String[] { "/bin/bash", "-c",
"echo " + rootPassword + " | sudo -S bash -c '" + command + "'" };
Much of your problem, however, arises from your reliance on echo to feed data to your commands. You should consider instead using your Process's OutputStream to feed data to it from your Java program. That does not lend itself to quite as simple a Java-side API, but I think you're fooling yourself a bit about how simple your API really is. It has hidden gotchas, as you discovered, and even the revision I offered above still has at least one: it will likely break if the command submitted to it contains a single-quote character (').
Update:
Here's one way to feed the process the needed data via its OutputStream:
public void run2(String command, String commandInput)
throws IOException, InterruptedException {
String[] args = new String[] { "sudo", "-S", "bash", "-c", command };
Process proc = new ProcessBuilder(args).start();
Writer toProc = new OutputStreamWriter(proc.getOutputStream());
toProc.write(rootPassword, 0, rootPassword.length());
toProc.write('\n');
toProc.write(commandInput, 0, commandInput.length());
toProc.close();
proc.waitFor();
BufferedReader input = new BufferedReader(new InputStreamReader(proc.getInputStream()));
StringBuilder output = new StringBuilder();
String line;
while ((line = input.readLine()) != null) {
output.append(line).append('\n');
}
System.out.println("Command output:");
System.out.println(output.toString());
input.close();
}
You might use that from within its class as
run2("smbpasswd -a -s client1", "newPassword\nnewPassword");
Note in particular that the Process's output stream must, in general, be closed after you've written everything you intend to write to it. Some commands you could run will not exit if you do not do so.
Note also
For the general case, you must read every Process's input stream and error stream (or just the former if you combine them), in parallel with the process itself. If you have not combined the streams then you must read them in parallel with each other. If you are writing to the process's output stream then that, too, must be in parallel. If you do not arrange to handle each stream via its own thread then it is possible for the process to get stuck.
On the other hand, you don't need a separate thread to run the Process itself in -- after all, it represents an entire separate process. Just defer waiting for it until you have closed its output stream and read the end of its input and error streams.

Runtime.exec command not working

I have a java application that downloads a file from a web service using wget. When executing the command through java it returns with: "wget: not an http or ftp url:"
When i execute the command directly it runs without problems. Here is my code:
try {
Debug.println("Starting copy of "+srcFile+" to "+destFile);
String command = "wget -O " + destFile + " \""+ srcFile +"\"";
Process p = Runtime.getRuntime().exec(command);
int exitCode = p.waitFor();
if(Debug.isDebugMode())
{
Debug.println(command);
BufferedReader stdInput = new BufferedReader(new InputStreamReader(p.getErrorStream()));
String s;
while((s = stdInput.readLine()) != null)
{
Debug.println(s);
}
}
Debug.println("Finished with code: " + String.valueOf(exitCode));
}
catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return false;
And this is the output:
24/04/2013 10:11:05 Starting copy of stoppenmetroken.webcolors.local/service/track?track=3b1ac68a288345c183a08c714901a398&mac=089000A09090 to /opt/byato/data/song/3b1ac68a288345c183a08c714901a398
24/04/2013 10:11:05 wget -O /opt/byato/data/song/3b1ac68a288345c183a08c714901a398 "stoppenmetroken.webcolors.local/service/track?track=3b1ac68a288345c183a08c714901a398&mac=089000A09090"
24/04/2013 10:11:05 wget: not an http or ftp url: "http://stoppenmetroken.webcolors.local/service/track?track=3b1ac68a288345c183a08c714901a398&mac=089000A09090"
24/04/2013 10:11:05 Finished with code: 1
ps: i removed the http:// part of the output because i dont have enough reputation points -.-
What am i missing?

Can you try to execute the command like this :
Process p = Runtime.getRuntime().exec("/bin/bash -c "+command); //for linux
or
Process p = Runtime.getRuntime().exec("cmd.exe /c "+command); //for Windows
Sometimes we need to explicitly invoke Linux shell or command prompt.
Hope this will work.

I suspect this:
String command = "wget -O " + destFile + " \""+ srcFile +"\"";
is the problem. When you run in a shell, the quotes around the URL will be removed. However when you run via Java you're not running via a shell and your URL starts with "http... (look closely at the error message).
If you don't want Runtime.exec() to parse and split your arguments then you might consider the variant that takes individual arguments. A more efficient solution still would be to download using HttpComponents.