We Keep Coding

Doubts about ProcessBuilder.start() [duplicate]

I'm trying to use Java's ProcessBuilder class to execute a command that has a pipe in it. For example:
ls -l | grep foo
However, I get an error:
ls: |: no such file or directory
Followed by:
ls: grep: no such file or directory
Even though that command works perfectly from the command line, I can not get ProcessBuilder to execute a command that redirects its output to another.
Is there any way to accomplish this?

This should work:
ProcessBuilder b = new ProcessBuilder("/bin/sh", "-c", "ls -l| grep foo");
To execute a pipeline, you have to invoke a shell, and then run your commands inside that shell.

The simplest way is to invoke the shell with the command line as the parameter. After all, it's the shell which is interpreting "|" to mean "pipe the data between two processes".
Alternatively, you could launch each process separately, and read from the standard output of "ls -l", writing the data to the standard input of "grep" in your example.

Since Java 9, there’s genuine support for piplines in ProcessBuilder.
So you can use
List<String> result;
List<Process> processes = ProcessBuilder.startPipeline(List.of(
new ProcessBuilder("ls", "-l")
.inheritIO().redirectOutput(ProcessBuilder.Redirect.PIPE),
new ProcessBuilder("grep", "foo")
.redirectError(ProcessBuilder.Redirect.INHERIT)
));
try(Scanner s = new Scanner(processes.get(processes.size() - 1).getInputStream())) {
result = s.useDelimiter("\\R").tokens().toList();
}
to get the matching lines in a list.
Or, for Windows
List<String> result;
List<Process> processes = ProcessBuilder.startPipeline(List.of(
new ProcessBuilder("cmd", "/c", "dir")
.inheritIO().redirectOutput(ProcessBuilder.Redirect.PIPE),
new ProcessBuilder("find", "\"foo\"")
.redirectError(ProcessBuilder.Redirect.INHERIT)
));
try(Scanner s = new Scanner(processes.get(processes.size() - 1).getInputStream())) {
result = s.useDelimiter("\\R").tokens().toList();
}
These examples redirect stdin of the first process and all error streams to inherit, to use the same as the Java process.
You can also call .redirectOutput(ProcessBuilder.Redirect.INHERIT) on the ProcessBuilder of the last process, to print the results directly to the console (or wherever stdout has been redirected to).

Why does the Java CRLF token does not work with batch file inputs?

background:
I once answered this question
that was about flushing two input strings from a Java process to a batch script. Since I found a workaround solution I am still
very interested to solve the remaining mystery and find out why the obvious solution is not working.
problem description
See this very simple batch script:
#ECHO OFF
SET /P input1=1st Input:
SET /P input2=2nd Input:
ECHO 1st Input: %input1% and 2nd Input: %input2%
If you run this batch script with Java using ProcessBuilder and flush two input strings into it you will notice that only the
first input string will be consumed while the second will be ignored.
I found out that SET /P command consumes input from pipes when
CRLF token is found
by timeout
by full buffer(1024 Bytes)
My accepted workaround was based on the last two options by using a Thread.sleep(100) statement between the inputs or using a 1024 Byte
Buffer for each input.
It always works for single input or in this case the first input because closing the stream has the effect
that the batch script reads one input and empty returns all following SET /P statements.
the question
Why is the first option by using the CRLF token "input\r\n" not working?
research
I already tried to workaround the String.getBytes() method by creating a Byte Buffer myself using \x0d and \x0a as last
bytes for CRLF token but it has no effect.
And I tried all other OutputStream wrappers like PrintWriter to check if there is
a problem with the flush() implementation without any success.
I created a C++ program that basically does the same as the java programm by using CreateProcess and stangely it works like a charm.
testing code
Not working Java code:
ProcessBuilder builder = new ProcessBuilder("test.bat");
Process process = builder.start();
OutputStream out = process.getOutputStream();
out.write("foo\r\n".getBytes());
out.flush();
out.write("bar\r\n".getBytes());
out.flush();
out.close();
BufferedReader in = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
while ((line = in.readLine()) != null)
System.out.println(line);
in.close();
Full working C++ code:
DWORD dwWritten;
char cmdline[] = "test.bat";
CHAR Input1[] = "foo\r\n";
CHAR Input2[] = "bar\r\n";
HANDLE hStdInRd = NULL;
HANDLE hStdInWr = NULL;
SECURITY_ATTRIBUTES saAttr;
PROCESS_INFORMATION piProcInfo;
STARTUPINFO siStartInfo;
// Create Pipe
saAttr.nLength = sizeof(SECURITY_ATTRIBUTES);
saAttr.bInheritHandle = TRUE;
saAttr.lpSecurityDescriptor = NULL;
CreatePipe(&hStdInRd, &hStdInWr, &saAttr, 0);
SetHandleInformation(hStdInWr, HANDLE_FLAG_INHERIT, 0);
// Create Process
ZeroMemory( &piProcInfo, sizeof(PROCESS_INFORMATION));
ZeroMemory( &siStartInfo, sizeof(STARTUPINFO));
siStartInfo.cb = sizeof(STARTUPINFO);
siStartInfo.hStdError = GetStdHandle(STD_ERROR_HANDLE);
siStartInfo.hStdOutput = GetStdHandle(STD_OUTPUT_HANDLE);
siStartInfo.hStdInput = hStdInRd;
siStartInfo.dwFlags |= STARTF_USESTDHANDLES;
CreateProcess(NULL, cmdline, NULL, NULL, TRUE, 0, NULL, NULL, &siStartInfo, &piProcInfo);
CloseHandle(piProcInfo.hProcess);
CloseHandle(piProcInfo.hThread);
// Write to Pipe
WriteFile(hStdInWr, Input1, (DWORD)strlen(Input1), &dwWritten, NULL);
FlushFileBuffers(hStdInWr);
WriteFile(hStdInWr, Input2, (DWORD)strlen(Input2), &dwWritten, NULL);
FlushFileBuffers(hStdInWr);
CloseHandle(hStdInWr);
the question again
The problem does not make any sense to me and is bugging me a lot. Why does sending the CRLF token from Java does not have any
effect on batch file inputs while it does when sending from C++ program?

About "put /p" and the pipe and child processes on Windows O.S.
Just for a test I've a little expanded your test batch to get four input instead of two
Now have a look at this nice test
>type test.txt | test.bat
1st Input:2nd Input:3rd Input:4th Input:1st Input: one and 2nd Input: and 3rd
Input: and 4rd Input:
"--"
>test.bat < test.txt
1st Input:2nd Input:3rd Input:4th Input:1st Input: one and 2nd Input: two and
3rd Input: three and 4rd Input: four
"--"
The interesting thing in here is that the first example works exactly as the java code (only the first "set /P" receive a value, while the second one works as expected
More interesting in if you put a line somewhere in the batch file like this one: wmic Process >> TestProcesses.txt
by inspecting the TestProcesses.txt, in my enviromnet, I can see that whith the first method (pipe) is present cmd.exe C:\Windows\system32\cmd.exe /S /D /c" test.bat" that isn't present when we use the second one (redirection)
I we run out new test batch (including wmic diagnostics) from java; when we inspect the TestProcesses we should see two different processes:
java.exe java -cp .\build\classes javaappcrlf.JavaAppCRLF
cmd.exe C:\Windows\system32\cmd.exe /c C:\projects\JavaAppCRLF\test.bat
as in the first method (pipe) we have a separate process for the batch where "put /p" doesnt works
From the chapter Pipes and CMD.exe of the article Pipes and CMD.exe
This has several side effects: Any newline (CR/LF) characters in the
batch_command will be turned into & operators. see StackOverflow If
the batch_command includes any caret escape characters ^ they will
need to be doubled up so that the escape survives into the new CMD
shell.
also the linked article on stack overflow is interesting
About C++ Testing
I make a little change to the c++ program described in
Creating a Child Process with Redirected Input and Output just to read a file of four lines ad passing its content to a child process that execute our batch through a pipe and the results are the same of your Java program
Alternative refactoring/workaround
from the findings mentioned above, it comes that a java program that read and write to (temporary) files ( ...I know is not the same thing ) should works; I successfully tested a working solution by changing the builder this way
ProcessBuilder builder = new ProcessBuilder(
"cmd",
"/c",
"(C:\\projects\\JavaAppCRLF\\test4.bat < C:\\projects\\JavaAppCRLF\\tmp-test4.in)",
">",
"C:\\projects\\JavaAppCRLF\\tmp-test4.out"
);
Post Scriptum: an interesting note about other shell (i.e: bash on ”os x" or linux )
AFAIK not all the other platforms suffers this "issue" in the same way; i.e.
on a bash (os x terminal) I made the following test with a script that acts just as our previous testing under Windows:
cd ~/projects/so-test/java-crlf-token/JavaAppCRLF
$ cat test.sh
#!/bin/bash -
# SET /P input1=1st Input:
echo -n "1st Input:";
read input1;
#SET /P input2=2nd Input:
echo -n "2nd Input:";
read input2;
#ECHO 1st Input: %input1% and 2nd Input: %input2%
echo -n "1st Input: ${input1} and 2nd Input: ${input2}"
then the only one changed to the java program is to reference the script:
ProcessBuilder builder = new ProcessBuilder("/Users/userx/projects/so-test/java-crlf-token/JavaAppCRLF/test.sh");
let's see what happens:
$ cat test.txt
abc
cde
# :pipe
$ cat test.txt | test.sh
$ cat test.txt | ./test.sh
1st Input:2nd Input:1st Input: abc and 2nd Input: cde
# :redirection
$ ./test.sh < test.txt
1st Input:2nd Input:1st Input: abc and 2nd Input: cde
# :java
$ java -cp build/classes/ javaappcrlf.JavaAppCRLF
1st Input:2nd Input:1st Input: foo
and 2nd Input: bar

Executing awk command in java

i am trying to execute awk command in java for linux/unix os but the thing is when i execute the command it does not show any error it.But after execution there is no output and it takes fraction of second to execute i dont know the problem please help .
the code is
process p =new process():
yes = "awk '{print $1}' /root/Desktop/net/net.zone >> /root/Desktop/net/net.txt";
p = Runtime.getRuntime().exec(yes);
Thank you for your help

Starting command line processes correctly with Java isn't easy. I suggest you use commons-exec instead of trying it yourself.
Now you have two things in the command line which need special handing:
Single quotes around the AWK script. When you pass each argument as a individual strings to CommandLine via addArgument, you don't need the quotes anymore.
The output redirection.
Since you create a child process, you are in control of stdin and stout. That means you need to open the target file for append in Java, wrap it in a PumpStreamHandler and pass that to DefaultExecutor. See this question for details: Process output from apache-commons exec

Running Unix Command in Java

I am running the following code, and it stops at waitfor() function. What could be the reason and how can I solve it?
String line;
Process albumProcess;
try {
albumProcess = Runtime.getRuntime().exec(
"iconv -f UTF-16 -t UTF-8 /home/gozenem/"+ xmlFileName +
".xml | grep albumID");
albumProcess.waitFor();
BufferedReader in = new BufferedReader(
new InputStreamReader(albumProcess.getInputStream()));
ArrayList<String> lineList = new ArrayList<String>();
while ((line = in.readLine()) != null) {
lineList.add(line);
}
result[0] = lineList.size();
albumProcess.destroy();
} catch (Exception e) {}

The | grep ... is not consuming the output from the command as you expect because getRuntime().exec does not understand piping symbols. The process gets bogged down waiting for something to consume its output and its getting passed bogus command line arguments "|", "grep", and "albumId".
A shell will understand | but execv will not, so you need to use bash -c instead to get a shell to do the piping (see java shell for executing/coordinating processes? do the piping yourself (see Pipe between java processes on command shell not reliable working). Java 7 has a new ProcessBuilder class that makes it easy to set up pipes so you can use those if you're only running on a bleeding edge JVM.
Once you've got grep running, if there's a bunch of lines that match, it may still fill up the buffer, so you need something sitting on the buffer consuming the process's output stream. Moving
albumProcess.waitFor();
after the while loop should do it.

I think you should try to read the output from the process before waiting on it. Otherwise, if the command outputs to much then the buffer may get filled.
Have a look at this article which explains how to read from the process: http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html?page=4

Java exec() does not return expected result of pipes' connected commands

I'm calling command line programs connected by pipes. All this works on Linux for sure.
My method:
protected String execCommand(String command) throws IOException {
String line = null;
if (command.length() > 0) {
Process child = Runtime.getRuntime().exec(command);
InputStream lsOut = child.getInputStream();
InputStreamReader r = new InputStreamReader(lsOut);
BufferedReader in = new BufferedReader(r);
String readline = null;
while ((readline = in.readLine()) != null) {
line = line + readline;
}
}
return line;
}
If I'm calling some cat file | grep asd, I'm getting the expected result. But not all commands works correctly. For example with this:
cat /proc/cpuinfo | wc -l
or this:
cat /proc/cpuinfo | grep "model name" | head -n 1 | awk -F":" '{print substr($2, 2, length($2))}
the method will return null. I'm guessing this problem depends on output formatting commands like head, tail, wc, etc. How I can work around this problem and get the final result of the output?

The pipe (like redirection, or >) is a function of the shell, and so execing directly from Java won't work. You need to do something like:
/bin/sh -c "your | piped | commands | here"
which executes a shell process with the command line (including pipes) specified after the -c (in quotes).
Note also that you have to consume stdout and stderr concurrently, otherwise your spawned process will block waiting for your process to consume the output (or errors). More info here.

Everyone who uses Runtime.exec should read this.

It might be a good idea to check the error stream of the Process as well.

Still didn't found proper solution to execute piped commands with Runtime.exec, but found a workaround. I've simply wrote these scripts to separate bash files. Then Runtime.exec calls these bash scripts and gets expected result.

The quick-and-dirty thing to do would be:
command = "/bin/sh -c '" + command.replaceAll("'", "'\''") + "'"
Normally, you'll have to watch out for shell injection (i.e. someone sneaks "; rm -rf /;" into the command). But that's only an issue if part of the command can be supplied from some other user input.
The slow and painful approach would be to do the Bash piping yourself in Java. If you go down this road, you'll find out all the wonderful things that Bash gives you that's not directly available from Process.exec (pipes, redirection, compound commands, variable expansion, arithmetic evaluation, ...).
Parse the command for | characters. Be sure to watch out for || and quoted strings.
Spawn a new Process for every piped command.
Create Threads that read the output from one command and write it to the input of the next command.

Probably a little too late but for others looking for a solution, try this...
String[] cmd = {
"/bin/sh",
"-c",
"cat /proc/cpuinfo | wc -l"
};
Process process = Runtime.getRuntime().exec(cmd);
All the best..