We Keep Coding

Java - Use Input and OutputStream of ProcessBuilder continuously

I want to use an external tool while extracting some data (loop through lines).
For that I first used Runtime.getRuntime().exec() to execute it.
But then my extraction got really slow. So I am searching for a possibility to exec the external tool in each instance of the loop, using the same instance of shell.
I found out, that I should use ProcessBuilder. But it's not working yet.
Here is my code to test the execution (with input from the answers here in the forum already):
public class ExecuteShell {
ProcessBuilder builder;
Process process = null;
BufferedWriter process_stdin;
BufferedReader reader, errReader;
public ExecuteShell() {
String command;
command = getShellCommandForOperatingSystem();
if(command.equals("")) {
return; //Fehler! No error handling yet
}
//init shell
builder = new ProcessBuilder( command);
builder.redirectErrorStream(true);
try {
process = builder.start();
} catch (IOException e) {
System.out.println(e);
}
//get stdout of shell
reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
errReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
//get stdin of shell
process_stdin = new BufferedWriter(new OutputStreamWriter(process.getOutputStream()));
System.out.println("ExecuteShell: Constructor successfully finished");
}
public String executeCommand(String commands) {
StringBuffer output;
String line;
try {
//single execution
process_stdin.write(commands);
process_stdin.newLine();
process_stdin.flush();
} catch (IOException e) {
System.out.println(e);
}
output = new StringBuffer();
line = "";
try {
if (!reader.ready()) {
output.append("Reader empty \n");
return output.toString();
}
while ((line = reader.readLine())!= null) {
output.append(line + "\n");
return output.toString();
}
if (!reader.ready()) {
output.append("errReader empty \n");
return output.toString();
}
while ((line = errReader.readLine())!= null) {
output.append(line + "\n");
}
} catch (Exception e) {
System.out.println("ExecuteShell: error in executeShell2File");
e.printStackTrace();
return "";
}
return output.toString();
}
public int close() {
// finally close the shell by execution exit command
try {
process_stdin.write("exit");
process_stdin.newLine();
process_stdin.flush();
}
catch (IOException e) {
System.out.println(e);
return 1;
}
return 0;
}
private static String getShellCommandForOperatingSystem() {
Properties prop = System.getProperties( );
String os = prop.getProperty( "os.name" );
if ( os.startsWith("Windows") ) {
//System.out.println("WINDOWS!");
return "C:/cygwin64/bin/bash";
} else if (os.startsWith("Linux") ) {
//System.out.println("Linux!");
return"/bin/sh";
}
return "";
}
}
I want to call it in another Class like this Testclass:
public class TestExec{
public static void main(String[] args) {
String result = "";
ExecuteShell es = new ExecuteShell();
for (int i=0; i<5; i++) {
// do something
result = es.executeCommand("date"); //execute some command
System.out.println("result:\n" + result); //do something with result
// do something
}
es.close();
}
}
My Problem is, that the output stream is always empty:
ExecuteShell: Constructor successfully finished
result:
Reader empty
result:
Reader empty
result:
Reader empty
result:
Reader empty
result:
Reader empty
I read the thread here: Java Process with Input/Output Stream
But the code snippets were not enough to get me going, I am missing something. I have not really worked with different threads much. And I am not sure if/how a Scanner is of any help to me. I would really appreciate some help.
Ultimatively, my goal is to call an external command repeatetly and make it fast.
EDIT:
I changed the loop, so that the es.close() is outside. And I wanted to add, that I do not want only this inside the loop.
EDIT:
The problem with the time was, that the command I called caused an error. When the command does not cause an error, the time is acceptable.
Thank you for your answers

You are probably experiencing a race condition: after writing the command to the shell, your Java program continues to run, and almost immediately calls reader.ready(). The command you wanted to execute has probably not yet output anything, so the reader has no data available. An alternative explanation would be that the command does not write anything to stdout, but only to stderr (or the shell, maybe it has failed to start the command?). You are however not reading from stderr in practice.
To properly handle output and error streams, you cannot check reader.ready() but need to call readLine() (which waits until data is available) in a loop. With your code, even if the program would come to that point, you would read only exactly one line from the output. If the program would output more than one line, this data would get interpreted as the output of the next command. The typical solution is to read in a loop until readLine() returns null, but this does not work here because this would mean your program would wait in this loop until the shell terminates (which would never happen, so it would just hang infinitely).
Fixing this would be pretty much impossible, if you do not know exactly how many lines each command will write to stdout and stderr.
However, your complicated approach of using a shell and sending commands to it is probably completely unnecessary. Starting a command from within your Java program and from within the shell is equally fast, and much easier to write. Similarly, there is no performance difference between Runtime.exec() and ProcessBuilder (the former just calls the latter), you only need ProcessBuilder if you need its advanced features.
If you are experiencing performance problems when calling external programs, you should find out where they are exactly and try to solve them, but not with this approach. For example, normally one starts a thread for reading from both the output and the error stream (if you do not start separate threads and the command produces large output, everything might hang). This could be slow, so you could use a thread pool to avoid repeated spawning of processes.

Is there a better way to read from a process's inputstream and then handle using specified methods?

I am writing a program doing the following works:
Run a command using ProcessBuilder (like "svn info" or "svn diff");
Read the output of the command from the process's getInputStream();
With the output of the command, I want either:
Parse the output and get what I want and use it later, OR:
Write the output directly to a specified file.
Now what I am doing is using BufferedReader to read whatever the command outputs by lines and save them to an ArrayList, and then decide if I would just scan the lines to find out something or write the lines to a file.
Obviously this is an ugly implement because the ArrayList should not be needed if I want a command's output to be saved to a file. So what will you suggest, to do it in a better way?
Here is some of my codes:
Use this to run command and read from the output of the process
private ArrayList<String> runCommand(String[] command) throws IOException {
ArrayList<String> result = new ArrayList<>();
_processBuilder.command(command);
Process process = null;
try {
process = _processBuilder.start();
try (InputStream inputStream = process.getInputStream();
InputStreamReader inputStreamReader = new InputStreamReader(inputStream);
BufferedReader bufferedReader = new BufferedReader(inputStreamReader)) {
String line;
while ((line = bufferedReader.readLine()) != null) {
result.add(line);
}
}
}
catch (IOException ex) {
_logger.log(Level.SEVERE, "Error!", ex);
}
finally {
if (process != null) {
try {
process.waitFor();
}
catch (InterruptedException ex) {
_logger.log(Level.SEVERE, null, ex);
}
}
}
return result;
}
and in one method I may do like this:
ArrayList<String> reuslt = runCommand(command1);
for (String line: result) {
// ...parse the line here...
}
and in another I may do like this:
ArrayList<String> result = runCommand(command2);
File file = new File(...filename, etc...);
try (PrintWriter printWriter = new PrintWriter(new FileWriter(file, false))) {
for (String line: result) {
printWriter.println(line);
}
}

Returning the process output in an ArrayList seems like a fine abstraction to me. Then the caller of runCommand() doesn't need to worry about how the command was run or the output read. The memory used by the extra list is probably not significant unless your command is very prolix.
The only time I could see this being an issue would be if the caller wanted to start processing the output while the command was still running, which doesn't seem to be the case here.
For very big output that you don't want to copy into memory first, one option would be to have runCommand() take a callback like Guava's LineProcessor that it will call for each line of the output. Then runCommand() can still abstract away the whole deal of running the process, reading the output, and closing everything afterwards, but data can be passed out to the callback as it runs rather than waiting for the method to return the whole response in one array.

I don't think it's a performance issue that you store the text uselessly in some cases. Nonetheless, for cleanliness, it might be better to write two methods:
private ArrayList<String> runCommand(String[] command)
private void runCommandAndDumpToFile(String[] command, File file)
(It wasn't quite clear from your question, but I assume that you know before running your process whether you'll just write the output to file or process it.)

process.waitFor() never returns

Process process = Runtime.getRuntime().exec("tasklist");
BufferedReader reader =
new BufferedReader(new InputStreamReader(process.getInputStream()));
process.waitFor();

There are many reasons that waitFor() doesn't return.
But it usually boils down to the fact that the executed command doesn't quit.
This, again, can have many reasons.
One common reason is that the process produces some output and you don't read from the appropriate streams. This means that the process is blocked as soon as the buffer is full and waits for your process to continue reading. Your process in turn waits for the other process to finish (which it won't because it waits for your process, ...). This is a classical deadlock situation.
You need to continually read from the processes input stream to ensure that it doesn't block.
There's a nice article that explains all the pitfalls of Runtime.exec() and shows ways around them called "When Runtime.exec() won't" (yes, the article is from 2000, but the content still applies!)

It appears you are not reading the output before waiting for it to finish. This is fine only if the output doesn't fill the buffer. If it does, it will wait until you read the output, catch-22.
Perhaps you have some errors which you are not reading. This would case the application to stop and waitFor to wait forever. A simple way around this is to re-direct the errors to the regular output.
ProcessBuilder pb = new ProcessBuilder("tasklist");
pb.redirectErrorStream(true);
Process process = pb.start();
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
while ((line = reader.readLine()) != null)
System.out.println("tasklist: " + line);
process.waitFor();

Also from Java doc:
java.lang
Class Process
Because some native platforms only provide limited buffer size for standard input and
output streams, failure to promptly write the input stream or read the output stream of
the subprocess may cause the subprocess to block, and even deadlock.
Fail to clear the buffer of input stream (which pipes to the output stream of subprocess)
from Process may lead to a subprocess blocking.
Try this:
Process process = Runtime.getRuntime().exec("tasklist");
BufferedReader reader =
new BufferedReader(new InputStreamReader(process.getInputStream()));
while ((reader.readLine()) != null) {}
process.waitFor();

I would like to add something to the previous answers but since I don't have the rep to comment, I will just add an answer. This is directed towards android users which are programming in Java.
Per the post from RollingBoy, this code almost worked for me:
Process process = Runtime.getRuntime().exec("tasklist");
BufferedReader reader =
new BufferedReader(new InputStreamReader(process.getInputStream()));
while ((reader.readLine()) != null) {}
process.waitFor();
In my case, the waitFor() was not releasing because I was executing a statement with no return ("ip adddr flush eth0"). An easy way to fix this is to simply ensure you always return something in your statement. For me, that meant executing the following: "ip adddr flush eth0 && echo done". You can read the buffer all day, but if there is nothing ever returned, your thread will never release its wait.
Hope that helps someone!

There are several possibilities:
You haven't consumed all the output on the process's stdout.
You haven't consumed all the output on the process's stderr.
The process is waiting for input from you and you haven't provided it, or you haven't closed the process's stdin.
The process is spinning in a hard loop.

As others have mentioned you have to consume stderr and stdout.
Compared to the other answers, since Java 1.7 it is even more easy. You do not have to create threads yourself anymore to read stderr and stdout.
Just use the ProcessBuilder and use the methods redirectOutput in combination with either redirectError or redirectErrorStream.
String directory = "/working/dir";
File out = new File(...); // File to write stdout to
File err = new File(...); // File to write stderr to
ProcessBuilder builder = new ProcessBuilder();
builder.directory(new File(directory));
builder.command(command);
builder.redirectOutput(out); // Redirect stdout to file
if(out == err) {
builder.redirectErrorStream(true); // Combine stderr into stdout
} else {
builder.redirectError(err); // Redirect stderr to file
}
Process process = builder.start();

For the same reason you can also use inheritIO() to map Java console with external app console like:
ProcessBuilder pb = new ProcessBuilder(appPath, arguments);
pb.directory(new File(appFile.getParent()));
pb.inheritIO();
Process process = pb.start();
int success = process.waitFor();

You should try consume output and error in the same while
private void runCMD(String CMD) throws IOException, InterruptedException {
System.out.println("Standard output: " + CMD);
Process process = Runtime.getRuntime().exec(CMD);
// Get input streams
BufferedReader stdInput = new BufferedReader(new InputStreamReader(process.getInputStream()));
BufferedReader stdError = new BufferedReader(new InputStreamReader(process.getErrorStream()));
String line = "";
String newLineCharacter = System.getProperty("line.separator");
boolean isOutReady = false;
boolean isErrorReady = false;
boolean isProcessAlive = false;
boolean isErrorOut = true;
boolean isErrorError = true;
System.out.println("Read command ");
while (process.isAlive()) {
//Read the stdOut
do {
isOutReady = stdInput.ready();
//System.out.println("OUT READY " + isOutReady);
isErrorOut = true;
isErrorError = true;
if (isOutReady) {
line = stdInput.readLine();
isErrorOut = false;
System.out.println("=====================================================================================" + line + newLineCharacter);
}
isErrorReady = stdError.ready();
//System.out.println("ERROR READY " + isErrorReady);
if (isErrorReady) {
line = stdError.readLine();
isErrorError = false;
System.out.println("ERROR::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::" + line + newLineCharacter);
}
isProcessAlive = process.isAlive();
//System.out.println("Process Alive " + isProcessAlive);
if (!isProcessAlive) {
System.out.println(":::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: Process DIE " + line + newLineCharacter);
line = null;
isErrorError = false;
process.waitFor(1000, TimeUnit.MILLISECONDS);
}
} while (line != null);
//Nothing else to read, lets pause for a bit before trying again
System.out.println("PROCESS WAIT FOR");
process.waitFor(100, TimeUnit.MILLISECONDS);
}
System.out.println("Command finished");
}

I think I observed a similar problem: some processes started, seemed to run successfully but never completed. The function waitFor() was waiting forever except if I killed the process in Task Manager.
However, everything worked well in cases the length of the command line was 127 characters or shorter. If long file names are inevitable you may want to use environmental variables, which may allow you keeping the command line string short. You can generate a batch file (using FileWriter) in which you set your environmental variables before calling the program you actually want to run.
The content of such a batch could look like:
set INPUTFILE="C:\Directory 0\Subdirectory 1\AnyFileName"
set OUTPUTFILE="C:\Directory 2\Subdirectory 3\AnotherFileName"
set MYPROG="C:\Directory 4\Subdirectory 5\ExecutableFileName.exe"
%MYPROG% %INPUTFILE% %OUTPUTFILE%
Last step is running this batch file using Runtime.

Here is a method that works for me.
NOTE: There is some code within this method that may not apply to you, so try and ignore it. For example "logStandardOut(...), git-bash, etc".
private String exeShellCommand(String doCommand, String inDir, boolean ignoreErrors) {
logStandardOut("> %s", doCommand);
ProcessBuilder builder = new ProcessBuilder();
StringBuilder stdOut = new StringBuilder();
StringBuilder stdErr = new StringBuilder();
boolean isWindows = System.getProperty("os.name").toLowerCase().startsWith("windows");
if (isWindows) {
String gitBashPathForWindows = "C:\\Program Files\\Git\\bin\\bash";
builder.command(gitBashPathForWindows, "-c", doCommand);
} else {
builder.command("bash", "-c", doCommand);
}
//Do we need to change dirs?
if (inDir != null) {
builder.directory(new File(inDir));
}
//Execute it
Process process = null;
BufferedReader brStdOut;
BufferedReader brStdErr;
try {
//Start the command line process
process = builder.start();
//This hangs on a large file
// https://stackoverflow.com/questions/5483830/process-waitfor-never-returns
//exitCode = process.waitFor();
//This will have both StdIn and StdErr
brStdOut = new BufferedReader(new InputStreamReader(process.getInputStream()));
brStdErr = new BufferedReader(new InputStreamReader(process.getErrorStream()));
//Get the process output
String line = null;
String newLineCharacter = System.getProperty("line.separator");
while (process.isAlive()) {
//Read the stdOut
while ((line = brStdOut.readLine()) != null) {
stdOut.append(line + newLineCharacter);
}
//Read the stdErr
while ((line = brStdErr.readLine()) != null) {
stdErr.append(line + newLineCharacter);
}
//Nothing else to read, lets pause for a bit before trying again
process.waitFor(100, TimeUnit.MILLISECONDS);
}
//Read anything left, after the process exited
while ((line = brStdOut.readLine()) != null) {
stdOut.append(line + newLineCharacter);
}
//Read anything left, after the process exited
while ((line = brStdErr.readLine()) != null) {
stdErr.append(line + newLineCharacter);
}
//cleanup
if (brStdOut != null) {
brStdOut.close();
}
if (brStdErr != null) {
brStdOut.close();
}
//Log non-zero exit values
if (!ignoreErrors && process.exitValue() != 0) {
String exMsg = String.format("%s%nprocess.exitValue=%s", stdErr, process.exitValue());
throw new ExecuteCommandException(exMsg);
}
} catch (ExecuteCommandException e) {
throw e;
} catch (Exception e) {
throw new ExecuteCommandException(stdErr.toString(), e);
} finally {
//Log the results
logStandardOut(stdOut.toString());
logStandardError(stdErr.toString());
}
return stdOut.toString();
}

Asynchronous reading of stream combined with avoiding Wait with a timeout will solve the problem.
You can find a page explaining this here http://simplebasics.net/.net/process-waitforexit-with-a-timeout-will-not-be-able-to-collect-the-output-message/

public static void main(String[] args) throws PyException, IOException, InterruptedException
these should be the exceptions thrown

Redirect process output to stdout

I would like to execute foo.bat from within a Groovy program and have the resulting process' output redirected to stdout. Either a Java or Groovy code example would be fine.
foo.bat can take several minutes to run and generates a lot of output, so I would like to see the output as soon as it is generated, rather than having to wait until the process has completed before seeing all the output at once.

It is simple to redirect all your stream to standard output using inheritIO() method. This will print the output to the stdout of the process from which you are running this command.
ProcessBuilder pb = new ProcessBuilder("command", "argument");
pb.directory(new File(<directory from where you want to run the command>));
pb.inheritIO();
Process p = pb.start();
p.waitFor();
There exist other methods too, like as mentioned below. These individual methods will help redirect only required stream.
pb.redirectInput(Redirect.INHERIT)
pb.redirectOutput(Redirect.INHERIT)
pb.redirectError(Redirect.INHERIT)

This uses a class which reads all output the executed program generates and displays it in it's own stdout.
class StreamGobbler extends Thread {
InputStream is;
// reads everything from is until empty.
StreamGobbler(InputStream is) {
this.is = is;
}
public void run() {
try {
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line=null;
while ( (line = br.readLine()) != null)
System.out.println(line);
} catch (IOException ioe) {
ioe.printStackTrace();
}
}
}
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec("javac");
//output both stdout and stderr data from proc to stdout of this process
StreamGobbler errorGobbler = new StreamGobbler(proc.getErrorStream());
StreamGobbler outputGobbler = new StreamGobbler(proc.getInputStream());
errorGobbler.start();
outputGobbler.start();
proc.waitFor();

If you're looking to do this with more Groovy and less java, this will print each line as it happens:
def cmd = "./longRunningProcess"
def process = cmd.execute()
process.in.eachLine { line -> println line }
Alternatively, if you want to see both stdout and stderr
def cmd = "./longRunningProcess"
def process = cmd.execute()
process.waitForProcessOutput( System.out, System.err )

Here's something a little simpler if you're just trying to grab the output of a simple command. You'll need to use threads like jitter does if you want to process in parallel or if your command takes stdin or generates stderr.
Use a buffered copy (like this) if you're getting lots of output.
import java.io.*;
public class test {
static void copy(InputStream in, OutputStream out) throws IOException {
while (true) {
int c = in.read();
if (c == -1) break;
out.write((char)c);
}
}
public static void main(String[] args) throws IOException, InterruptedException {
String cmd = "echo foo";
Process p = Runtime.getRuntime().exec(cmd);
copy(p.getInputStream(), System.out);
p.waitFor();
}
}

The following Groovy code will execute foo.bat and send the output to stdout:
println "foo.bat".execute().text

Asynchronous way to achieve it.
void inputStreamToOutputStream(final InputStream inputStream, final OutputStream out) {
Thread t = new Thread(new Runnable() {
public void run() {
try {
int d;
while ((d = inputStream.read()) != -1) {
out.write(d);
}
} catch (IOException ex) {
//TODO make a callback on exception.
}
}
});
t.setDaemon(true);
t.start();
}
{
Process p = ...;
inputStreamToOutputStream(p.getErrorStream(), System.out);
inputStreamToOutputStream(p.getInputStream(), System.out);
}

VerboseProcess from jcabi-log can help you:
String output = new VerboseProcess(new ProcessBuilder("foo.bat")).stdout();

How can i get the console output generated by a exe file?

without using a redirect to file (">", ">>")

Process p = Runtime.getRuntime().exec("executable.exec");
BufferedReader input = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line;
while ((line = input.readLine()) != null) {
System.out.println(line);
}

Note that you should consume stdout and stderr concurrently, to prevent blocking. See this answer for more details.
Note that you may be able to get away with just taking stdout and not stderr. However if you .exe generates an error in some scenario, then your parent process can then block on the extra (unexpected) stream data. So it's always best to run your stream gathering concurrently.

If you use the Commandline type from plexus-utils, you can avoid a lot of the heavy lifting associated with commandline interaction, for example waiting for the process, escaping arguments etc. You can set the command to timeout if needed as well.
You can pass StreamConsumers to capture the stdout and stderr, the Commandline handling will pass the output to the consumers one line at a time.
Commandline cl = new Commandline();
cl.setExecutable( "dir" );
cl.setWorkingDirectory( workingDirectory.getAbsolutePath() );
cl.createArg().setValue( "/S" );
StreamConsumer consumer = new StreamConsumer() {
public void consumeLine( String line ) {
//do something with the line
}
};
StreamConsumer stderr = new StreamConsumer() {
public void consumeLine( String line ) {
//do something with the line
}
};
int exitCode;
try {
exitCode = CommandLineUtils.execute( cl, consumer, stderr, getLogger() );
} catch ( CommandLineException ex ) {
//handle exception
}