We Keep Coding

process.waitFor(timeout, timeUnit) does not quit the process after specified time

I'm trying to execute a visual basic script code in my java application using process builder. As script provided by the user might not finish its execution in time, I want to provide means to limit this execution time. In the following code, you can see my logic but it doesn't really do what it supposed to do. How can I make this waitfor work in order to limit the execution time?
private void run(String scriptFilePath) throws ScriptPluginException {
BufferedReader input = null;
BufferedReader error = null;
try {
ProcessBuilder p = new ProcessBuilder("cscript.exe", "//U", "\"" + scriptFilePath + "\"");
String path = "";
if (scriptFilePath.indexOf("/") != -1) {
path = scriptFilePath.substring(0, scriptFilePath.lastIndexOf("/"));
}
path += "/" + "tempvbsoutput.txt";
p.redirectOutput(new File(path));
Process pp = p.start();
try {
pp.waitFor(executionTimeout, TimeUnit.MINUTES);
} catch (InterruptedException e) {
SystemLog.writeError(jobId, ScriptConsts.COMPONENT_ID, "VBScriptExecutor", "run", 80401104,
"VB Script executes fail.");
}
if (!pp.isAlive()) {
pp.getOutputStream().close();
}
// rest of the code flow
}

Process.waitFor(long, TimeUnit) waits until the process has terminated or the specified time elapsed (Javadoc). The return value indicates whether the process exited or not.
if (process.waitFor(1, TimeUnit.MINUTES)) {
System.out.println("process exited");
} else {
System.out.println("process is still running");
}
waitFor() does not kill the process after the time elapsed.
If you want to kill the subprocess, use either destroy() or destroyForcibly().

Java - Use Input and OutputStream of ProcessBuilder continuously

I want to use an external tool while extracting some data (loop through lines).
For that I first used Runtime.getRuntime().exec() to execute it.
But then my extraction got really slow. So I am searching for a possibility to exec the external tool in each instance of the loop, using the same instance of shell.
I found out, that I should use ProcessBuilder. But it's not working yet.
Here is my code to test the execution (with input from the answers here in the forum already):
public class ExecuteShell {
ProcessBuilder builder;
Process process = null;
BufferedWriter process_stdin;
BufferedReader reader, errReader;
public ExecuteShell() {
String command;
command = getShellCommandForOperatingSystem();
if(command.equals("")) {
return; //Fehler! No error handling yet
}
//init shell
builder = new ProcessBuilder( command);
builder.redirectErrorStream(true);
try {
process = builder.start();
} catch (IOException e) {
System.out.println(e);
}
//get stdout of shell
reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
errReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
//get stdin of shell
process_stdin = new BufferedWriter(new OutputStreamWriter(process.getOutputStream()));
System.out.println("ExecuteShell: Constructor successfully finished");
}
public String executeCommand(String commands) {
StringBuffer output;
String line;
try {
//single execution
process_stdin.write(commands);
process_stdin.newLine();
process_stdin.flush();
} catch (IOException e) {
System.out.println(e);
}
output = new StringBuffer();
line = "";
try {
if (!reader.ready()) {
output.append("Reader empty \n");
return output.toString();
}
while ((line = reader.readLine())!= null) {
output.append(line + "\n");
return output.toString();
}
if (!reader.ready()) {
output.append("errReader empty \n");
return output.toString();
}
while ((line = errReader.readLine())!= null) {
output.append(line + "\n");
}
} catch (Exception e) {
System.out.println("ExecuteShell: error in executeShell2File");
e.printStackTrace();
return "";
}
return output.toString();
}
public int close() {
// finally close the shell by execution exit command
try {
process_stdin.write("exit");
process_stdin.newLine();
process_stdin.flush();
}
catch (IOException e) {
System.out.println(e);
return 1;
}
return 0;
}
private static String getShellCommandForOperatingSystem() {
Properties prop = System.getProperties( );
String os = prop.getProperty( "os.name" );
if ( os.startsWith("Windows") ) {
//System.out.println("WINDOWS!");
return "C:/cygwin64/bin/bash";
} else if (os.startsWith("Linux") ) {
//System.out.println("Linux!");
return"/bin/sh";
}
return "";
}
}
I want to call it in another Class like this Testclass:
public class TestExec{
public static void main(String[] args) {
String result = "";
ExecuteShell es = new ExecuteShell();
for (int i=0; i<5; i++) {
// do something
result = es.executeCommand("date"); //execute some command
System.out.println("result:\n" + result); //do something with result
// do something
}
es.close();
}
}
My Problem is, that the output stream is always empty:
ExecuteShell: Constructor successfully finished
result:
Reader empty
result:
Reader empty
result:
Reader empty
result:
Reader empty
result:
Reader empty
I read the thread here: Java Process with Input/Output Stream
But the code snippets were not enough to get me going, I am missing something. I have not really worked with different threads much. And I am not sure if/how a Scanner is of any help to me. I would really appreciate some help.
Ultimatively, my goal is to call an external command repeatetly and make it fast.
EDIT:
I changed the loop, so that the es.close() is outside. And I wanted to add, that I do not want only this inside the loop.
EDIT:
The problem with the time was, that the command I called caused an error. When the command does not cause an error, the time is acceptable.
Thank you for your answers

You are probably experiencing a race condition: after writing the command to the shell, your Java program continues to run, and almost immediately calls reader.ready(). The command you wanted to execute has probably not yet output anything, so the reader has no data available. An alternative explanation would be that the command does not write anything to stdout, but only to stderr (or the shell, maybe it has failed to start the command?). You are however not reading from stderr in practice.
To properly handle output and error streams, you cannot check reader.ready() but need to call readLine() (which waits until data is available) in a loop. With your code, even if the program would come to that point, you would read only exactly one line from the output. If the program would output more than one line, this data would get interpreted as the output of the next command. The typical solution is to read in a loop until readLine() returns null, but this does not work here because this would mean your program would wait in this loop until the shell terminates (which would never happen, so it would just hang infinitely).
Fixing this would be pretty much impossible, if you do not know exactly how many lines each command will write to stdout and stderr.
However, your complicated approach of using a shell and sending commands to it is probably completely unnecessary. Starting a command from within your Java program and from within the shell is equally fast, and much easier to write. Similarly, there is no performance difference between Runtime.exec() and ProcessBuilder (the former just calls the latter), you only need ProcessBuilder if you need its advanced features.
If you are experiencing performance problems when calling external programs, you should find out where they are exactly and try to solve them, but not with this approach. For example, normally one starts a thread for reading from both the output and the error stream (if you do not start separate threads and the command produces large output, everything might hang). This could be slow, so you could use a thread pool to avoid repeated spawning of processes.

Exec'ing multiple processes from Java: outputs mixed up?

I have a servlet which creates a new Action object inside doGet(), and this object uses exec() to run external processes. There may be several requests to the servlet at the same time, so I might have several Action objects where each one is running an external process at the same time. Occasionally when this happens I find that the output from one process gets mixed up with the output from one of the others.
Each Action creates a unique temporary directory and runs the process with this as its current directory. Separate threads then read the output streams from the process into a string buffer. The code that the Action object executes looks basically like this:
Process proc = null;
ReaderThread stdout = null;
ReaderThread stderr = null;
StringBuffer buff = new StringBuffer();
int exit = -1;
try {
//
// Run the process
//
proc = Runtime.getRuntime().exec(command,null,directory);
//
// Read the output from the process
//
stdout = new ReaderThread(proc.getInputStream(),buff);
stderr = new ReaderThread(proc.getErrorStream(),buff);
stdout.start();
stderr.start();
//
// Get the exit code
//
exit = proc.waitFor();
//
// Wait for all the output to be read
//
stdout.join();
stderr.join();
}
catch (InterruptedException e) {
if (proc != null) {
proc.destroy();
}
}
catch (Exception e) {
buff.append(e.getClass() + ": " + e.getMessage());
if (proc != null) {
proc.destroy();
}
}
So each request uses a separate Action object to run a process, and this has its own StringBuffer "buff" that the output of the process is accumulated into by the two ReaderThreads. But what I find is that, when two requests are running two processes at the same time, the output of one will sometimes end up in the StringBuffer of the thread that is running the other one, and one of the two servlet requests will see output intended for the other one. It basically behaves as if Runtime.exec() provides a single global pipe to which the output streams of all the processes are connected.
The ReaderThread looks like this:
public class ReaderThread extends Thread {
private BufferedReader reader;
private StringBuffer buffer;
public ReaderThread (InputStream stream, StringBuffer buffer) {
this.reader = new BufferedReader(new InputStreamReader(stream));
this.buffer = buffer;
}
#Override
public void run () {
try {
String line;
while ((line = reader.readLine()) != null) {
synchronized (buffer) {
buffer.append(line + "\n");
}
}
}
catch (IOException e) {
synchronized (buffer) {
buffer.append(e.getMessage() + "\n");
}
}
}
}
Can anyone suggest what I can do to fix this?

Use a ThreadLocal variable to store the output of each thread.
When and how should I use a ThreadLocal variable?

Here's the explanation, partly as a cautionary tale:
the output was being added to an ArrayList in the XML node that
accessed it
the XML nodes were created by cloning prototypes
the ArrayList was initialised in the declaration, not in the initialise()
method of the class, so every instance ended up referring to the same
object.
Duh.
Another two days of my life down the drain!
Happy new year...

Java Process.waitFor() and IO streams

I have the following code;
String[] cmd = { "bash", "-c", "~/path/to/script.sh" };
Process p = Runtime.getRuntime().exec(cmd);
PipeThread a = new PipeThread(p.getInputStream(), System.out);
PipeThread b = new PipeThread(p.getErrorStream(), System.err);
p.waitFor();
a.die();
b.die();
The PipeThread class is quite simple so I will include it in full;
public class PipeThread implements Runnable {
private BufferedInputStream in;
private BufferedOutputStream out;
public Thread thread;
private boolean die = false;
public PipeThread(InputStream i, OutputStream o) {
in = new BufferedInputStream(i);
out = new BufferedOutputStream(o);
thread = new Thread(this);
thread.start();
}
public void die() { die = true; }
public void run() {
try {
byte[] b = new byte[1024];
while(!die) {
int x = in.read(b, 0, 1024);
if(x > 0) out.write(b, 0, x);
else die();
out.flush();
}
}
catch(Exception e) { e.printStackTrace(); }
try {
in.close();
out.close();
}
catch(Exception e) { }
}
}
My problem is this; p.waitFor() blocks endlessly, even after the subprocess has terminated. If I do not create the pair of PipeThread instances, then p.waitFor() works perfectly. What is it about the piping of io streams that is causing p.waitFor() to continue blocking?
I'm confused as I thought the IO streams would be passive, unable to keep a process alive, or to make Java think the process is still alive.

In your PipeThread code you are looping forever until !die - but you call PipeThread.die() after p.waitFor() - what's exactly stopping the PipeThread threads?

So after much head-scratching, I realised what is happening. p.waitFor() actually isn't blocking indefinitely, rather my method of checking it is failing; a System.out.println() statement after p.waitFor().
PipeThread is a class that I had lying around from a previous project, and I have often used it to pipe one stream onto another in a separate thread.
This must be the first time I have used it with System.out. PipeThread closes whatever streams it is passed upon reading EOF. Both streams in this case included my standard output, so debugging with System.out.println becomes impossible... :(
Weirdly, no IOException was thrown by System.out.println(), I will investigate.

process.waitFor() never returns

Process process = Runtime.getRuntime().exec("tasklist");
BufferedReader reader =
new BufferedReader(new InputStreamReader(process.getInputStream()));
process.waitFor();

There are many reasons that waitFor() doesn't return.
But it usually boils down to the fact that the executed command doesn't quit.
This, again, can have many reasons.
One common reason is that the process produces some output and you don't read from the appropriate streams. This means that the process is blocked as soon as the buffer is full and waits for your process to continue reading. Your process in turn waits for the other process to finish (which it won't because it waits for your process, ...). This is a classical deadlock situation.
You need to continually read from the processes input stream to ensure that it doesn't block.
There's a nice article that explains all the pitfalls of Runtime.exec() and shows ways around them called "When Runtime.exec() won't" (yes, the article is from 2000, but the content still applies!)

It appears you are not reading the output before waiting for it to finish. This is fine only if the output doesn't fill the buffer. If it does, it will wait until you read the output, catch-22.
Perhaps you have some errors which you are not reading. This would case the application to stop and waitFor to wait forever. A simple way around this is to re-direct the errors to the regular output.
ProcessBuilder pb = new ProcessBuilder("tasklist");
pb.redirectErrorStream(true);
Process process = pb.start();
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
while ((line = reader.readLine()) != null)
System.out.println("tasklist: " + line);
process.waitFor();

Also from Java doc:
java.lang
Class Process
Because some native platforms only provide limited buffer size for standard input and
output streams, failure to promptly write the input stream or read the output stream of
the subprocess may cause the subprocess to block, and even deadlock.
Fail to clear the buffer of input stream (which pipes to the output stream of subprocess)
from Process may lead to a subprocess blocking.
Try this:
Process process = Runtime.getRuntime().exec("tasklist");
BufferedReader reader =
new BufferedReader(new InputStreamReader(process.getInputStream()));
while ((reader.readLine()) != null) {}
process.waitFor();

I would like to add something to the previous answers but since I don't have the rep to comment, I will just add an answer. This is directed towards android users which are programming in Java.
Per the post from RollingBoy, this code almost worked for me:
Process process = Runtime.getRuntime().exec("tasklist");
BufferedReader reader =
new BufferedReader(new InputStreamReader(process.getInputStream()));
while ((reader.readLine()) != null) {}
process.waitFor();
In my case, the waitFor() was not releasing because I was executing a statement with no return ("ip adddr flush eth0"). An easy way to fix this is to simply ensure you always return something in your statement. For me, that meant executing the following: "ip adddr flush eth0 && echo done". You can read the buffer all day, but if there is nothing ever returned, your thread will never release its wait.
Hope that helps someone!

There are several possibilities:
You haven't consumed all the output on the process's stdout.
You haven't consumed all the output on the process's stderr.
The process is waiting for input from you and you haven't provided it, or you haven't closed the process's stdin.
The process is spinning in a hard loop.

As others have mentioned you have to consume stderr and stdout.
Compared to the other answers, since Java 1.7 it is even more easy. You do not have to create threads yourself anymore to read stderr and stdout.
Just use the ProcessBuilder and use the methods redirectOutput in combination with either redirectError or redirectErrorStream.
String directory = "/working/dir";
File out = new File(...); // File to write stdout to
File err = new File(...); // File to write stderr to
ProcessBuilder builder = new ProcessBuilder();
builder.directory(new File(directory));
builder.command(command);
builder.redirectOutput(out); // Redirect stdout to file
if(out == err) {
builder.redirectErrorStream(true); // Combine stderr into stdout
} else {
builder.redirectError(err); // Redirect stderr to file
}
Process process = builder.start();

For the same reason you can also use inheritIO() to map Java console with external app console like:
ProcessBuilder pb = new ProcessBuilder(appPath, arguments);
pb.directory(new File(appFile.getParent()));
pb.inheritIO();
Process process = pb.start();
int success = process.waitFor();

You should try consume output and error in the same while
private void runCMD(String CMD) throws IOException, InterruptedException {
System.out.println("Standard output: " + CMD);
Process process = Runtime.getRuntime().exec(CMD);
// Get input streams
BufferedReader stdInput = new BufferedReader(new InputStreamReader(process.getInputStream()));
BufferedReader stdError = new BufferedReader(new InputStreamReader(process.getErrorStream()));
String line = "";
String newLineCharacter = System.getProperty("line.separator");
boolean isOutReady = false;
boolean isErrorReady = false;
boolean isProcessAlive = false;
boolean isErrorOut = true;
boolean isErrorError = true;
System.out.println("Read command ");
while (process.isAlive()) {
//Read the stdOut
do {
isOutReady = stdInput.ready();
//System.out.println("OUT READY " + isOutReady);
isErrorOut = true;
isErrorError = true;
if (isOutReady) {
line = stdInput.readLine();
isErrorOut = false;
System.out.println("=====================================================================================" + line + newLineCharacter);
}
isErrorReady = stdError.ready();
//System.out.println("ERROR READY " + isErrorReady);
if (isErrorReady) {
line = stdError.readLine();
isErrorError = false;
System.out.println("ERROR::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::" + line + newLineCharacter);
}
isProcessAlive = process.isAlive();
//System.out.println("Process Alive " + isProcessAlive);
if (!isProcessAlive) {
System.out.println(":::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: Process DIE " + line + newLineCharacter);
line = null;
isErrorError = false;
process.waitFor(1000, TimeUnit.MILLISECONDS);
}
} while (line != null);
//Nothing else to read, lets pause for a bit before trying again
System.out.println("PROCESS WAIT FOR");
process.waitFor(100, TimeUnit.MILLISECONDS);
}
System.out.println("Command finished");
}

I think I observed a similar problem: some processes started, seemed to run successfully but never completed. The function waitFor() was waiting forever except if I killed the process in Task Manager.
However, everything worked well in cases the length of the command line was 127 characters or shorter. If long file names are inevitable you may want to use environmental variables, which may allow you keeping the command line string short. You can generate a batch file (using FileWriter) in which you set your environmental variables before calling the program you actually want to run.
The content of such a batch could look like:
set INPUTFILE="C:\Directory 0\Subdirectory 1\AnyFileName"
set OUTPUTFILE="C:\Directory 2\Subdirectory 3\AnotherFileName"
set MYPROG="C:\Directory 4\Subdirectory 5\ExecutableFileName.exe"
%MYPROG% %INPUTFILE% %OUTPUTFILE%
Last step is running this batch file using Runtime.

Here is a method that works for me.
NOTE: There is some code within this method that may not apply to you, so try and ignore it. For example "logStandardOut(...), git-bash, etc".
private String exeShellCommand(String doCommand, String inDir, boolean ignoreErrors) {
logStandardOut("> %s", doCommand);
ProcessBuilder builder = new ProcessBuilder();
StringBuilder stdOut = new StringBuilder();
StringBuilder stdErr = new StringBuilder();
boolean isWindows = System.getProperty("os.name").toLowerCase().startsWith("windows");
if (isWindows) {
String gitBashPathForWindows = "C:\\Program Files\\Git\\bin\\bash";
builder.command(gitBashPathForWindows, "-c", doCommand);
} else {
builder.command("bash", "-c", doCommand);
}
//Do we need to change dirs?
if (inDir != null) {
builder.directory(new File(inDir));
}
//Execute it
Process process = null;
BufferedReader brStdOut;
BufferedReader brStdErr;
try {
//Start the command line process
process = builder.start();
//This hangs on a large file
// https://stackoverflow.com/questions/5483830/process-waitfor-never-returns
//exitCode = process.waitFor();
//This will have both StdIn and StdErr
brStdOut = new BufferedReader(new InputStreamReader(process.getInputStream()));
brStdErr = new BufferedReader(new InputStreamReader(process.getErrorStream()));
//Get the process output
String line = null;
String newLineCharacter = System.getProperty("line.separator");
while (process.isAlive()) {
//Read the stdOut
while ((line = brStdOut.readLine()) != null) {
stdOut.append(line + newLineCharacter);
}
//Read the stdErr
while ((line = brStdErr.readLine()) != null) {
stdErr.append(line + newLineCharacter);
}
//Nothing else to read, lets pause for a bit before trying again
process.waitFor(100, TimeUnit.MILLISECONDS);
}
//Read anything left, after the process exited
while ((line = brStdOut.readLine()) != null) {
stdOut.append(line + newLineCharacter);
}
//Read anything left, after the process exited
while ((line = brStdErr.readLine()) != null) {
stdErr.append(line + newLineCharacter);
}
//cleanup
if (brStdOut != null) {
brStdOut.close();
}
if (brStdErr != null) {
brStdOut.close();
}
//Log non-zero exit values
if (!ignoreErrors && process.exitValue() != 0) {
String exMsg = String.format("%s%nprocess.exitValue=%s", stdErr, process.exitValue());
throw new ExecuteCommandException(exMsg);
}
} catch (ExecuteCommandException e) {
throw e;
} catch (Exception e) {
throw new ExecuteCommandException(stdErr.toString(), e);
} finally {
//Log the results
logStandardOut(stdOut.toString());
logStandardError(stdErr.toString());
}
return stdOut.toString();
}

Asynchronous reading of stream combined with avoiding Wait with a timeout will solve the problem.
You can find a page explaining this here http://simplebasics.net/.net/process-waitforexit-with-a-timeout-will-not-be-able-to-collect-the-output-message/

public static void main(String[] args) throws PyException, IOException, InterruptedException
these should be the exceptions thrown