I am running a command-line program (ClamAV virus scanner) from Java as below. I need to run it synchronously since it's running in AWS Lambda and if it goes async, the lambda will terminate early. It seems "many" times it comes out without completing. What is the best way to fix this? I added ugly Thread. Sleep improved the situation a little. I actually don't mind going that route but need to find the exact place where it should sleep. Please help!
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec(command);
BufferedReader inputReader = new BufferedReader(new InputStreamReader(proc.getInputStream()));
BufferedReader errorReader = new BufferedReader(new InputStreamReader(proc.getErrorStream()));
String line;
StringBuilder out = new StringBuilder();
int exitValue = proc.waitFor();
logger.info("Exit Value:" + exitValue);
while ((line = inputReader.readLine()) != null) {
out.append("\n").append(line);
}
if (exitValue != 0) {
while ((line = errorReader.readLine()) != null) {
out.append("\n").append(line);
logger.error(line);
}
}
The method exec executes in a different process as stated by the javadoc:
Executes the specified string command in a separate process.
What you can do is put the main thread in a wait state until you know that the separated thread has completed his task. You can do that using the method waitFor of the Process class:
Causes the current thread to wait, if necessary, until the process represented by this Process object has terminated
For some terminal commands, they repeatedly output. For example, for something that's generating a file, it may output the percent that it is complete.
I know how to call terminal commands in Java using
Process p = Runtime.getRuntim().exec("command goes here");
but that doesn't give me a live feed of the current output of the command. How can I do this so that I can do a System.out.println() every 100 milliseconds, for example, to see what the most recent output of the process was.
You need to read InputStream from the process, here is an example:
Edit I modified the code as suggested here to receive the errStream with the stdInput
ProcessBuilder builder = new ProcessBuilder("command goes here");
builder.redirectErrorStream(true);
Process process = builder.start();
InputStream is = process.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
String line = null;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
For debugging purpose, you can read the input as bytes instead of using readLine just in case that the process does not terminate messages with newLine
I'm trying to capture output of an external program using java but I can't.
I have the code to show it, but not to put it into a variable.
I will use, for example, sqlplus to execute my oracle code "into exec.sql"
system/orcl#orcl : user/password/DB name
public static String test_script () {
String RESULT="";
String fileName = "#src\\exec.sql";
String sqlPath = ".";
String arg1="system/orcl#orcl";
String sqlCmd = "sqlplus";
String arg2 = fileName;
try {
String line;
ProcessBuilder pb = new ProcessBuilder(sqlCmd, arg1, arg2);
Map<String, String> env = pb.environment();
env.put("VAR1", arg1);
env.put("VAR2", arg2);
pb.directory(new File(sqlPath));
pb.redirectErrorStream(true);
Process p = pb.start();
BufferedReader bri = new BufferedReader
(new InputStreamReader(p.getInputStream()));
while ((line = bri.readLine()) != null) {
RESULT+=line;
}
System.out.println("Done.");
}
catch (Exception err) {
err.printStackTrace();
}
return RESULT;
}
Because the Process will execute in a new thread it's likely that there is no output or incomplete output available when you come to your while loop.
Process p = pb.start();
// process runs in another thread parallel to this one
BufferedReader bri = new BufferedReader(new InputStreamReader(p.getInputStream()));
// bri may be empty or incomplete.
while ((line = bri.readLine()) != null) {
RESULT+=line;
}
So you need to wait for the process to complete before attempting to interact with it's output. Try using the Process.waitFor() method to pause the current thread until your process has had an opportunity to complete.
Process p = pb.start();
p.waitFor(); // wait for process to finish then continue.
BufferedReader bri = new BufferedReader(new InputStreamReader(p.getInputStream()));
while ((line = bri.readLine()) != null) {
RESULT+=line;
}
This is only a simple approach you could also process the output of the process while it runs in parallel but then you would need to monitor the status of the process i.e. is it still running or has it completed, and the availability of output.
Use Apache Commons Exec, it will make your life much easier. Check the tutorials for information about basic usage. To read the command line output after obtaining an executor object (probably DefaultExecutor), create an OutputStream to whatever stream you wish (i.e a FileOutputStream instance may be, or System.out), and:
executor.setStreamHandler(new PumpStreamHandler(yourOutputStream));
I currently start a shell script from my Java by code looking like this:
ProcessBuilder processBuilder = new ProcessBuilder();
processBuilder.directory("/directory/where/the/script/is/located/");
String[] command = new String[]{"sh", "myScript.sh"};
processBuilder.command(command);
Map<String, String> env = processBuilder.environment();
//tweak the environment with needed additions
env.put(...);
Process p = processBuilder.start();
stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));
stdError = new BufferedReader(new InputStreamReader(p.getErrorStream()));
// read the output from the command
String line;
while ((line = stdInput.readLine()) != null)
{
logger.fine(line);
}
p.waitFor();
int returnCode = p.exitValue();
// Return something according to the return code
return ...;
If I now want to start the script and not wait for it to end (and thus losing the ability to return according to return code), but still with being able to tweak the environment beforehand? how should I proceed?
Thank you
Just do the entire thing in a new thread.
Wrap the entire above code in a Runnable and start a Thread for it. Your main code can continue doing something else.
In the runnable once you get the result from the child process use some notify(or observer pattern) to notify interested parties.
Process process = Runtime.getRuntime().exec("tasklist");
BufferedReader reader =
new BufferedReader(new InputStreamReader(process.getInputStream()));
process.waitFor();
There are many reasons that waitFor() doesn't return.
But it usually boils down to the fact that the executed command doesn't quit.
This, again, can have many reasons.
One common reason is that the process produces some output and you don't read from the appropriate streams. This means that the process is blocked as soon as the buffer is full and waits for your process to continue reading. Your process in turn waits for the other process to finish (which it won't because it waits for your process, ...). This is a classical deadlock situation.
You need to continually read from the processes input stream to ensure that it doesn't block.
There's a nice article that explains all the pitfalls of Runtime.exec() and shows ways around them called "When Runtime.exec() won't" (yes, the article is from 2000, but the content still applies!)
It appears you are not reading the output before waiting for it to finish. This is fine only if the output doesn't fill the buffer. If it does, it will wait until you read the output, catch-22.
Perhaps you have some errors which you are not reading. This would case the application to stop and waitFor to wait forever. A simple way around this is to re-direct the errors to the regular output.
ProcessBuilder pb = new ProcessBuilder("tasklist");
pb.redirectErrorStream(true);
Process process = pb.start();
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
while ((line = reader.readLine()) != null)
System.out.println("tasklist: " + line);
process.waitFor();
Also from Java doc:
java.lang
Class Process
Because some native platforms only provide limited buffer size for standard input and
output streams, failure to promptly write the input stream or read the output stream of
the subprocess may cause the subprocess to block, and even deadlock.
Fail to clear the buffer of input stream (which pipes to the output stream of subprocess)
from Process may lead to a subprocess blocking.
Try this:
Process process = Runtime.getRuntime().exec("tasklist");
BufferedReader reader =
new BufferedReader(new InputStreamReader(process.getInputStream()));
while ((reader.readLine()) != null) {}
process.waitFor();
I would like to add something to the previous answers but since I don't have the rep to comment, I will just add an answer. This is directed towards android users which are programming in Java.
Per the post from RollingBoy, this code almost worked for me:
Process process = Runtime.getRuntime().exec("tasklist");
BufferedReader reader =
new BufferedReader(new InputStreamReader(process.getInputStream()));
while ((reader.readLine()) != null) {}
process.waitFor();
In my case, the waitFor() was not releasing because I was executing a statement with no return ("ip adddr flush eth0"). An easy way to fix this is to simply ensure you always return something in your statement. For me, that meant executing the following: "ip adddr flush eth0 && echo done". You can read the buffer all day, but if there is nothing ever returned, your thread will never release its wait.
Hope that helps someone!
There are several possibilities:
You haven't consumed all the output on the process's stdout.
You haven't consumed all the output on the process's stderr.
The process is waiting for input from you and you haven't provided it, or you haven't closed the process's stdin.
The process is spinning in a hard loop.
As others have mentioned you have to consume stderr and stdout.
Compared to the other answers, since Java 1.7 it is even more easy. You do not have to create threads yourself anymore to read stderr and stdout.
Just use the ProcessBuilder and use the methods redirectOutput in combination with either redirectError or redirectErrorStream.
String directory = "/working/dir";
File out = new File(...); // File to write stdout to
File err = new File(...); // File to write stderr to
ProcessBuilder builder = new ProcessBuilder();
builder.directory(new File(directory));
builder.command(command);
builder.redirectOutput(out); // Redirect stdout to file
if(out == err) {
builder.redirectErrorStream(true); // Combine stderr into stdout
} else {
builder.redirectError(err); // Redirect stderr to file
}
Process process = builder.start();
For the same reason you can also use inheritIO() to map Java console with external app console like:
ProcessBuilder pb = new ProcessBuilder(appPath, arguments);
pb.directory(new File(appFile.getParent()));
pb.inheritIO();
Process process = pb.start();
int success = process.waitFor();
You should try consume output and error in the same while
private void runCMD(String CMD) throws IOException, InterruptedException {
System.out.println("Standard output: " + CMD);
Process process = Runtime.getRuntime().exec(CMD);
// Get input streams
BufferedReader stdInput = new BufferedReader(new InputStreamReader(process.getInputStream()));
BufferedReader stdError = new BufferedReader(new InputStreamReader(process.getErrorStream()));
String line = "";
String newLineCharacter = System.getProperty("line.separator");
boolean isOutReady = false;
boolean isErrorReady = false;
boolean isProcessAlive = false;
boolean isErrorOut = true;
boolean isErrorError = true;
System.out.println("Read command ");
while (process.isAlive()) {
//Read the stdOut
do {
isOutReady = stdInput.ready();
//System.out.println("OUT READY " + isOutReady);
isErrorOut = true;
isErrorError = true;
if (isOutReady) {
line = stdInput.readLine();
isErrorOut = false;
System.out.println("=====================================================================================" + line + newLineCharacter);
}
isErrorReady = stdError.ready();
//System.out.println("ERROR READY " + isErrorReady);
if (isErrorReady) {
line = stdError.readLine();
isErrorError = false;
System.out.println("ERROR::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::" + line + newLineCharacter);
}
isProcessAlive = process.isAlive();
//System.out.println("Process Alive " + isProcessAlive);
if (!isProcessAlive) {
System.out.println(":::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: Process DIE " + line + newLineCharacter);
line = null;
isErrorError = false;
process.waitFor(1000, TimeUnit.MILLISECONDS);
}
} while (line != null);
//Nothing else to read, lets pause for a bit before trying again
System.out.println("PROCESS WAIT FOR");
process.waitFor(100, TimeUnit.MILLISECONDS);
}
System.out.println("Command finished");
}
I think I observed a similar problem: some processes started, seemed to run successfully but never completed. The function waitFor() was waiting forever except if I killed the process in Task Manager.
However, everything worked well in cases the length of the command line was 127 characters or shorter. If long file names are inevitable you may want to use environmental variables, which may allow you keeping the command line string short. You can generate a batch file (using FileWriter) in which you set your environmental variables before calling the program you actually want to run.
The content of such a batch could look like:
set INPUTFILE="C:\Directory 0\Subdirectory 1\AnyFileName"
set OUTPUTFILE="C:\Directory 2\Subdirectory 3\AnotherFileName"
set MYPROG="C:\Directory 4\Subdirectory 5\ExecutableFileName.exe"
%MYPROG% %INPUTFILE% %OUTPUTFILE%
Last step is running this batch file using Runtime.
Here is a method that works for me.
NOTE: There is some code within this method that may not apply to you, so try and ignore it. For example "logStandardOut(...), git-bash, etc".
private String exeShellCommand(String doCommand, String inDir, boolean ignoreErrors) {
logStandardOut("> %s", doCommand);
ProcessBuilder builder = new ProcessBuilder();
StringBuilder stdOut = new StringBuilder();
StringBuilder stdErr = new StringBuilder();
boolean isWindows = System.getProperty("os.name").toLowerCase().startsWith("windows");
if (isWindows) {
String gitBashPathForWindows = "C:\\Program Files\\Git\\bin\\bash";
builder.command(gitBashPathForWindows, "-c", doCommand);
} else {
builder.command("bash", "-c", doCommand);
}
//Do we need to change dirs?
if (inDir != null) {
builder.directory(new File(inDir));
}
//Execute it
Process process = null;
BufferedReader brStdOut;
BufferedReader brStdErr;
try {
//Start the command line process
process = builder.start();
//This hangs on a large file
// https://stackoverflow.com/questions/5483830/process-waitfor-never-returns
//exitCode = process.waitFor();
//This will have both StdIn and StdErr
brStdOut = new BufferedReader(new InputStreamReader(process.getInputStream()));
brStdErr = new BufferedReader(new InputStreamReader(process.getErrorStream()));
//Get the process output
String line = null;
String newLineCharacter = System.getProperty("line.separator");
while (process.isAlive()) {
//Read the stdOut
while ((line = brStdOut.readLine()) != null) {
stdOut.append(line + newLineCharacter);
}
//Read the stdErr
while ((line = brStdErr.readLine()) != null) {
stdErr.append(line + newLineCharacter);
}
//Nothing else to read, lets pause for a bit before trying again
process.waitFor(100, TimeUnit.MILLISECONDS);
}
//Read anything left, after the process exited
while ((line = brStdOut.readLine()) != null) {
stdOut.append(line + newLineCharacter);
}
//Read anything left, after the process exited
while ((line = brStdErr.readLine()) != null) {
stdErr.append(line + newLineCharacter);
}
//cleanup
if (brStdOut != null) {
brStdOut.close();
}
if (brStdErr != null) {
brStdOut.close();
}
//Log non-zero exit values
if (!ignoreErrors && process.exitValue() != 0) {
String exMsg = String.format("%s%nprocess.exitValue=%s", stdErr, process.exitValue());
throw new ExecuteCommandException(exMsg);
}
} catch (ExecuteCommandException e) {
throw e;
} catch (Exception e) {
throw new ExecuteCommandException(stdErr.toString(), e);
} finally {
//Log the results
logStandardOut(stdOut.toString());
logStandardError(stdErr.toString());
}
return stdOut.toString();
}
Asynchronous reading of stream combined with avoiding Wait with a timeout will solve the problem.
You can find a page explaining this here http://simplebasics.net/.net/process-waitforexit-with-a-timeout-will-not-be-able-to-collect-the-output-message/
public static void main(String[] args) throws PyException, IOException, InterruptedException
these should be the exceptions thrown