I need to convert a lot of wave files simultaneously. About 300 files in parallel. And new files come constantly. I use ffmpeg process call from my Java 1.8 app, which is running on CentOS. I know that I have to read error and input streams for making created process from Java possible to exit.
My code after several expirements:
private void ffmpegconverter(String fileIn, String fileOut){
String[] comand = new String[]{"ffmpeg", "-v", "-8", "-i", fileIn, "-acodec", "pcm_s16le", fileOut};
Process process = null;
BufferedReader reader = null;
try {
ProcessBuilder pb = new ProcessBuilder(comand);
pb.redirectErrorStream(true);
process = pb.start();
//Reading from error and standard output console buffer of process. Or it could halts because of nobody
//reads its buffer
reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String s;
//noinspection StatementWithEmptyBody
while ((s = reader.readLine()) != null) {
log.info(Thread.currentThread().getName() + " with fileIn " + fileIn + " and fileOut " + fileOut + " writes " + s);
//Ignored as we just need to empty the output buffer from process
}
log.info(Thread.currentThread().getName() + " ffmpeg process will be waited for");
if (process.waitFor( 10, TimeUnit.SECONDS )) {
log.info(Thread.currentThread().getName() + " ffmpeg process exited normally");
} else {
log.info(Thread.currentThread().getName() + " ffmpeg process timed out and will be killed");
}
} catch (IOException | InterruptedException e) {
log.error(Thread.currentThread().getName() + "Error during ffmpeg process executing", e);
} finally {
if (process != null) {
if (reader != null) {
try {
reader.close();
} catch (IOException e) {
log.error("Error during closing the process streams reader", e);
}
}
try {
process.getOutputStream().close();
} catch (IOException e) {
log.error("Error during closing the process output stream", e);
}
process.destroyForcibly();
log.info(Thread.currentThread().getName() + " ffmpeg process " + process + " must be dead now");
}
}
}
If I run separate test with this code it goes normally. But in my app I have hundreds of RUNNING deamon threads "process reaper" which are waiting for ffmpeg process finish. In my real app ffpmeg is started from timer thread. Also I have another activity in separate threads, but I don't think that this is the problem. Max CPU consume is about 10%.
Here is that I usual see in thread dump:
"process reaper" #454 daemon prio=10 os_prio=0 tid=0x00007f641c007000 nid=0x5247 runnable [0x00007f63ec063000]
java.lang.Thread.State: RUNNABLE
at java.lang.UNIXProcess.waitForProcessExit(Native Method)
at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:289)
at java.lang.UNIXProcess$$Lambda$32/2113551491.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
What am I doing wrong?
UPD:
My app accepts a lot of connects with voice traffic. So I have about 300-500 another "good" threads in every moment. Could it be the reason? Deamon threads have low priority. But I don't beleive that they really can't do their jobs in one hour. Ususally it takes some tens of millis.
UPD2:
My synthetic test that runs fine. I tried with new threads option and without it just with straigt calling of run method.
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.InputStreamReader;
public class FFmpegConvert {
public static void main(String[] args) throws Exception {
FFmpegConvert f = new FFmpegConvert();
f.processDir(args[0], args[1], args.length > 2);
}
private void processDir(String dirPath, String dirOutPath, boolean isNewThread) {
File dir = new File(dirPath);
File dirOut = new File(dirOutPath);
if(!dirOut.exists()){
dirOut.mkdir();
}
for (int i = 0; i < 1000; i++) {
for (File f : dir.listFiles()) {
try {
System.out.println(f.getName());
FFmpegRunner fFmpegRunner = new FFmpegRunner(f.getAbsolutePath(), dirOut.getAbsolutePath() + "/" + System.currentTimeMillis() + f.getName());
if (isNewThread) {
new Thread(fFmpegRunner).start();
} else {
fFmpegRunner.run();
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
class FFmpegRunner implements Runnable {
private String fileIn;
private String fileOut;
FFmpegRunner(String fileIn, String fileOut) {
this.fileIn = fileIn;
this.fileOut = fileOut;
}
#Override
public void run() {
try {
ffmpegconverter(fileIn, fileOut);
} catch (Exception e) {
e.printStackTrace();
}
}
private void ffmpegconverter(String fileIn, String fileOut) throws Exception{
String[] comand = new String[]{"ffmpeg", "-i", fileIn, "-acodec", "pcm_s16le", fileOut};
Process process = null;
try {
ProcessBuilder pb = new ProcessBuilder(comand);
pb.redirectErrorStream(true);
process = pb.start();
//Reading from error and standard output console buffer of process. Or it could halts because of nobody
//reads its buffer
BufferedReader reader =
new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
//noinspection StatementWithEmptyBody
while ((line = reader.readLine()) != null) {
System.out.println(line);
//Ignored as we just need to empty the output buffer from process
}
process.waitFor();
} catch (IOException | InterruptedException e) {
throw e;
} finally {
if (process != null)
process.destroy();
}
}
}
}
UPD3:
Sorry, I forgot to notice that I see the work of all these process - they created new converted files but anyway don't exit.
Your application is I/O bound, not CPU bound. If all your files are in the same HDD, then opening simultaneously 300 files will definitely degrade the performance. (that is a likely reason on why you have hundreds of processes waiting).
If I understood correctly, you mentioned that processing 1 file takes some tens of millis? (and this is doing sequential reads - the fastest that your HDD will read a file)
in this case, processing 300 files sequentially should take no more than 30 seconds.
100 millis - process 1 file
1 second - process 10 files
30 second - process 300 files
EDIT
I did 2 simple changes to your sample code (I removed the first loop, then changed the codec) finally I put one song in "ogg" format in "/tmp/origin" directory. now the program works well).
see code below:
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.InputStreamReader;
public class FFMpegConvert {
public static void main(String[] args) throws Exception {
FFMpegConvert f = new FFMpegConvert();
f.processDir("/tmp/origin", "/tmp/destination", false);
}
private void processDir(String dirPath, String dirOutPath, boolean isNewThread) {
File dir = new File(dirPath);
File dirOut = new File(dirOutPath);
if (!dirOut.exists()) {
dirOut.mkdir();
}
for (File f : dir.listFiles()) {
try {
System.out.println(f.getName());
FFmpegRunner fFmpegRunner = new FFmpegRunner(f.getAbsolutePath(), dirOut.getAbsolutePath() + "/" + System.currentTimeMillis() + f.getName());
if (isNewThread) {
new Thread(fFmpegRunner).start();
} else {
fFmpegRunner.run();
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
class FFmpegRunner implements Runnable {
private String fileIn;
private String fileOut;
FFmpegRunner(String fileIn, String fileOut) {
this.fileIn = fileIn;
this.fileOut = fileOut;
}
#Override
public void run() {
try {
ffmpegconverter(fileIn, fileOut);
} catch (Exception e) {
e.printStackTrace();
}
}
private void ffmpegconverter(String fileIn, String fileOut) throws Exception {
String[] comand = new String[]{"ffmpeg", "-i", fileIn, "-acodec", "copy", fileOut};
Process process = null;
try {
ProcessBuilder pb = new ProcessBuilder(comand);
pb.redirectErrorStream(true);
process = pb.start();
//Reading from error and standard output console buffer of process. Or it could halts because of nobody
//reads its buffer
BufferedReader reader =
new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
//noinspection StatementWithEmptyBody
while ((line = reader.readLine()) != null) {
System.out.println(line);
//Ignored as we just need to empty the output buffer from process
}
process.waitFor();
} catch (IOException | InterruptedException e) {
throw e;
} finally {
if (process != null)
process.destroy();
}
}
}
}
Got it!
In some cases ffmpeg wants to ask me should it override already existed file. In my code I close outputstream of this child process. But as it appears this only closes outputstream for Java but not for the process.
So my solution is to make ffmpeg silent at all: no output from this process with "-v -8", no asking question with default "Yes" "-y".
I use StreamGobbler to read output from external program. Before I used to read and parse big(20-40 strings) output from stderr of ffmpeg. Now I want to read and parse only one string from stdout(one string is all output of program and this program ends very quickly) of identify(ImageMagick). And sometimes it works, but sometimes I have error "Stream closed". I think that StreamGobbler has no time to work with stream(process ends before streamGobler do some work).
Below you can see class ExecThread and example of it's usage.
Sorry for my English...
String command = ffmpegExe.getAbsolutePath()+ " -i \""+ fileName +"\"";
ExecThread thread = new ExecThread(command);
thread.setPriority(ExecThread.MIN_PRIORITY);
thread.start();
thread.waitFor(DEFAULT_WAIT);
//reading ffmpeg stderr
BufferedReader ffmpegErr = thread.getErrBufferedReader();
String line;
try
{
while((line = ffmpegErr.readLine()) !=null)
//some code
private static class ExecThread extends Thread
{
public ExecThread(String command)
{
this.command = command;
}
#Override
public void run()
{
try
{
ExecCommand(command);
exitVal = process.waitFor();
}
catch (Exception e )
{
logger.error("Error while executing command: " + command + e.getMessage(),e);
System.err.println("Error");
System.exit(1);
}
}
public void waitFor(long millis)
{
try
{
this.join(millis);
process.destroy();
} catch (Exception e)
{
logger.error("Error while interupting command: " + command + e.getMessage(), e);
}
}
private void ExecCommand(String command) throws IOException
{
//run command
Runtime rt = Runtime.getRuntime();
process = rt.exec(new String[]{shellName,shellOption, command}, null, null);
//get stderr buffered reader
StreamGobbler errGobbler = new StreamGobbler(process.getErrorStream());
StreamGobbler outGobbler = new StreamGobbler(process.getInputStream());
ErrBufferedReader = new BufferedReader(new InputStreamReader(errGobbler));
OutBufferedReader = new BufferedReader(new InputStreamReader(outGobbler));
}
private BufferedReader ErrBufferedReader;
public BufferedReader getOutBufferedReader()
{
return OutBufferedReader;
}
private BufferedReader OutBufferedReader;
private Process process;
private String command;
private int exitVal=-1;
public BufferedReader getErrBufferedReader()
{
return ErrBufferedReader;
}
public Process getProcess()
{
return process;
}
public String getCommand()
{
return command;
}
public int getExitVal()
{
return exitVal;
}
}
I'd recommend two changes:
Use the Java Process instead of writing a Thread.
Close your input, output, and error streams and destroy the Process. You'll have file handle leaks if you don't.
This is not good decision but it works. Before it was String command = identifyExe.getAbsolutePath() + " \"" + file.getAbsolutePath() + "\"";
now it String command = "sleep 1 ; " + identifyExe.getAbsolutePath() + " \"" + file.getAbsolutePath() + "\"";
Sleep is very usefull function :-)
But, if you know better decision please tell...
I want to create diff of two files. I tried searching for code in Java that does it, but didnt find any simple code/ utility code for this. Hence, I thought if I can somehow run linux diff/sdiff command from my java code and make it return a file that stores the diff then it would be great.
Suppose there are two files fileA and fileB. I should be able to store their diff in a file called fileDiff through my java code. Then fetching data from fileDiff would be no big deal.
You can use java.lang.Runtime.exec to run simple code. This gives you back a Process and you can read its standard output directly without having to temporarily store the output on disk.
For example, here's a complete program that will showcase how to do it:
import java.io.BufferedReader;
import java.io.InputStreamReader;
public class testprog {
public static void main(String args[]) {
String s;
Process p;
try {
p = Runtime.getRuntime().exec("ls -aF");
BufferedReader br = new BufferedReader(
new InputStreamReader(p.getInputStream()));
while ((s = br.readLine()) != null)
System.out.println("line: " + s);
p.waitFor();
System.out.println ("exit: " + p.exitValue());
p.destroy();
} catch (Exception e) {}
}
}
When compiled and run, it outputs:
line: ./
line: ../
line: .classpath*
line: .project*
line: bin/
line: src/
exit: 0
as expected.
You can also get the error stream for the process standard error, and output stream for the process standard input, confusingly enough. In this context, the input and output are reversed since it's input from the process to this one (i.e., the standard output of the process).
If you want to merge the process standard output and error from Java (as opposed to using 2>&1 in the actual command), you should look into ProcessBuilder.
You can also write a shell script file and invoke that file from the java code. as shown below
{
Process proc = Runtime.getRuntime().exec("./your_script.sh");
proc.waitFor();
}
Write the linux commands in the script file, once the execution is over you can read the diff file in Java.
The advantage with this approach is you can change the commands with out changing java code.
You need not store the diff in a 3rd file and then read from in. Instead you make use of the Runtime.exec
Process p = Runtime.getRuntime().exec("diff fileA fileB");
BufferedReader stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));
while ((s = stdInput.readLine()) != null) {
System.out.println(s);
}
try to use unix4j. it s about a library in java to run linux command. for instance if you got a command like:
cat test.txt | grep "Tuesday" | sed "s/kilogram/kg/g" | sort
in this program will become:
Unix4j.cat("test.txt").grep("Tuesday").sed("s/kilogram/kg/g").sort();
You can call run-time commands from java for both Windows and Linux.
import java.io.*;
public class Test{
public static void main(String[] args)
{
try
{
Process process = Runtime.getRuntime().exec("pwd"); // for Linux
//Process process = Runtime.getRuntime().exec("cmd /c dir"); //for Windows
process.waitFor();
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
while ((line=reader.readLine())!=null)
{
System.out.println(line);
}
}
catch(Exception e)
{
System.out.println(e);
}
finally
{
process.destroy();
}
}
}
Hope it Helps.. :)
Runtime run = Runtime.getRuntime();
//The best possible I found is to construct a command which you want to execute
//as a string and use that in exec. If the batch file takes command line arguments
//the command can be constructed a array of strings and pass the array as input to
//the exec method. The command can also be passed externally as input to the method.
Process p = null;
String cmd = "ls";
try {
p = run.exec(cmd);
p.getErrorStream();
p.waitFor();
}
catch (IOException e) {
e.printStackTrace();
System.out.println("ERROR.RUNNING.CMD");
}finally{
p.destroy();
}
The suggested solutions could be optimized using commons.io, handling the error stream, and using Exceptions. I would suggest to wrap like this for use in Java 8 or later:
public static List<String> execute(final String command) throws ExecutionFailedException, InterruptedException, IOException {
try {
return execute(command, 0, null, false);
} catch (ExecutionTimeoutException e) { return null; } /* Impossible case! */
}
public static List<String> execute(final String command, final long timeout, final TimeUnit timeUnit) throws ExecutionFailedException, ExecutionTimeoutException, InterruptedException, IOException {
return execute(command, 0, null, true);
}
public static List<String> execute(final String command, final long timeout, final TimeUnit timeUnit, boolean destroyOnTimeout) throws ExecutionFailedException, ExecutionTimeoutException, InterruptedException, IOException {
Process process = new ProcessBuilder().command("bash", "-c", command).start();
if(timeUnit != null) {
if(process.waitFor(timeout, timeUnit)) {
if(process.exitValue() == 0) {
return IOUtils.readLines(process.getInputStream(), StandardCharsets.UTF_8);
} else {
throw new ExecutionFailedException("Execution failed: " + command, process.exitValue(), IOUtils.readLines(process.getInputStream(), StandardCharsets.UTF_8));
}
} else {
if(destroyOnTimeout) process.destroy();
throw new ExecutionTimeoutException("Execution timed out: " + command);
}
} else {
if(process.waitFor() == 0) {
return IOUtils.readLines(process.getInputStream(), StandardCharsets.UTF_8);
} else {
throw new ExecutionFailedException("Execution failed: " + command, process.exitValue(), IOUtils.readLines(process.getInputStream(), StandardCharsets.UTF_8));
}
}
}
public static class ExecutionFailedException extends Exception {
private static final long serialVersionUID = 1951044996696304510L;
private final int exitCode;
private final List<String> errorOutput;
public ExecutionFailedException(final String message, final int exitCode, final List<String> errorOutput) {
super(message);
this.exitCode = exitCode;
this.errorOutput = errorOutput;
}
public int getExitCode() {
return this.exitCode;
}
public List<String> getErrorOutput() {
return this.errorOutput;
}
}
public static class ExecutionTimeoutException extends Exception {
private static final long serialVersionUID = 4428595769718054862L;
public ExecutionTimeoutException(final String message) {
super(message);
}
}
if the opening in windows
try {
//chm file address
String chmFile = System.getProperty("user.dir") + "/chm/sample.chm";
Desktop.getDesktop().open(new File(chmFile));
} catch (IOException ex) {
Logger.getLogger(Frame.class.getName()).log(Level.SEVERE, null, ex);
{
JOptionPane.showMessageDialog(null, "Terjadi Kesalahan", "Error", JOptionPane.WARNING_MESSAGE);
}
}
ProcessBuilder processBuilder = new ProcessBuilder();
// -- Linux --
// Run a shell command
processBuilder.command("bash", "-c", "ls /home/kk/");
// Run a shell script
//processBuilder.command("path/to/hello.sh");
// -- Windows --
// Run a command
//processBuilder.command("cmd.exe", "/c", "dir C:\\Users\\kk");
// Run a bat file
//processBuilder.command("C:\\Users\\kk\\hello.bat");
try {
Process process = processBuilder.start();
StringBuilder output = new StringBuilder();
BufferedReader reader = new BufferedReader(
new InputStreamReader(process.getInputStream()));
String line;
while ((line = reader.readLine()) != null) {
output.append(line + "\n");
}
int exitVal = process.waitFor();
if (exitVal == 0) {
System.out.println("Success!");
System.out.println(output);
System.exit(0);
} else {
//abnormal...
}
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
I have a method I am using to execute a command on the local host. I'd like to add a timeout parameter to the method so that if the command being called doesn't finish in a reasonable amount of time the method will return with an error code. Here's what it looks like so far, without the ability to timeout:
public static int executeCommandLine(final String commandLine,
final boolean printOutput,
final boolean printError)
throws IOException, InterruptedException
{
Runtime runtime = Runtime.getRuntime();
Process process = runtime.exec(commandLine);
if (printOutput)
{
BufferedReader outputReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
System.out.println("Output: " + outputReader.readLine());
}
if (printError)
{
BufferedReader errorReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
System.out.println("Error: " + errorReader.readLine());
}
return process.waitFor();
}
Can anyone suggest a good way for me to implement a timeout parameter?
If you're using Java 8 or later you could simply use the new waitFor with timeout:
Process p = ...
if(!p.waitFor(1, TimeUnit.MINUTES)) {
//timeout - kill the process.
p.destroy(); // consider using destroyForcibly instead
}
public static int executeCommandLine(final String commandLine,
final boolean printOutput,
final boolean printError,
final long timeout)
throws IOException, InterruptedException, TimeoutException {
Runtime runtime = Runtime.getRuntime();
Process process = runtime.exec(commandLine);
/* Set up process I/O. */
...
Worker worker = new Worker(process);
worker.start();
try {
worker.join(timeout);
if (worker.exit != null)
return worker.exit;
else
throw new TimeoutException();
} catch(InterruptedException ex) {
worker.interrupt();
Thread.currentThread().interrupt();
throw ex;
} finally {
process.destroyForcibly();
}
}
private static class Worker extends Thread {
private final Process process;
private Integer exit;
private Worker(Process process) {
this.process = process;
}
public void run() {
try {
exit = process.waitFor();
} catch (InterruptedException ignore) {
return;
}
}
}
Following the answer by erickson, I created a more generic way to do the same thing.
public class ProcessWithTimeout extends Thread
{
private Process m_process;
private int m_exitCode = Integer.MIN_VALUE;
public ProcessWithTimeout(Process p_process)
{
m_process = p_process;
}
public int waitForProcess(int p_timeoutMilliseconds)
{
this.start();
try
{
this.join(p_timeoutMilliseconds);
}
catch (InterruptedException e)
{
this.interrupt();
}
return m_exitCode;
}
#Override
public void run()
{
try
{
m_exitCode = m_process.waitFor();
}
catch (InterruptedException ignore)
{
// Do nothing
}
catch (Exception ex)
{
// Unexpected exception
}
}
}
Now, all you have to do is as follows:
Process process = Runtime.getRuntime().exec("<your command goes here>");
ProcessWithTimeout processWithTimeout = new ProcessWithTimeout(process);
int exitCode = processWithTimeout.waitForProcess(5000);
if (exitCode == Integer.MIN_VALUE)
{
// Timeout
}
else
{
// No timeout !
}
I implemented this using the three approaches suggested which came with detailed code example (I am a novice with thread programming and these example codes were invaluable -- I would still be scratching my head as to how to do this if it was just explained in English without code).
I implemented the utility class I'm using for this with the three methods for executing a command with a timeout like so:
package com.abc.network.lifecycle.util;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
/**
* Utility class for performing process related functions such as command line processing.
*/
public class ProcessUtility
{
static Log log = LogFactory.getLog(ProcessUtility.class);
/**
* Thread class to be used as a worker
*/
private static class Worker
extends Thread
{
private final Process process;
private Integer exitValue;
Worker(final Process process)
{
this.process = process;
}
public Integer getExitValue()
{
return exitValue;
}
#Override
public void run()
{
try
{
exitValue = process.waitFor();
}
catch (InterruptedException ignore)
{
return;
}
}
}
/**
* Executes a command.
*
* #param command
* #param printOutput
* #param printError
* #param timeOut
* #return
* #throws java.io.IOException
* #throws java.lang.InterruptedException
*/
public static int executeCommandWithExecutors(final String command,
final boolean printOutput,
final boolean printError,
final long timeOut)
{
// validate the system and command line and get a system-appropriate command line
String massagedCommand = validateSystemAndMassageCommand(command);
try
{
// create the process which will run the command
Runtime runtime = Runtime.getRuntime();
final Process process = runtime.exec(massagedCommand);
// consume and display the error and output streams
StreamGobbler outputGobbler = new StreamGobbler(process.getInputStream(), "OUTPUT", printOutput);
StreamGobbler errorGobbler = new StreamGobbler(process.getErrorStream(), "ERROR", printError);
outputGobbler.start();
errorGobbler.start();
// create a Callable for the command's Process which can be called by an Executor
Callable<Integer> call = new Callable<Integer>()
{
public Integer call()
throws Exception
{
process.waitFor();
return process.exitValue();
}
};
// submit the command's call and get the result from a
Future<Integer> futureResultOfCall = Executors.newSingleThreadExecutor().submit(call);
try
{
int exitValue = futureResultOfCall.get(timeOut, TimeUnit.MILLISECONDS);
return exitValue;
}
catch (TimeoutException ex)
{
String errorMessage = "The command [" + command + "] timed out.";
log.error(errorMessage, ex);
throw new RuntimeException(errorMessage, ex);
}
catch (ExecutionException ex)
{
String errorMessage = "The command [" + command + "] did not complete due to an execution error.";
log.error(errorMessage, ex);
throw new RuntimeException(errorMessage, ex);
}
}
catch (InterruptedException ex)
{
String errorMessage = "The command [" + command + "] did not complete due to an unexpected interruption.";
log.error(errorMessage, ex);
throw new RuntimeException(errorMessage, ex);
}
catch (IOException ex)
{
String errorMessage = "The command [" + command + "] did not complete due to an IO error.";
log.error(errorMessage, ex);
throw new RuntimeException(errorMessage, ex);
}
}
/**
* Executes a command.
*
* #param command
* #param printOutput
* #param printError
* #param timeOut
* #return
* #throws java.io.IOException
* #throws java.lang.InterruptedException
*/
public static int executeCommandWithSleep(final String command,
final boolean printOutput,
final boolean printError,
final long timeOut)
{
// validate the system and command line and get a system-appropriate command line
String massagedCommand = validateSystemAndMassageCommand(command);
try
{
// create the process which will run the command
Runtime runtime = Runtime.getRuntime();
Process process = runtime.exec(massagedCommand);
// consume and display the error and output streams
StreamGobbler outputGobbler = new StreamGobbler(process.getInputStream(), "OUTPUT", printOutput);
StreamGobbler errorGobbler = new StreamGobbler(process.getErrorStream(), "ERROR", printError);
outputGobbler.start();
errorGobbler.start();
// run a thread which will set a flag once it has slept for the timeout period
final boolean[] flags = { true };
new Thread()
{
#Override
public void run()
{
try
{
Thread.sleep(timeOut);
}
catch (InterruptedException ex)
{
String errorMessage = "Timeout loop thread unexpectedly interrupted.";
log.error(errorMessage, ex);
throw new RuntimeException(errorMessage, ex);
}
flags[0] = false;
}
}.start();
// execute the command and wait
int returnValue = -1;
while (flags[0] && (returnValue < 0))
{
returnValue = process.waitFor();
}
// if the command timed out then log it
if (returnValue < 0)
{
log.warn("The command [" + command + "] did not complete before the timeout period expired (timeout: " +
timeOut + " ms)");
}
return returnValue;
}
catch (InterruptedException ex)
{
String errorMessage = "The command [" + command + "] did not complete due to an unexpected interruption.";
log.error(errorMessage, ex);
throw new RuntimeException(errorMessage, ex);
}
catch (IOException ex)
{
String errorMessage = "The command [" + command + "] did not complete due to an IO error.";
log.error(errorMessage, ex);
throw new RuntimeException(errorMessage, ex);
}
}
/**
* Executes a command.
*
* #param command
* #param printOutput
* #param printError
* #param timeOut
* #return
* #throws java.io.IOException
* #throws java.lang.InterruptedException
*/
public static int executeCommandWithWorker(final String command,
final boolean printOutput,
final boolean printError,
final long timeOut)
{
// validate the system and command line and get a system-appropriate command line
String massagedCommand = validateSystemAndMassageCommand(command);
try
{
// create the process which will run the command
Runtime runtime = Runtime.getRuntime();
Process process = runtime.exec(massagedCommand);
// consume and display the error and output streams
StreamGobbler outputGobbler = new StreamGobbler(process.getInputStream(), "OUTPUT", printOutput);
StreamGobbler errorGobbler = new StreamGobbler(process.getErrorStream(), "ERROR", printError);
outputGobbler.start();
errorGobbler.start();
// create and start a Worker thread which this thread will join for the timeout period
Worker worker = new Worker(process);
worker.start();
try
{
worker.join(timeOut);
Integer exitValue = worker.getExitValue();
if (exitValue != null)
{
// the worker thread completed within the timeout period
return exitValue;
}
// if we get this far then we never got an exit value from the worker thread as a result of a timeout
String errorMessage = "The command [" + command + "] timed out.";
log.error(errorMessage);
throw new RuntimeException(errorMessage);
}
catch (InterruptedException ex)
{
worker.interrupt();
Thread.currentThread().interrupt();
throw ex;
}
}
catch (InterruptedException ex)
{
String errorMessage = "The command [" + command + "] did not complete due to an unexpected interruption.";
log.error(errorMessage, ex);
throw new RuntimeException(errorMessage, ex);
}
catch (IOException ex)
{
String errorMessage = "The command [" + command + "] did not complete due to an IO error.";
log.error(errorMessage, ex);
throw new RuntimeException(errorMessage, ex);
}
}
/**
* Validates that the system is running a supported OS and returns a system-appropriate command line.
*
* #param originalCommand
* #return
*/
private static String validateSystemAndMassageCommand(final String originalCommand)
{
// make sure that we have a command
if (originalCommand.isEmpty() || (originalCommand.length() < 1))
{
String errorMessage = "Missing or empty command line parameter.";
log.error(errorMessage);
throw new RuntimeException(errorMessage);
}
// make sure that we are running on a supported system, and if so set the command line appropriately
String massagedCommand;
String osName = System.getProperty("os.name");
if (osName.equals("Windows XP"))
{
massagedCommand = "cmd.exe /C " + originalCommand;
}
else if (osName.equals("Solaris") || osName.equals("SunOS") || osName.equals("Linux"))
{
massagedCommand = originalCommand;
}
else
{
String errorMessage = "Unable to run on this system which is not Solaris, Linux, or Windows XP (actual OS type: \'" +
osName + "\').";
log.error(errorMessage);
throw new RuntimeException(errorMessage);
}
return massagedCommand;
}
}
I created a class to consume and display the output and error streams from a command (taken from http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html?page=4):
package com.abc.network.lifecycle.util;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
/**
* Utility thread class which consumes and displays stream input.
*
* Original code taken from http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html?page=4
*/
class StreamGobbler
extends Thread
{
static private Log log = LogFactory.getLog(StreamGobbler.class);
private InputStream inputStream;
private String streamType;
private boolean displayStreamOutput;
/**
* Constructor.
*
* #param inputStream the InputStream to be consumed
* #param streamType the stream type (should be OUTPUT or ERROR)
* #param displayStreamOutput whether or not to display the output of the stream being consumed
*/
StreamGobbler(final InputStream inputStream,
final String streamType,
final boolean displayStreamOutput)
{
this.inputStream = inputStream;
this.streamType = streamType;
this.displayStreamOutput = displayStreamOutput;
}
/**
* Consumes the output from the input stream and displays the lines consumed if configured to do so.
*/
#Override
public void run()
{
try
{
InputStreamReader inputStreamReader = new InputStreamReader(inputStream);
BufferedReader bufferedReader = new BufferedReader(inputStreamReader);
String line = null;
while ((line = bufferedReader.readLine()) != null)
{
if (displayStreamOutput)
{
System.out.println(streamType + ">" + line);
}
}
}
catch (IOException ex)
{
log.error("Failed to successfully consume and display the input stream of type " + streamType + ".", ex);
ex.printStackTrace();
}
}
}
I created a test command which takes roughly 10 seconds to complete:
#!/bin/bash
sleep 10
echo 'TEST COMMAND RAN OK'
Then I created a test program to test the three different methods, calling each with a timeout value of 5 seconds (command should fail) and with a timeout value of 15 seconds (command should succeed):
package com.abc.network.lifecycle.util;
public class ProcessUtilityTester
{
/**
* #param args
*/
public static void main(final String[] args)
{
try
{
String command = args[0];
int exitValue = -1;
System.out.println("\n\n5000ms timeout With Executors:");
try
{
exitValue = -1;
exitValue = ProcessUtility.executeCommandWithExecutors(command, true, true, 5000);
}
catch (Exception ex)
{
ex.printStackTrace();
}
finally
{
System.out.println("\nExit value:" + exitValue);
}
System.out.println("\n\n5000ms timeout With Sleep:");
try
{
exitValue = -1;
exitValue = ProcessUtility.executeCommandWithSleep(command, true, true, 5000);
}
catch (Exception ex)
{
ex.printStackTrace();
}
finally
{
System.out.println("\nExit value:" + exitValue);
}
System.out.println("\n\n5000ms timeout With Worker:");
try
{
exitValue = -1;
exitValue = ProcessUtility.executeCommandWithWorker(command, true, true, 5000);
}
catch (Exception ex)
{
ex.printStackTrace();
}
finally
{
System.out.println("\nExit value:" + exitValue);
}
System.out.println("\n\n15000ms timeout With Executors:");
try
{
exitValue = -1;
exitValue = ProcessUtility.executeCommandWithExecutors(command, true, true, 15000);
}
catch (Exception ex)
{
ex.printStackTrace();
}
finally
{
System.out.println("\nExit value:" + exitValue);
}
System.out.println("\n\n15000ms timeout With Sleep:");
try
{
exitValue = -1;
exitValue = ProcessUtility.executeCommandWithSleep(command, true, true, 15000);
}
catch (Exception ex)
{
ex.printStackTrace();
}
finally
{
System.out.println("\nExit value:" + exitValue);
}
System.out.println("\n\n15000ms timeout With Worker:");
try
{
exitValue = -1;
exitValue = ProcessUtility.executeCommandWithWorker(command, true, true, 15000);
}
catch (Exception ex)
{
ex.printStackTrace();
}
finally
{
System.out.println("\nExit value:" + exitValue);
}
}
catch (Exception ex)
{
ex.printStackTrace();
}
finally
{
System.exit(0);
}
}
}
Here's what I see when I run the test program:
5000ms timeout With Executors:
May 1, 2009 1:55:19 AM com.abc.network.lifecycle.util.ProcessUtility executeCommandWithExecutors
SEVERE: The command [/tmp/testcmd.sh] timed out.
java.util.concurrent.TimeoutException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:228)
at java.util.concurrent.FutureTask.get(FutureTask.java:91)
at com.abc.network.lifecycle.util.ProcessUtility.executeCommandWithExecutors(ProcessUtility.java:179)
at com.abc.network.lifecycle.util.ProcessUtilityTester.main(ProcessUtilityTester.java:19)
java.lang.RuntimeException: The command [/tmp/testcmd.sh] timed out.
at com.abc.network.lifecycle.util.ProcessUtility.executeCommandWithExecutors(ProcessUtility.java:186)
at com.abc.network.lifecycle.util.ProcessUtilityTester.main(ProcessUtilityTester.java:19)
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:228)
at java.util.concurrent.FutureTask.get(FutureTask.java:91)
at com.abc.network.lifecycle.util.ProcessUtility.executeCommandWithExecutors(ProcessUtility.java:179)
... 1 more
Exit value:-1
5000ms timeout With Sleep:
OUTPUT>TEST COMMAND RAN OK
OUTPUT>TEST COMMAND RAN OK
Exit value:0
5000ms timeout With Worker:
May 1, 2009 1:55:34 AM com.abc.network.lifecycle.util.ProcessUtility executeCommandWithWorker
SEVERE: The command [/tmp/testcmd.sh] timed out.
java.lang.RuntimeException: The command [/tmp/testcmd.sh] timed out.
at com.abc.network.lifecycle.util.ProcessUtility.executeCommandWithWorker(ProcessUtility.java:338)
at com.abc.network.lifecycle.util.ProcessUtilityTester.main(ProcessUtilityTester.java:47)
Exit value:-1
15000ms timeout With Executors:
OUTPUT>TEST COMMAND RAN OK
OUTPUT>TEST COMMAND RAN OK
Exit value:0
15000ms timeout With Sleep:
OUTPUT>TEST COMMAND RAN OK
Exit value:0
15000ms timeout With Worker:
OUTPUT>TEST COMMAND RAN OK
Exit value:0
So from what I can tell the approach using a Worker thread class works the best, in that it gives the expected results in both cases. The approach using Executors works as expected as well, with the caveat that it appears to be running the command twice in the 15000ms timout case (i.e. I see the output for the command twice). The approach using the sleep() method does not timeout the command as expected in the 5000ms timeout case, and displays the output twice, but runs the command as expected in the 15000ms timeout case.
For everybody using the executor framework: you are all forgetting to shutdown the executor. So change it to the following:
ExecutorService service = Executors.newSingleThreadExecutor();
try {
Future<Integer> ft = service.submit(call);
try {
int exitVal = ft.get(2000L, TimeUnit.MILLISECONDS);
return exitVal;
} catch (TimeoutException to) {
p.destroy();
throw to;
}
}
finally {
service.shutdown();
}
If you don't your program will keep an active non-daemon thread, ensuring your program will never exit until you call System.exit
For those who can't use the new Java 8 method waitFor(long timeout, TimeUnit unit) (because they are on Android or simply can't upgrade) you can simply rip it from the JDK source code and add it somewhere in your utils file :
public boolean waitFor(long timeout, TimeUnit unit, final Process process)
throws InterruptedException
{
long startTime = System.nanoTime();
long rem = unit.toNanos(timeout);
do {
try {
process.exitValue();
return true;
} catch(IllegalThreadStateException ex) {
if (rem > 0)
Thread.sleep(
Math.min(TimeUnit.NANOSECONDS.toMillis(rem) + 1, 100));
}
rem = unit.toNanos(timeout) - (System.nanoTime() - startTime);
} while (rem > 0);
return false;
}
The only change I've made to the original one from JDK8 source code is the addition of the Process parameter so that we can call the exitValue method from the process.
The exitValue method will directly try to return or throw an IllegalThreadStateException if the process has not yet terminated. In that case, we wait the received timeout and terminate.
The method return a boolean, so if it return false then you know you need to manually kill the process.
This way seems simplier than anything posted above (expect the direct call to waitFor for sure).
A light-weight solution for small apps:
public class Test {
public static void main(String[] args) throws java.io.IOException, InterruptedException {
Process process = new ProcessBuilder().command("sleep", "10").start();
int i=0;
boolean deadYet = false;
do {
Thread.sleep(1000);
try {
process.exitValue();
deadYet = true;
} catch (IllegalThreadStateException e) {
System.out.println("Not done yet...");
if (++i >= 5) throw new RuntimeException("timeout");
}
} while (!deadYet);
}
}
Implement as a delegate and fail the call if it takes above your threshold to complete.
Try using a Timer (or Sleep()), in a separate thread or in your event queue if you have one available.
There are various ways to do this, but I'd consider using an Executor-- it just helps you encapsulate passing the exit value or exception from the thread back to the original caller.
final Process p = ...
Callable<Integer> call = new Callable<Integer>() {
public Integer call() throws Exception {
p.waitFor();
return p.exitValue();
}
};
Future<Integer> ft = Executors.newSingleThreadExecutor().submit(call);
try {
int exitVal = ft.get(2000L, TimeUnit.MILLISECONDS);
return exitVal;
} catch (TimeoutException to) {
p.destroy();
throw to;
}
I think you can't get round the race condition whereby the wait times out, and then process terminates just before you call destroy().
I also tested the worker implementation and works like a charm. Under handling process io, I added threads to handle stde and stdo. If the worker thread times out I also exit the io threads.
Process p = Runtime.getRuntime().exec(cmd.trim());
//setup error and output stream threads
CommandStreamThread eStream = new CommandStreamThread(p.getErrorStream(), "STDE");
CommandStreamThread oStream = new CommandStreamThread(p.getInputStream(), "STDO");
// kick them off
eStream.start();
oStream.start();
//setup a worker thread so we can time it out when we need
CommandWorkerThread worker=new CommandWorkerThread(p);
worker.start();
try {
worker.join(this.getTimeout());
if (worker.getExit() != null)
return worker.getExit();
else
throw new TimeoutException("Timeout reached:"+this.getTimeout()+" ms");
} catch(InterruptedException ex) {
eStream.interrupt();
oStream.interrupt();
worker.interrupt();
Thread.currentThread().interrupt();
throw ex;
} finally {
p.destroy();
}
First some background info, I came across the issue to have a timeout while running a command because the program that I tried to execute would never print any debug or error information incase of error and would just keep on retrying internally by itself resulting in process stuck because there was never an error or output stream when it was retrying.
So after process.exec() or process.start() ,
It would be stuck forever at this line,
BufferedReader input = new BufferedReader(newInputStreamReader(process.getInputStream()));
As per java 1.8 with public boolean waitFor(long timeout,TimeUnit unit) method it should have "ideally" timed out after the specified timeout but in my case for some reason it never timed out may be because I was running application as a windows service (I have checked the user permissions and everything on the account but it didn't help).
So I tried to implement it with the below logic, where we would keep checking the input stream with input.ready() and a timeout flag.This simple solution worked like a charm compared to all other that existed.
Code:
public boolean runCommand() throws IOException, InterruptedException, Exception {
StringBuilder rawResponse = new StringBuilder();
System.out.println("Running Command " + Arrays.toString(command));
ProcessBuilder processBuilder = new ProcessBuilder(Arrays.asList(command));
processBuilder.redirectErrorStream(true);
Process process = processBuilder.start(); //Executing the process
BufferedReader input = new BufferedReader(new InputStreamReader(process.getInputStream()));
waitForTimeout(input, process); //Waiting for Timout
String line;
while ((line = input.readLine()) != null) {
rawResponse.append(line).append("\n");
}
return true;
}
//Timeout method
private void waitForTimeout(BufferedReader input, Process process) throws InterruptedException, Exception {
int timeout = 5;
while (timeout > 0) {
if (!process.isAlive() || input.ready()) {
break;
} else {
timeout--;
Thread.sleep(1000);
if (timeout == 0 && !input.ready()) {
destroyProcess(process);
throw new Exception("Timeout in executing the command "+Arrays.toString(command));
}
}
}
}
You can launch a Thread that sleeps for the time you want and after the sleep changing a boolean that you loop on in your executeCommandLine method.
Something like that (not tested nor compiled, this solution is a prototype you should refactor it if it suit you needs):
public static int executeCommandLine(final String commandLine,
final boolean printOutput,
final boolean printError)
throws IOException, InterruptedException
{
Runtime runtime = Runtime.getRuntime();
Process process = runtime.exec(commandLine);
if (printOutput)
{
BufferedReader outputReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
System.out.println("Output: " + outputReader.readLine());
}
if (printError)
{
BufferedReader errorReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
System.out.println("Error: " + errorReader.readLine());
}
ret = -1;
final[] b = {true};
new Thread(){
public void run(){
Thread.sleep(2000); //to adapt
b[0] = false;
}
}.start();
while(b[0])
{
ret = process.waitFor();
}
return ret;
}
and here is the StreamThread
public class CommandStreamThread extends Thread{
private InputStream iStream;
private String cPrompt;
CommandStreamThread (InputStream is, String cPrompt)
{
this.iStream = is;
this.cPrompt = cPrompt;
}
public void run()
{
try
{
InputStreamReader streamReader= new InputStreamReader(this.iStream);
BufferedReader reader = new BufferedReader(streamReader);
String linesep=System.getProperty("line.separator");
String line=null;
while ((line=reader.readLine())!=null){
System.out.println(line);
//Process the next line seperately in case this is EOF is not preceded by EOL
int in;
char[] buffer=new char[linesep.length()];
while ( (in = reader.read(buffer)) != -1){
String bufferValue=String.valueOf(buffer, 0, in);
System.out.print(bufferValue);
if (bufferValue.equalsIgnoreCase(linesep))
break;
}
}
//Or the easy way out with commons utils!
//IOUtils.copy(this.iStream, System.out);
} catch (Exception e){
e.printStackTrace();
}
}
public InputStream getIStream() {
return iStream;
}
public void setIStream(InputStream stream) {
iStream = stream;
}
public String getCPrompt() {
return cPrompt;
}
public void setCPrompt(String prompt) {
cPrompt = prompt;
}
}
Apache Commons Exec can help you to do it.
See http://commons.apache.org/proper/commons-exec/tutorial.html
String line = "your command line";
CommandLine cmdLine = CommandLine.parse(line);
DefaultExecutor executor = new DefaultExecutor();
ExecuteWatchdog watchdog = new ExecuteWatchdog(60000);
executor.setWatchdog(watchdog);
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
PumpStreamHandler streamHandler = new PumpStreamHandler(outputStream);
executor.setStreamHandler(streamHandler);
int exitValue = executor.execute(cmdLine);
System.out.println(exitValue);
System.out.println(outputStream.toString());
If using Java 8 I'd go with Aleksander Blomskøld answer i.e. p.waitFor(1, TimeUnit.MINUTE)
else if Java 6/7 and using Swing, then you can use a SwingWorker:
final Process process = ...
SwingWorker<Integer, Integer> sw = new SwingWorker<>() {
#Override
protected Integer doInBackground() throws Exception {
process.waitFor();
return process.exitValue();
}
};
sw.execute();
int exitValue = sw.get(1, TimeUnit.SECONDS);
if (exitValue == 0) {
//everything was fine
} else {
//process exited with issues
}
I know this is really old post; i needed some help with a similar project so I thought I might give some of my code that I worked and ones that work.
long current = System.currentTimeMillis();
ProcessBuilder pb = new ProcessBuilder(arguments);
try{
pb.redirectErrorStream(true);
process = pb.start();
int c ;
while((c = process.getInputStream().read()) != -1 )
if(System.currentTimeMillis() - current < timeOutMilli)
result += (char)c;
else throw new Exception();
return result.trim();
}catch(Exception e){
e.printStackTrace();
}
return result;
Hope this helps the future :D