I need to execute an external application which returns large data (takes more than 2 hours to complete ) nand which continuously outputs data.
What I need to do is execute this program asynchronously and capture the output in a file.
I tried using java process builder, however it seems to hang and return output only when the program is exited or forcefully terminated.
I tried to use process builder and spwaned a new thread to capture the output, but still it did not help.
Then I read about apache commons exec and tried the same . however this also seems to be taking a long time and returns different error codes ( for the same input)
CommandLine cmdLine = new CommandLine("/opt/testsimulator");
DefaultExecuteResultHandler resultHandler = new DefaultExecuteResultHandler();
ByteArrayOutputStream stdout = new ByteArrayOutputStream();
PumpStreamHandler psh = new PumpStreamHandler(stdout);
ExecuteWatchdog watchdog = new ExecuteWatchdog(60*1000);
Executor executor = new DefaultExecutor();
executor.setStreamHandler(psh);
executor.setWatchdog(watchdog);
try {
executor.execute(cmdLine);
} catch (ExecuteException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Any help or working examples whould be very helpful
Huh. Using ProcessBuilder should work for your configuration. For example, the following pattern works for me:
ProcessBuilder pb = new ProcessBuilder("/tmp/x");
Process process = pb.start();
final InputStream is = process.getInputStream();
// the background thread watches the output from the process
new Thread(new Runnable() {
public void run() {
try {
BufferedReader reader =
new BufferedReader(new InputStreamReader(is));
String line;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
} catch (IOException e) {
e.printStackTrace();
} finally {
is.close();
}
}
}).start();
// the outer thread waits for the process to finish
process.waitFor();
The program that I'm running is just a script with a whole bunch of sleep 1 and echo lines:
#!/bin/sh
sleep 1
echo "Hello"
sleep 1
echo "Hello"
...
The thread reading from the process spits out a Hello every second or so.
Related
I am trying to run an async process and I do not want the program to wait until the end of these processes executions. I found this question how to run shell script asynchronously from within Java program but it doesn't have the answer that I am looking for.
What I am doing is I am simply running bash processes and after I run it, I do not want the Java program to wait until it's finished. This is what I have done:
public void runCommandLine(String directory) throws IOException {
Thread commandLineThread = new Thread(() -> {
try {
ProcessBuilder processBuilder = new ProcessBuilder(
"/bin/bash");
processBuilder.directory(new File(directory));
Process process = processBuilder.start();
try (OutputStreamWriter osw = new OutputStreamWriter(process.getOutputStream())) {
osw.write(command);
}
printStream(process.getErrorStream(), true);
printStream(process.getInputStream(), true);
} catch (IOException ex) {
ex.printStackTrace();
}
});
commandLineThread.start();
System.out.println("Task Dispatched");
}
I also put another print out at the end of the main method so I get this output:
Task Dispatched
Task Dispatched
End of psvm
However the program does not terminate as these two processes have not terminated.
How can I solve this issue?
You need to make your thread a daemon thread. Use setDaemon(true) before starting it.
commandLineThread.setDaemon(true);
A daemon thread is a thread that does not prevent the JVM from exiting. See this question: What is Daemon thread in Java? for more information about daemon threads.
Edit:
By judging from your comments you need the command to run even though the JVM is about to exit. I assume the command variable contains the script you want to run? You could make two changes to make the program behave as I think you want.
Start bash with -c to execute your command, then you do not have to send things to an output stream.
Start the process before starting your thread that waits for the output.
The resulting code would look something like:
public void runCommandLine(String directory) throws IOException {
ProcessBuilder processBuilder = new ProcessBuilder(
"/bin/bash -c " + command);
processBuilder.directory(new File(directory));
Process process = processBuilder.start();
Thread commandLineThread = new Thread(() -> {
try {
printStream(process.getErrorStream(), true);
printStream(process.getInputStream(), true);
} catch (IOException ex) {
ex.printStackTrace();
}
});
commandLineThread.setDaemon(true);
commandLineThread.start();
System.out.println("Task Dispatched");
}
You are reading the output streams of your process, and thats the reason your java program does not exit:
printStream(process.getErrorStream(), true);
printStream(process.getInputStream(), true);
Your stream reading will keep blocking your code.
You may like to redirect output of your launched process to a log file and read that later.
Thread commandLineThread = new Thread(() -> {
try {
BufferedReader br=new BufferedReader(
new InputStreamReader(
process.getInputStream()));
String line;
while((line=br.readLine())!=null){
System.out.println(line);
}
} catch (IOException ex) {
ex.printStackTrace();
}
});
commandLineThread.setDaemon(true);
commandLineThread.start();
I am trying to run an async process and I do not want the program to wait until the end of these processes executions. I found this question how to run shell script asynchronously from within Java program but it doesn't have the answer that I am looking for.
What I am doing is I am simply running bash processes and after I run it, I do not want the Java program to wait until it's finished. This is what I have done:
public void runCommandLine(String directory) throws IOException {
Thread commandLineThread = new Thread(() -> {
try {
ProcessBuilder processBuilder = new ProcessBuilder(
"/bin/bash");
processBuilder.directory(new File(directory));
Process process = processBuilder.start();
try (OutputStreamWriter osw = new OutputStreamWriter(process.getOutputStream())) {
osw.write(command);
}
printStream(process.getErrorStream(), true);
printStream(process.getInputStream(), true);
} catch (IOException ex) {
ex.printStackTrace();
}
});
commandLineThread.start();
System.out.println("Task Dispatched");
}
I also put another print out at the end of the main method so I get this output:
Task Dispatched
Task Dispatched
End of psvm
However the program does not terminate as these two processes have not terminated.
How can I solve this issue?
You need to make your thread a daemon thread. Use setDaemon(true) before starting it.
commandLineThread.setDaemon(true);
A daemon thread is a thread that does not prevent the JVM from exiting. See this question: What is Daemon thread in Java? for more information about daemon threads.
Edit:
By judging from your comments you need the command to run even though the JVM is about to exit. I assume the command variable contains the script you want to run? You could make two changes to make the program behave as I think you want.
Start bash with -c to execute your command, then you do not have to send things to an output stream.
Start the process before starting your thread that waits for the output.
The resulting code would look something like:
public void runCommandLine(String directory) throws IOException {
ProcessBuilder processBuilder = new ProcessBuilder(
"/bin/bash -c " + command);
processBuilder.directory(new File(directory));
Process process = processBuilder.start();
Thread commandLineThread = new Thread(() -> {
try {
printStream(process.getErrorStream(), true);
printStream(process.getInputStream(), true);
} catch (IOException ex) {
ex.printStackTrace();
}
});
commandLineThread.setDaemon(true);
commandLineThread.start();
System.out.println("Task Dispatched");
}
You are reading the output streams of your process, and thats the reason your java program does not exit:
printStream(process.getErrorStream(), true);
printStream(process.getInputStream(), true);
Your stream reading will keep blocking your code.
You may like to redirect output of your launched process to a log file and read that later.
Thread commandLineThread = new Thread(() -> {
try {
BufferedReader br=new BufferedReader(
new InputStreamReader(
process.getInputStream()));
String line;
while((line=br.readLine())!=null){
System.out.println(line);
}
} catch (IOException ex) {
ex.printStackTrace();
}
});
commandLineThread.setDaemon(true);
commandLineThread.start();
For a project I need to start python.exe via Runtime.getRuntime.exec(). However, when I try to run it it won't execute, but it doesn't throw up an IOException. Here's the code:
try
{
Process process=Runtime.getRuntime().exec("C:\\Program Files (x86)\\PythonTest\\python.exe");
}
catch (IOException e)
{
System.out.println("Cannot find python.exe");
e.printStackTrace();
}
You need to get the output from the process and (waitFor() it to finish). Something like,
final String cmd = "C:/Program Files (x86)/PythonTest/python.exe";
Process p = Runtime.getRuntime().exec(cmd);
final InputStream is = p.getInputStream();
Thread t = new Thread(new Runnable() {
public void run() {
InputStreamReader isr = new InputStreamReader(is);
int ch;
try {
while ((ch = isr.read()) != -1) {
System.out.print((char) ch);
}
} catch (IOException e) {
e.printStackTrace();
}
}
});
t.start();
p.waitFor();
t.join();
To actually do something with python you'll want to get the OutputStream.
I think that the problem is due to eval incorrectly splitting the command string. My understanding is that exec("C:\\Program Files (x86)\\PythonTest\\python.exe") will attempt to run an application called "C:\\Program", passing it 2 command line arguments.
Try this instead:
exec(new String[]{"C:\\Program Files (x86)\\PythonTest\\python.exe"});
The exec(String, ...) command line parsing is primitive, and often has the incorrect behaviour from the programmer's perspective. The best bet is often to split the command and arguments yourself.
I'm building a simple UI for ffmpeg launching ffmpeg.exe with parameters using exec(). it works on Os X but on Windows 7-8 after few seconds the ffmpeg process suspends itself and resumes only when I kill the father process. (also ddlhost.exe is created)
Ffmpeg.exe converts successfully the same video from cmd.
Searching on the internet I've found this answer but I have the same problem running a simple test program which is not using the Input and Error streams.
Here is the test program code which has the same problem of the main one:
public class Main {
static String param_ffmpeg_1 = "./data/ffmpeg.exe";
static String param_ffmpeg_2 = "-i";
static String in = "./data/source.mov";
static String out = "./data/out.flv";
static Process p;
public static void main(String[] args) {
/*File f = new File(out);
if (f.exists()){
f.delete();
}*/
Runtime rt = Runtime.getRuntime() ;
//String cmd1 = param_ffmpeg_1 + param_ffmpeg_2 + in_path + param_ffmpeg_3 + out_path ;
System.out.println(in);
System.out.println(out);
String[] cmd1 = new String[] { param_ffmpeg_1, param_ffmpeg_2, in, "-ar", "44100", "-vb", "2500k", "-s", "882x496", "-f", "flv", out};
try {
p = rt.exec(cmd1);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
int r = 123456;
try {
r = p.waitFor();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println(r);
}
}
Does ffmpg write anything to the stdout or stderr? If yes you have to consume that. (In seperate threads as you need to consume the stderr and the stdout in parallel) see Process.getInputStream() and Process.getErrorStream() for the details. When the buffer is buffer is full your called programm is stopped and hangs.
The fact that it works in OS/X but not Windows might be caused by different buffer sizes.
You should call getInputStream() and getErrorStream() on the Process returned by Runtime.exec and drain that all the time the process is running. If you do not then it will eventually fill up and block and stop the process from running.
See Java Process with Input/Output Stream for examples.
From the accepted answer
ProcessBuilder builder = new ProcessBuilder("/bin/bash");
builder.redirectErrorStream(true);
Process process = builder.start();
InputStream itsOutput = process.getInputStream();
// Wrap the stream in a Reader ...
while ((line = reader.readLine ()) != null) {
System.out.println ("Stdout: " + line);
}
I am trying to run plink in my own console window. I started by using Process.exec() and that worked fine. The I moved to using ProcessBuilder and now the output is not sent out until I kill the process.
My code looks like this:
class ConsoleOutputThread extends Thread {
public void start(String processName) {
// this was old code: Runtime r = Runtime.getRuntime();
try {
builder = new ProcessBuilder("plink", "-ssh", "192.168.3.21");
builder.redirectErrorStream(true);
process = builder.start();
//this was old code: process = r.exec (processName);
} catch (IOException ex) {
}
start();
}
#Override
public void run() {
try {
BufferedReader is = new BufferedReader(new InputStreamReader(process.getInputStream()));
writer = new BufferedWriter(new OutputStreamWriter(process.getOutputStream()));
try {
process.waitFor();
} catch (InterruptedException ex) {
}
char b[];
b = new char[1];
while(is.read(b, 0, 1)> 0) {
// this is for debug, normally sent to console
System.out.println("Got character: " + b[0]);
}
} catch (IOException ex) {
}
}
}
So, when using Runtime.exec() everything worked fine. Now, with ProcessBuilder, the read function blocks forever (actually until I kill the process, when everuthing is spitted out). However, the error stream works, i.e. if I put a bad option I get the messages in the console.
I am probably missing something here and looking for help.
Thank you
You've set the plink process to write its output to a pipe which is connected to the java process. Anything output by the plink process will be saved in an operating-system buffer until your process reads it. The OS buffer has a limited capacity,, and if plink writes too much data, then it will block until your process reads some data from the buffer.
Unfortunately, the java process waits for the plink process to complete before reading anything from the pipe. So, if the plink process writes too much output, it will block indefinitely.
You should change the java logic to read the plink process's output before calling waitfor().