We Keep Coding

In Java, execute other application and receive the screen output

In windows, use Java, mostly we can call Runtime.getRuntime().exec to execute an executable application or batch file, then call proc.getErrorStream() proc.getInputStream() to get the standard output/error stream.
but in this time, I have an application called 'caption2ass.exe' (caption2ass.exe is a well knownd popular tool that can extract ass subtitle from Transport Stream), it prints a lot of information into the screen, but it seems that Java program CAN NOT receive the information by calling proc.getErrorStream() or proc.getInputStream().
Manually I typed 'caption2ass.exe' in the command line, and then I pressed [enter]. after that, the screen will show:
I am trying to receive The infomation in the screen and put it into sysout, or put it into an string array in future.
My Java code is as below:
main program:
String cmd = "E:\\program_media\\Mikey's Fansub Utilities\\TS-OneKeyProcess\\tools\\caption2ass-pcr\\Caption2Ass_PCR.exe";
Runtime run = Runtime.getRuntime();
Process proc = run.exec(cmd);
StreamGobbler errorGobbler = new StreamGobbler(
proc.getErrorStream(), "GBK", "ERR", System.err);
StreamGobbler outputGobbler = new StreamGobbler(
proc.getInputStream(), "GBK", "OUT", System.out);
errorGobbler.start();
outputGobbler.start();
int exitVal = proc.waitFor();
System.out.println("ExitValue: " + exitVal);
StreamGobbler.java :
public class StreamGobbler extends Thread {
InputStream in;
String charsetName;
String type;
PrintStream out;
StreamGobbler(InputStream inputStream, String charsetName, String type, PrintStream out) {
this.in = inputStream;
this.charsetName = charsetName;
this.type = type;
this.out = out;
}
#Override
public void run() {
try {
InputStreamReader isr = new InputStreamReader(in, charsetName);
char[] cbuf = new char[256];
int len = -1;
while ( -1 != (len=isr.read(cbuf))){
out.print(Arrays.copyOf(cbuf, len));
}
} catch (IOException ioe) {
ioe.printStackTrace();
} finally {
IOUtils.closeQuietly(in);
}
}
}
After running this java program, I only got a strange character before ExitValue:
So, My question is: How to get the output information in the screen of this 'caption2ass.exe' using java?
you can get caption2ass from here : http://pan.baidu.com/s/1nuCClXR
, you can run this program in your sandbox if you dare not run an unknown program, especially from Baidu.
Any tests are welcome.

谢谢各位的回复。
I have found the source code of caption2ass, it is a c++ program.
I have changed every logging statment, let them log information into stdout, and then recompiled it.
so i can easily receive the output of caption2ass in the java program, using proc.getInputStream() now.

Java - Use Input and OutputStream of ProcessBuilder continuously

I want to use an external tool while extracting some data (loop through lines).
For that I first used Runtime.getRuntime().exec() to execute it.
But then my extraction got really slow. So I am searching for a possibility to exec the external tool in each instance of the loop, using the same instance of shell.
I found out, that I should use ProcessBuilder. But it's not working yet.
Here is my code to test the execution (with input from the answers here in the forum already):
public class ExecuteShell {
ProcessBuilder builder;
Process process = null;
BufferedWriter process_stdin;
BufferedReader reader, errReader;
public ExecuteShell() {
String command;
command = getShellCommandForOperatingSystem();
if(command.equals("")) {
return; //Fehler! No error handling yet
}
//init shell
builder = new ProcessBuilder( command);
builder.redirectErrorStream(true);
try {
process = builder.start();
} catch (IOException e) {
System.out.println(e);
}
//get stdout of shell
reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
errReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
//get stdin of shell
process_stdin = new BufferedWriter(new OutputStreamWriter(process.getOutputStream()));
System.out.println("ExecuteShell: Constructor successfully finished");
}
public String executeCommand(String commands) {
StringBuffer output;
String line;
try {
//single execution
process_stdin.write(commands);
process_stdin.newLine();
process_stdin.flush();
} catch (IOException e) {
System.out.println(e);
}
output = new StringBuffer();
line = "";
try {
if (!reader.ready()) {
output.append("Reader empty \n");
return output.toString();
}
while ((line = reader.readLine())!= null) {
output.append(line + "\n");
return output.toString();
}
if (!reader.ready()) {
output.append("errReader empty \n");
return output.toString();
}
while ((line = errReader.readLine())!= null) {
output.append(line + "\n");
}
} catch (Exception e) {
System.out.println("ExecuteShell: error in executeShell2File");
e.printStackTrace();
return "";
}
return output.toString();
}
public int close() {
// finally close the shell by execution exit command
try {
process_stdin.write("exit");
process_stdin.newLine();
process_stdin.flush();
}
catch (IOException e) {
System.out.println(e);
return 1;
}
return 0;
}
private static String getShellCommandForOperatingSystem() {
Properties prop = System.getProperties( );
String os = prop.getProperty( "os.name" );
if ( os.startsWith("Windows") ) {
//System.out.println("WINDOWS!");
return "C:/cygwin64/bin/bash";
} else if (os.startsWith("Linux") ) {
//System.out.println("Linux!");
return"/bin/sh";
}
return "";
}
}
I want to call it in another Class like this Testclass:
public class TestExec{
public static void main(String[] args) {
String result = "";
ExecuteShell es = new ExecuteShell();
for (int i=0; i<5; i++) {
// do something
result = es.executeCommand("date"); //execute some command
System.out.println("result:\n" + result); //do something with result
// do something
}
es.close();
}
}
My Problem is, that the output stream is always empty:
ExecuteShell: Constructor successfully finished
result:
Reader empty
result:
Reader empty
result:
Reader empty
result:
Reader empty
result:
Reader empty
I read the thread here: Java Process with Input/Output Stream
But the code snippets were not enough to get me going, I am missing something. I have not really worked with different threads much. And I am not sure if/how a Scanner is of any help to me. I would really appreciate some help.
Ultimatively, my goal is to call an external command repeatetly and make it fast.
EDIT:
I changed the loop, so that the es.close() is outside. And I wanted to add, that I do not want only this inside the loop.
EDIT:
The problem with the time was, that the command I called caused an error. When the command does not cause an error, the time is acceptable.
Thank you for your answers

You are probably experiencing a race condition: after writing the command to the shell, your Java program continues to run, and almost immediately calls reader.ready(). The command you wanted to execute has probably not yet output anything, so the reader has no data available. An alternative explanation would be that the command does not write anything to stdout, but only to stderr (or the shell, maybe it has failed to start the command?). You are however not reading from stderr in practice.
To properly handle output and error streams, you cannot check reader.ready() but need to call readLine() (which waits until data is available) in a loop. With your code, even if the program would come to that point, you would read only exactly one line from the output. If the program would output more than one line, this data would get interpreted as the output of the next command. The typical solution is to read in a loop until readLine() returns null, but this does not work here because this would mean your program would wait in this loop until the shell terminates (which would never happen, so it would just hang infinitely).
Fixing this would be pretty much impossible, if you do not know exactly how many lines each command will write to stdout and stderr.
However, your complicated approach of using a shell and sending commands to it is probably completely unnecessary. Starting a command from within your Java program and from within the shell is equally fast, and much easier to write. Similarly, there is no performance difference between Runtime.exec() and ProcessBuilder (the former just calls the latter), you only need ProcessBuilder if you need its advanced features.
If you are experiencing performance problems when calling external programs, you should find out where they are exactly and try to solve them, but not with this approach. For example, normally one starts a thread for reading from both the output and the error stream (if you do not start separate threads and the command produces large output, everything might hang). This could be slow, so you could use a thread pool to avoid repeated spawning of processes.

ffmpeg process saying it is erroring

I am using FFMPEG to convert an audio file. I call the command line and in turn FFMPEG from Java. I do this using Runtime and Process Runtime.exec(). For the process I have an InputStream and an ErrorStream. For some reason, even though the FFMPEG commands works fine, it prints out from the Error stream giving the impression it has failed. Anyone know why this might be happening? Obviously it's not a major issue as it does work fine, but if for some reason it does actually error, or someone new to the project does not realise this is how it works it could be confusing.
Any ideas?
Here is the relevant code:
Runtime rt = Runtime.getRuntime();
System.out.println("Execing " + cmd);
Process proc = rt.exec(cmd);
StreamGobbler errorGobbler = new StreamGobbler(proc.getErrorStream(), "ERROR");
// any output?
StreamGobbler outputGobbler = new StreamGobbler(proc.getInputStream(), "OUTPUT");
// kick them off
errorGobbler.start();
outputGobbler.start();
// any error???
int exitVal = proc.waitFor();
System.out.println("ExitValue: " + exitVal);
And the StreamGobbler class:
class StreamGobbler extends Thread {
InputStream is;
String type;
StreamGobbler(InputStream is, String type) {
this.is = is;
this.type = type;
}
public void run() {
try {
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line = null;
while ((line = br.readLine()) != null)
System.out.println(type + ">" + line);
} catch (IOException ioe) {
ioe.printStackTrace();
}
}

What you are experiencing is the standard practice across all *NIX tools: they send the "business value" data to stdout and separately send any diagnostic output to stderr. This is an important hallmark of tools written in the "small is beautiful" paradigm, which greatly values the concept of piping one tool's output into the next one's input, thereby creating an ad-hoc chain of information processing. The complete scheme would break down if the tool sent "Processing... 50% done" kind of output to stdout.
Feel free to ignore such output on stderr, but make sure you consume it, as you are doing now, in order to avoid blocking due to overfull output buffers.

Java Process.waitFor() and Readline hangs

First, this is my code :
import java.io.*;
import java.util.Date;
import com.banctecmtl.ca.vlp.shared.exceptions.*;
public class PowershellTest implements Runnable {
public static final String PATH_TO_SCRIPT = "C:\\Scripts\\ScriptTest.ps1";
public static final String SERVER_IP = "XX.XX.XX.XXX";
public static final String MACHINE_TO_MOD = "MachineTest";
/**
* #param args
* #throws OperationException
*/
public static void main(String[] args) throws OperationException {
new PowershellTest().run();
}
public PowershellTest(){}
#Override
public synchronized void run() {
String input = "";
String error = "";
boolean isHanging = false;
try {
Runtime runtime = Runtime.getRuntime();
Process proc = runtime.exec("powershell -file " + PATH_TO_SCRIPT +" "+ SERVER_IP +" "+ MACHINE_TO_MOD);
proc.getOutputStream().close();
InputStream inputstream = proc.getInputStream();
InputStreamReader inputstreamreader = new InputStreamReader(inputstream);
BufferedReader bufferedreader = new BufferedReader(inputstreamreader);
proc.waitFor();
String line;
while (!isHanging && (line = bufferedreader.readLine()) != null) {
input += (line + "\n");
Date date = new Date();
while(!bufferedreader.ready()){
this.wait(1000);
//if its been more then 1 minute since a line has been read, its hanging.
if(new Date().getTime() - date.getTime() >= 60000){
isHanging = true;
break;
}
}
}
inputstream.close();
inputstream = proc.getErrorStream();
inputstreamreader = new InputStreamReader(inputstream);
bufferedreader = new BufferedReader(inputstreamreader);
isHanging = false;
while (!isHanging && (line = bufferedreader.readLine()) != null) {
error += (line + "\n");
Date date = new Date();
while(!bufferedreader.ready()){
this.wait(1000);
//if its been more then 1 minute since a line has been read, its hanging.
if(new Date().getTime() - date.getTime() >= 60000){
isHanging = true;
break;
}
}
}
inputstream.close();
proc.destroy();
} catch (IOException e) {
//throw new OperationException("File IO problem.", e);
} catch (InterruptedException e) {
//throw new OperationException("Script thread problem.",e);
}
System.out.println("Error : " + error + "\nInput : " + input);
}
}
I'm currently trying to run a powershell script that will start/stop a vm (VMWARE) on a remote server. The script work from command line and so does this code. The thing is, I hate how I have to use a thread (and make it wait for the script to respond, as explained further) for such a job. I had to do it because both BufferedReader.readline() and proc.waitFor() hang forever.
The script, when ran from cmd, is long to execute. it stall for 30 sec to 1 min from validating authentification with the server and executing the actual script. From what I saw from debugging, the readline hang when it start receiving those delays from the script.
I'm also pretty sure it's not a memory problem since I never had any OOM error in any debugging session.
Now I understand that Process.waitFor() requires me to flush the buffer from both the error stream and the regular stream to work and so that's mainly why I don't use it (I need the output to manage VM specific errors, certificates issues, etc.).
I would like to know if someone could explain to me why it hangs and if there is a way to just use the typical readline() without having it to hang so hard. Even if the script should have ended since a while, it still hang (I tried to run both the java application and a cmd command using the exact same thing I use in the java application at the same time, left it runingfor 1h, nothing worked). It is not just stuck in the while loop, the readline() is where the hanging is.
Also this is a test version, nowhere close to the final code, so please spare me the : this should be a constant, this is useless, etc. I will clean the code later. Also the IP is not XX.XX.XX.XXX in my code, obviously.
Either explanation or suggestion on how to fix would be greatly appreciated.
Ho btw here is the script I currently use :
Add-PSSnapin vmware.vimautomation.core
Connect-VIServer -server $args[0]
Start-VM -VM "MachineTest"
If you need more details I will try to give as much as I can.
Thanks in advance for your help!
EDIT : I also previously tested the code with a less demanding script, which job was to get the content of a file and print it. Since no waiting was needed to get the information, the readline() worked well. I'm thus fairly certain that the problem reside on the wait time coming from the script execution.
Also, forgive my errors, English is not my main language.
Thanks in advance for your help!
EDIT2 : Since I cannot answer to my own Question :
Here is my "final" code, after using threads :
import java.io.*;
public class PowershellTest implements Runnable {
public InputStream is;
public PowershellTest(InputStream newIs){
this.is = newIs;
}
#Override
public synchronized void run() {
String input = "";
String error = "";
try {
InputStreamReader inputstreamreader = new InputStreamReader(is);
BufferedReader bufferedreader = new BufferedReader(inputstreamreader);
String line;
while ((line = bufferedreader.readLine()) != null) {
input += (line + "\n");
}
is.close();
} catch (IOException e) {
//throw new OperationException("File IO problem.", e);
}
System.out.println("Error : " + error + "\nInput : " + input);
}
}
And the main simply create and start 2 thread (PowerShellTest instances), 1 with the errorStream and 1 with the inputStream.
I believe I made a dumb error when I first coded the app and fixed it somehow as I reworked the code over and over. It still take a good 5-6 mins to run, which is somehow similar if not longer than my previous code (which is logical since the errorStream and inputStream get their information sequentially in my case).
Anyway, thanks to all your answer and especially Miserable Variable for his hint on threading.

First, don't call waitFor() until after you've finished reading the streams. I would highly recommend you look at ProcessBuilder instead of simply using Runtime.exec, and split the command up yourself rather than relying on Java to do it for you:
ProcessBuilder pb = new ProcessBuilder("powershell", "-file", PATH_TO_SCRIPT,
SERVER_IP, MACHINE_TO_MOD);
pb.redirectErrorStream(true); // merge stdout and stderr
Process proc = pb.start();
redirectErrorStream merges the error output into the normal output, so you only have to read proc.getInputStream(). You should then be able to just read that stream until EOF, then call proc.waitFor().

You are currently waiting to complete reading from inputStream before starting to read from errorStream. If the process writes to its stderr before stdout maybe you are getting into a deadlock situation.
Try reading from both streams from concurrently running threads. While you are at it, also remove proc.getOutputStream().close();. It shouldn't affect the behavior, but it is not required either.

Run external program from Java, read output, allow interruption

I want to launch a process from Java, read its output, and get its return code. But while it's executing, I want to be able to cancel it. I start out by launching the process:
ProcessBuilder pb = new ProcessBuilder(args);
pb.redirectErrorStream(true);
Process proc = pb.start();
If I call proc.waitFor(), I can't do anything until the process exits. So I'm assuming I need to something like this:
while (true) {
see if process has exited
capture some output from the process
decide if I want to cancel it, and if so, cancel it
sleep for a while
}
Is this right? Can someone give me an example of how to do this in Java?

Here's an example of what I think you want to do:
ProcessBuilder pb = new ProcessBuilder(args);
pb.redirectErrorStream(true);
Process proc = pb.start();
InputStream is = proc.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line;
int exit = -1;
while ((line = br.readLine()) != null) {
// Outputs your process execution
System.out.println(line);
try {
exit = proc.exitValue();
if (exit == 0) {
// Process finished
}
} catch (IllegalThreadStateException t) {
// The process has not yet finished.
// Should we stop it?
if (processMustStop())
// processMustStop can return true
// after time out, for example.
proc.destroy();
}
}
You can improve it :-) I don't have a real environment to test it now, but you can find some more information here.

I recommend checking out Apache Commons Exec to avoid recreating the wheel. It has some nice features like choosing between synchronous vs. asynchronous execution, as well as a standard solution to spawning a watchdog process that can help in timing out the execution in case it gets stuck.

A helper class like this would do the trick:
public class ProcessWatcher implements Runnable {
private Process p;
private volatile boolean finished = false;
public ProcessWatcher(Process p) {
this.p = p;
new Thread(this).start();
}
public boolean isFinished() {
return finished;
}
public void run() {
try {
p.waitFor();
} catch (Exception e) {}
finished = true;
}
}
You would then implement your loop exactly as you describe:
Process p = Runtime.getRuntime().exec("whatever command");
ProcessWatcher pw = new ProcessWatcher(p);
InputStream output = p.getInputStream();
while(!pw.isFinished()) {
processOutput(output);
if(shouldCancel()) p.destroy();
Thread.sleep(500);
}
Depending upon what conditions would make you want to destroy the process, you might want to do that in a separate thread. Otherwise, you may block while waiting for more program output to process and never really get the option to destroy it.
EDIT: McDowell is 100% right in his comment below, so I've made the finished variable volatile.

How about this (see how it works in jcabi-heroku-maven-plugin):
/**
* Wait for the process to stop, logging its output in parallel.
* #param process The process to wait for
* #return Stdout produced by the process
* #throws InterruptedException If interrupted in between
*/
private String waitFor(final Process process) throws InterruptedException {
final BufferedReader reader = new BufferedReader(
new InputStreamReader(process.getInputStream())
);
final CountDownLatch done = new CountDownLatch(1);
final StringBuffer stdout = new StringBuffer();
new Thread(
new VerboseRunnable(
new Callable<Void>() {
#Override
public Void call() throws Exception {
while (true) {
final String line = reader.readLine();
if (line == null) {
break;
}
System.out.println(">> " + line);
stdout.append(line);
}
done.countDown();
return null;
}
},
false
)
).start();
try {
process.waitFor();
} finally {
done.await();
IOUtils.closeQuietly(reader);
}
return stdout.toString();
}
ps. Now this implementation is available as com.jcabi.log.VerboseProcess class from jcabi-log artifact.

What would make you decide to kill the process -- an asynchronous event (such as input from the user), or a synchronous event (e.g., the process has done what you wanted it to do)? I'm guessing it's the former -- input from the user makes you decide to cancel the subprocess.
Also, how much output do you expect the subprocess to produce? If it's a lot, then the subprocess may block if you don't read from its output stream quickly enough.
Your situation may vary, but it seems that you're likely going to need at least two different threads -- one to decide whether to cancel the process, and one that handles the output from the subprocess.
Have a look here for a bit more detail: http://java.sun.com/developer/JDCTechTips/2005/tt0727.html#2