We Keep Coding

Handling interactive prompts in external process java

I'm wrapping command line application that I oftenly use with GUI interface. It basically comes down to executing it (as Java process) and then parsing its responses. However one of the usecases requires to take additional action by enduser (application asks if user wants to overwrite a file) and I'm not sure how to handle that. Both InputStream and ErrorStream freezes as soon as this prompt comes up. Here is a (pretty generic) code of executeCommand method:
private void executeCommand(String command) {
ProcessBuilder processBuilder = new ProcessBuilder();
processBuilder.command("bash", "-c", command);
try {
Process process = processBuilder.start();
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
BufferedReader errorReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
String line = null;
while ((line = reader.readLine()) != null) {
//some actions "File already exists. Do you want to overwrite ? [y/N]" line never gets to this point
}
while ((line = errorReader.readLine()) != null) {
//some actions "File already exists. Do you want to overwrite ? [y/N]" line never gets to this point
}
handleExitCode(process.waitFor(),"Success!");
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
}
I assume that "File already exists. Do you want to overwrite ? [y/N]" prompt is being passed through some other channel. I just don't know how to handle it. The ideal scenario for me would be if I could prompt messageBox with the same question and then pass the response accordingly.

When you are forking a child process, it needs to inherit I/O from parent process to talk stdin and stdout. Use ProcessBuilder.inheritIO to redirect your command streams in your stdin, stdout and stderr.

Java: Write to and read from same process multiple times

I've gone through so many related StackOverflow questions for this that I'm getting lost in them, and I've coded this multiple ways, but none seem to solve this problem in a way that works for me: How can I send output to the same command and process multiple times while at the same time receiving input from this same process?
(See Input various strings to same process in Java for a similar question, but this ended with only a theoretical answer.)
The command (command line, from a C++ executable) loads a large file, and then I want to send input to it very quickly, get back the answer, do other stuff in between, then send different input and get the corresponding answer. Multiply this by thousands or millions of times.
One implementation, with threads:
ProcessBuilder pb = new ProcessBuilder(command.split(" "));
kenLMProcess = pb.start();
KenLMInThread lmInput = new KenLMInThread(kenLMProcess.getInputStream());
KenLMInThread lmError = new KenLMInThread(kenLMProcess.getErrorStream());
KenLMOutThread lmOutput = new KenLMOutThread(kenLMProcess.getOutputStream());
lmOutput.inStr = "Test . \n";
lmInput.start();
lmOutput.start();
lmError.start();
lmOutput.join();
lmInput.join();
lmError.join();
outStr = lmInput.newStr;
But join waits until the thread ends. What if I don't want to wait for it to end? I can't seem to figure out how to use wait() for that purpose. For one I'd prefer to not have to keep opening and closing a new output stream and input stream every time I query the command. But at least that's better than starting a new ProcessBuilder every time.
Here's what run() looks like for KenLMOutThread:
public void run() {
try {
pw.write(inStr+"\n");
pw.write('\n');
} catch (Exception e) {
System.out.println("Error while inputting to KenLM.");
e.printStackTrace();
} finally {
pw.flush();
try {
pw.flush();
bw.close();
} catch (IOException e) {
e.printStackTrace();
}
}
Supposedly flush is supposed to let it move on, and "\n" at the end is supposed to help, but it just hangs unless I use close. And if I use close, I can't use the OutputStream anymore. I'm also then unable to make a new OutputStream from the Process.
If it helps, here's a more simple implementation with everything together (taken from How to send EOF to a process in Java?):
Note that close() is used, and using flush() without close() causes the program to hang.
public static String pipe(String str, String command2) throws IOException, InterruptedException {
Process p2 = Runtime.getRuntime().exec(command2);
OutputStream out = p2.getOutputStream();
out.write(str.getBytes());
out.close();
p2.waitFor();
BufferedReader reader
= new BufferedReader(new InputStreamReader(p2.getInputStream()));
StringBuilder sb = new StringBuilder();
String line;
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
return sb.toString();
}
Other things I've tried:
Using exec(): Process kenLMProcess=Runtime.getRuntime().exec(command);
Putting the command process in its own thread: KenLMProcessThread procThread = new KenLMProcessThread(pb.start());

If the target process is hanging unless you close the output stream, the problem is at that end: it is reading until end of stream before doing anything. Nothing you can do about that at the sending end.

Java - Use Input and OutputStream of ProcessBuilder continuously

I want to use an external tool while extracting some data (loop through lines).
For that I first used Runtime.getRuntime().exec() to execute it.
But then my extraction got really slow. So I am searching for a possibility to exec the external tool in each instance of the loop, using the same instance of shell.
I found out, that I should use ProcessBuilder. But it's not working yet.
Here is my code to test the execution (with input from the answers here in the forum already):
public class ExecuteShell {
ProcessBuilder builder;
Process process = null;
BufferedWriter process_stdin;
BufferedReader reader, errReader;
public ExecuteShell() {
String command;
command = getShellCommandForOperatingSystem();
if(command.equals("")) {
return; //Fehler! No error handling yet
}
//init shell
builder = new ProcessBuilder( command);
builder.redirectErrorStream(true);
try {
process = builder.start();
} catch (IOException e) {
System.out.println(e);
}
//get stdout of shell
reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
errReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
//get stdin of shell
process_stdin = new BufferedWriter(new OutputStreamWriter(process.getOutputStream()));
System.out.println("ExecuteShell: Constructor successfully finished");
}
public String executeCommand(String commands) {
StringBuffer output;
String line;
try {
//single execution
process_stdin.write(commands);
process_stdin.newLine();
process_stdin.flush();
} catch (IOException e) {
System.out.println(e);
}
output = new StringBuffer();
line = "";
try {
if (!reader.ready()) {
output.append("Reader empty \n");
return output.toString();
}
while ((line = reader.readLine())!= null) {
output.append(line + "\n");
return output.toString();
}
if (!reader.ready()) {
output.append("errReader empty \n");
return output.toString();
}
while ((line = errReader.readLine())!= null) {
output.append(line + "\n");
}
} catch (Exception e) {
System.out.println("ExecuteShell: error in executeShell2File");
e.printStackTrace();
return "";
}
return output.toString();
}
public int close() {
// finally close the shell by execution exit command
try {
process_stdin.write("exit");
process_stdin.newLine();
process_stdin.flush();
}
catch (IOException e) {
System.out.println(e);
return 1;
}
return 0;
}
private static String getShellCommandForOperatingSystem() {
Properties prop = System.getProperties( );
String os = prop.getProperty( "os.name" );
if ( os.startsWith("Windows") ) {
//System.out.println("WINDOWS!");
return "C:/cygwin64/bin/bash";
} else if (os.startsWith("Linux") ) {
//System.out.println("Linux!");
return"/bin/sh";
}
return "";
}
}
I want to call it in another Class like this Testclass:
public class TestExec{
public static void main(String[] args) {
String result = "";
ExecuteShell es = new ExecuteShell();
for (int i=0; i<5; i++) {
// do something
result = es.executeCommand("date"); //execute some command
System.out.println("result:\n" + result); //do something with result
// do something
}
es.close();
}
}
My Problem is, that the output stream is always empty:
ExecuteShell: Constructor successfully finished
result:
Reader empty
result:
Reader empty
result:
Reader empty
result:
Reader empty
result:
Reader empty
I read the thread here: Java Process with Input/Output Stream
But the code snippets were not enough to get me going, I am missing something. I have not really worked with different threads much. And I am not sure if/how a Scanner is of any help to me. I would really appreciate some help.
Ultimatively, my goal is to call an external command repeatetly and make it fast.
EDIT:
I changed the loop, so that the es.close() is outside. And I wanted to add, that I do not want only this inside the loop.
EDIT:
The problem with the time was, that the command I called caused an error. When the command does not cause an error, the time is acceptable.
Thank you for your answers

You are probably experiencing a race condition: after writing the command to the shell, your Java program continues to run, and almost immediately calls reader.ready(). The command you wanted to execute has probably not yet output anything, so the reader has no data available. An alternative explanation would be that the command does not write anything to stdout, but only to stderr (or the shell, maybe it has failed to start the command?). You are however not reading from stderr in practice.
To properly handle output and error streams, you cannot check reader.ready() but need to call readLine() (which waits until data is available) in a loop. With your code, even if the program would come to that point, you would read only exactly one line from the output. If the program would output more than one line, this data would get interpreted as the output of the next command. The typical solution is to read in a loop until readLine() returns null, but this does not work here because this would mean your program would wait in this loop until the shell terminates (which would never happen, so it would just hang infinitely).
Fixing this would be pretty much impossible, if you do not know exactly how many lines each command will write to stdout and stderr.
However, your complicated approach of using a shell and sending commands to it is probably completely unnecessary. Starting a command from within your Java program and from within the shell is equally fast, and much easier to write. Similarly, there is no performance difference between Runtime.exec() and ProcessBuilder (the former just calls the latter), you only need ProcessBuilder if you need its advanced features.
If you are experiencing performance problems when calling external programs, you should find out where they are exactly and try to solve them, but not with this approach. For example, normally one starts a thread for reading from both the output and the error stream (if you do not start separate threads and the command produces large output, everything might hang). This could be slow, so you could use a thread pool to avoid repeated spawning of processes.

Redirecting output of a process with Process Builder

I am trying to run plink in my own console window. I started by using Process.exec() and that worked fine. The I moved to using ProcessBuilder and now the output is not sent out until I kill the process.
My code looks like this:
class ConsoleOutputThread extends Thread {
public void start(String processName) {
// this was old code: Runtime r = Runtime.getRuntime();
try {
builder = new ProcessBuilder("plink", "-ssh", "192.168.3.21");
builder.redirectErrorStream(true);
process = builder.start();
//this was old code: process = r.exec (processName);
} catch (IOException ex) {
}
start();
}
#Override
public void run() {
try {
BufferedReader is = new BufferedReader(new InputStreamReader(process.getInputStream()));
writer = new BufferedWriter(new OutputStreamWriter(process.getOutputStream()));
try {
process.waitFor();
} catch (InterruptedException ex) {
}
char b[];
b = new char[1];
while(is.read(b, 0, 1)> 0) {
// this is for debug, normally sent to console
System.out.println("Got character: " + b[0]);
}
} catch (IOException ex) {
}
}
}
So, when using Runtime.exec() everything worked fine. Now, with ProcessBuilder, the read function blocks forever (actually until I kill the process, when everuthing is spitted out). However, the error stream works, i.e. if I put a bad option I get the messages in the console.
I am probably missing something here and looking for help.
Thank you

You've set the plink process to write its output to a pipe which is connected to the java process. Anything output by the plink process will be saved in an operating-system buffer until your process reads it. The OS buffer has a limited capacity,, and if plink writes too much data, then it will block until your process reads some data from the buffer.
Unfortunately, the java process waits for the plink process to complete before reading anything from the pipe. So, if the plink process writes too much output, it will block indefinitely.
You should change the java logic to read the plink process's output before calling waitfor().

Is there a better way to read from a process's inputstream and then handle using specified methods?

I am writing a program doing the following works:
Run a command using ProcessBuilder (like "svn info" or "svn diff");
Read the output of the command from the process's getInputStream();
With the output of the command, I want either:
Parse the output and get what I want and use it later, OR:
Write the output directly to a specified file.
Now what I am doing is using BufferedReader to read whatever the command outputs by lines and save them to an ArrayList, and then decide if I would just scan the lines to find out something or write the lines to a file.
Obviously this is an ugly implement because the ArrayList should not be needed if I want a command's output to be saved to a file. So what will you suggest, to do it in a better way?
Here is some of my codes:
Use this to run command and read from the output of the process
private ArrayList<String> runCommand(String[] command) throws IOException {
ArrayList<String> result = new ArrayList<>();
_processBuilder.command(command);
Process process = null;
try {
process = _processBuilder.start();
try (InputStream inputStream = process.getInputStream();
InputStreamReader inputStreamReader = new InputStreamReader(inputStream);
BufferedReader bufferedReader = new BufferedReader(inputStreamReader)) {
String line;
while ((line = bufferedReader.readLine()) != null) {
result.add(line);
}
}
}
catch (IOException ex) {
_logger.log(Level.SEVERE, "Error!", ex);
}
finally {
if (process != null) {
try {
process.waitFor();
}
catch (InterruptedException ex) {
_logger.log(Level.SEVERE, null, ex);
}
}
}
return result;
}
and in one method I may do like this:
ArrayList<String> reuslt = runCommand(command1);
for (String line: result) {
// ...parse the line here...
}
and in another I may do like this:
ArrayList<String> result = runCommand(command2);
File file = new File(...filename, etc...);
try (PrintWriter printWriter = new PrintWriter(new FileWriter(file, false))) {
for (String line: result) {
printWriter.println(line);
}
}

Returning the process output in an ArrayList seems like a fine abstraction to me. Then the caller of runCommand() doesn't need to worry about how the command was run or the output read. The memory used by the extra list is probably not significant unless your command is very prolix.
The only time I could see this being an issue would be if the caller wanted to start processing the output while the command was still running, which doesn't seem to be the case here.
For very big output that you don't want to copy into memory first, one option would be to have runCommand() take a callback like Guava's LineProcessor that it will call for each line of the output. Then runCommand() can still abstract away the whole deal of running the process, reading the output, and closing everything afterwards, but data can be passed out to the callback as it runs rather than waiting for the method to return the whole response in one array.

I don't think it's a performance issue that you store the text uselessly in some cases. Nonetheless, for cleanliness, it might be better to write two methods:
private ArrayList<String> runCommand(String[] command)
private void runCommandAndDumpToFile(String[] command, File file)
(It wasn't quite clear from your question, but I assume that you know before running your process whether you'll just write the output to file or process it.)