We Keep Coding

Java: Write to and read from same process multiple times

I've gone through so many related StackOverflow questions for this that I'm getting lost in them, and I've coded this multiple ways, but none seem to solve this problem in a way that works for me: How can I send output to the same command and process multiple times while at the same time receiving input from this same process?
(See Input various strings to same process in Java for a similar question, but this ended with only a theoretical answer.)
The command (command line, from a C++ executable) loads a large file, and then I want to send input to it very quickly, get back the answer, do other stuff in between, then send different input and get the corresponding answer. Multiply this by thousands or millions of times.
One implementation, with threads:
ProcessBuilder pb = new ProcessBuilder(command.split(" "));
kenLMProcess = pb.start();
KenLMInThread lmInput = new KenLMInThread(kenLMProcess.getInputStream());
KenLMInThread lmError = new KenLMInThread(kenLMProcess.getErrorStream());
KenLMOutThread lmOutput = new KenLMOutThread(kenLMProcess.getOutputStream());
lmOutput.inStr = "Test . \n";
lmInput.start();
lmOutput.start();
lmError.start();
lmOutput.join();
lmInput.join();
lmError.join();
outStr = lmInput.newStr;
But join waits until the thread ends. What if I don't want to wait for it to end? I can't seem to figure out how to use wait() for that purpose. For one I'd prefer to not have to keep opening and closing a new output stream and input stream every time I query the command. But at least that's better than starting a new ProcessBuilder every time.
Here's what run() looks like for KenLMOutThread:
public void run() {
try {
pw.write(inStr+"\n");
pw.write('\n');
} catch (Exception e) {
System.out.println("Error while inputting to KenLM.");
e.printStackTrace();
} finally {
pw.flush();
try {
pw.flush();
bw.close();
} catch (IOException e) {
e.printStackTrace();
}
}
Supposedly flush is supposed to let it move on, and "\n" at the end is supposed to help, but it just hangs unless I use close. And if I use close, I can't use the OutputStream anymore. I'm also then unable to make a new OutputStream from the Process.
If it helps, here's a more simple implementation with everything together (taken from How to send EOF to a process in Java?):
Note that close() is used, and using flush() without close() causes the program to hang.
public static String pipe(String str, String command2) throws IOException, InterruptedException {
Process p2 = Runtime.getRuntime().exec(command2);
OutputStream out = p2.getOutputStream();
out.write(str.getBytes());
out.close();
p2.waitFor();
BufferedReader reader
= new BufferedReader(new InputStreamReader(p2.getInputStream()));
StringBuilder sb = new StringBuilder();
String line;
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
return sb.toString();
}
Other things I've tried:
Using exec(): Process kenLMProcess=Runtime.getRuntime().exec(command);
Putting the command process in its own thread: KenLMProcessThread procThread = new KenLMProcessThread(pb.start());

If the target process is hanging unless you close the output stream, the problem is at that end: it is reading until end of stream before doing anything. Nothing you can do about that at the sending end.

Java - Use Input and OutputStream of ProcessBuilder continuously

I want to use an external tool while extracting some data (loop through lines).
For that I first used Runtime.getRuntime().exec() to execute it.
But then my extraction got really slow. So I am searching for a possibility to exec the external tool in each instance of the loop, using the same instance of shell.
I found out, that I should use ProcessBuilder. But it's not working yet.
Here is my code to test the execution (with input from the answers here in the forum already):
public class ExecuteShell {
ProcessBuilder builder;
Process process = null;
BufferedWriter process_stdin;
BufferedReader reader, errReader;
public ExecuteShell() {
String command;
command = getShellCommandForOperatingSystem();
if(command.equals("")) {
return; //Fehler! No error handling yet
}
//init shell
builder = new ProcessBuilder( command);
builder.redirectErrorStream(true);
try {
process = builder.start();
} catch (IOException e) {
System.out.println(e);
}
//get stdout of shell
reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
errReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
//get stdin of shell
process_stdin = new BufferedWriter(new OutputStreamWriter(process.getOutputStream()));
System.out.println("ExecuteShell: Constructor successfully finished");
}
public String executeCommand(String commands) {
StringBuffer output;
String line;
try {
//single execution
process_stdin.write(commands);
process_stdin.newLine();
process_stdin.flush();
} catch (IOException e) {
System.out.println(e);
}
output = new StringBuffer();
line = "";
try {
if (!reader.ready()) {
output.append("Reader empty \n");
return output.toString();
}
while ((line = reader.readLine())!= null) {
output.append(line + "\n");
return output.toString();
}
if (!reader.ready()) {
output.append("errReader empty \n");
return output.toString();
}
while ((line = errReader.readLine())!= null) {
output.append(line + "\n");
}
} catch (Exception e) {
System.out.println("ExecuteShell: error in executeShell2File");
e.printStackTrace();
return "";
}
return output.toString();
}
public int close() {
// finally close the shell by execution exit command
try {
process_stdin.write("exit");
process_stdin.newLine();
process_stdin.flush();
}
catch (IOException e) {
System.out.println(e);
return 1;
}
return 0;
}
private static String getShellCommandForOperatingSystem() {
Properties prop = System.getProperties( );
String os = prop.getProperty( "os.name" );
if ( os.startsWith("Windows") ) {
//System.out.println("WINDOWS!");
return "C:/cygwin64/bin/bash";
} else if (os.startsWith("Linux") ) {
//System.out.println("Linux!");
return"/bin/sh";
}
return "";
}
}
I want to call it in another Class like this Testclass:
public class TestExec{
public static void main(String[] args) {
String result = "";
ExecuteShell es = new ExecuteShell();
for (int i=0; i<5; i++) {
// do something
result = es.executeCommand("date"); //execute some command
System.out.println("result:\n" + result); //do something with result
// do something
}
es.close();
}
}
My Problem is, that the output stream is always empty:
ExecuteShell: Constructor successfully finished
result:
Reader empty
result:
Reader empty
result:
Reader empty
result:
Reader empty
result:
Reader empty
I read the thread here: Java Process with Input/Output Stream
But the code snippets were not enough to get me going, I am missing something. I have not really worked with different threads much. And I am not sure if/how a Scanner is of any help to me. I would really appreciate some help.
Ultimatively, my goal is to call an external command repeatetly and make it fast.
EDIT:
I changed the loop, so that the es.close() is outside. And I wanted to add, that I do not want only this inside the loop.
EDIT:
The problem with the time was, that the command I called caused an error. When the command does not cause an error, the time is acceptable.
Thank you for your answers

You are probably experiencing a race condition: after writing the command to the shell, your Java program continues to run, and almost immediately calls reader.ready(). The command you wanted to execute has probably not yet output anything, so the reader has no data available. An alternative explanation would be that the command does not write anything to stdout, but only to stderr (or the shell, maybe it has failed to start the command?). You are however not reading from stderr in practice.
To properly handle output and error streams, you cannot check reader.ready() but need to call readLine() (which waits until data is available) in a loop. With your code, even if the program would come to that point, you would read only exactly one line from the output. If the program would output more than one line, this data would get interpreted as the output of the next command. The typical solution is to read in a loop until readLine() returns null, but this does not work here because this would mean your program would wait in this loop until the shell terminates (which would never happen, so it would just hang infinitely).
Fixing this would be pretty much impossible, if you do not know exactly how many lines each command will write to stdout and stderr.
However, your complicated approach of using a shell and sending commands to it is probably completely unnecessary. Starting a command from within your Java program and from within the shell is equally fast, and much easier to write. Similarly, there is no performance difference between Runtime.exec() and ProcessBuilder (the former just calls the latter), you only need ProcessBuilder if you need its advanced features.
If you are experiencing performance problems when calling external programs, you should find out where they are exactly and try to solve them, but not with this approach. For example, normally one starts a thread for reading from both the output and the error stream (if you do not start separate threads and the command produces large output, everything might hang). This could be slow, so you could use a thread pool to avoid repeated spawning of processes.

java println blocked after communicating with exe if i do not close the stream

I am currently interfacing a GUI to an UCI chess engine. For this purpose i am creating the engine process using:
try {
process = Runtime.getRuntime().exec(enginePath);
} catch (IOException e) {
System.err.println("ENGINE NOT FOUND");
e.printStackTrace();
}
and i am sure that i am able to open the engine.
When engine is opened, there is no significant stream outputted from engine. In order to initiate communication, I have to send specific commands to the engine. The engine will respond then... Therefore it is working in a command/response approach(not immediately streaming data when opened or talking without spoken to). In order to communicate i have a send message block. In this block write a message to the engine using its outputstream and get input using its standart input stream as in the following send method:
private String sendCommand(String command) {
stdin = new PrintWriter(new OutputStreamWriter(process.getOutputStream()));
BufferedReader inputReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
BufferedReader errReader = new BufferedReader(newInputStreamReader(process.getErrorStream()));
String answer = "";
stdin.println(command);
stdin.flush();
stdin.close();
try {
String line = "";
while ((line = inputReader.readLine()) != null) {
System.out.println(line);
answer = line;
}
inputReader.close();
} catch (IOException ioe) {
System.err.println("READ ERROR");
ioe.printStackTrace();
}
try {
String line = "";
while ((line = errReader.readLine()) != null) {
System.err.println(line);
}
errReader.close();
} catch (IOException ioe) {
System.err.println("READ ERROR");
ioe.printStackTrace();
}
return answer;
}
By the way the whole engine is working in a single seperate thread other than the main GUI thread. Therefore no overloading for the GUI. However i didnot seperate the stdin stderr and output methods in individual threads since i suppose there will be only stdin and the one that i will write to the engine. For the time being i assume they will not collide.
With these codes i am able to communicate with the engine and i can see the outputs in the console. However due to stdin.close(); i can only use this method once (In the seconds time i have a STREAM CLOSED error). The chess engine is needed to be communicated in command/response approach many times when opened, without restarting the exe in each time. The problem is that if i remove the line : stdin.close(); yes the communication continues, but my console is blocked by this communication. I.e. i cannot println to the console for debugging purposes anymore. Which is very critical because my main debugging weapon is system.out.println. If i do not remove stdin.close(); i have to restart executable each time i want to send message and i do not want that. **The strange part is that i can see the messages coming from the engine in my console due to "System.out.println(line);" line in the send method; however i cannot print anything on the console once the code exits the method. **
EDIT: Actually after this block:
while ((line = inputReader.readLine()) != null) {
System.out.println(line);
answer = line;
}
System.out.println starts not to work.
EDIT: The problem seems not to be with println but any statement after while loop.
Why statements after while loop is not getting executed?
according to this, it seems that since the stream is never closed, "while loop" is stuck(?). Actually when printing inside while loop, at some point(after a stream is finished), the prints stop, which must indicate while loop is finished. Therefore if it is finished, it should continue on the next statements, shouldn't it? Anyway, the messages from the engine had a set of strings at end of each stream; therefore when i encounter one of them i am breaking the while loop.
Note: it is also interesting that : accumulating data as: "receivedString += line"; doesnot work,i.e. data is not accumulated. In order to fix it i luckily made it "receivedString = receivedString + line + "\n""; and it worked.. I dont know why.

You're reading the input until end of stream, so it won't stop reading until end of stream occurs. End of stream means that the peer has closed the connection in this case. That won't happen for a process's output or error stream until the process exits. You'll have to find some other loop termination condition, or else consume the stdout and stderr in separate threads.

BufferedReader blocking when reading from process.getErrorStream()

When I run this code and the call graph is really large, the program prints to the last line that opt outputs and is blocked at readLine, even though there is nothing left. Anyone know what the problem is? opt -print-callgraph file sends the call graph to the error stream. I tried executing opt -print-callgraph file 2> callgraph so that I can read from a file instead but it complains that there are too many positional arguments.
Oddly enough, the code runs fine for call graphs that are small in size.
I tried using ProcessBuilder as well but I get the same problem.
Runtime runtime = Runtime.getRuntime();
Process process = runtime.exec("opt -print-callgraph " + file);
BufferedReader in = new BufferedReader(new InputStreamReader(process.getErrorStream()));
String s = null;
try {
// Gets stuck at readLine after printing out the last line.
while ((s = in.readLine()) != null) {
System.out.println(s);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
in.close();
}

You need to read both streams, in separate threads, or else merge them so you're reading them both at the same time. Otherwise the process can block if output is unconsumed. In this case there must be unconsumed output in stdout which is blocking the process, which means it won't finish, which means it won't close stderr, which means reading stderr will block.

Error handling in java

Am running .exe file from java code using ProcessBulider, the code I have written is given below. The .exe file takes Input.txt(placed in same directory) as input and provide 3 output file in same directory.
public void ExeternalFileProcessing() throws IOException, InterruptedException {
String executableFileName = "I:/Rod/test.exe;
ProcessBuilder processBuilderObject=new ProcessBuilder(executableFileName,"Input.txt");
File absoluteDirectory = new File("I:/Rod");
processBuilderObject.directory(absoluteDirectory);
Process process = processBuilderObject.start();
process.waitFor();
}
this process is working fine by call ExeternalFileProcessing(). Now am doing validation process, If there is any crash/.exe file doesn't run, I should get the error message how can I get error message?
Note: It would be better that error message be simple like run successful/doesn't run successful or simply true/false, so that I can put this in If condition to continue the remaining process.

You can add exception handlers to get the error message.
public void externalFileProcessing() {
String executableFileName = "I:/Rod/test.exe";
ProcessBuilder processBuilderObject = new ProcessBuilder(
executableFileName, "Input.txt");
File absoluteDirectory = new File("I:/Rod");
processBuilderObject.directory(absoluteDirectory);
try {
Process process = processBuilderObject.start();
process.waitFor();
// this code will be executed if the process works
System.out.println("works");
} catch (IOException e) {
// this code will be executed if a IOException happens "e.getMessage()" will have an error
e.printStackTrace();
} catch (InterruptedException e) {
// this code will be executed if the thread is interrupted
e.printStackTrace();
}
}
But it would be better to handle it in the calling function by put a try catch handler in the calling function and handling it there.

Is it a third party .exe or do you have access to its sources? If so, you could work with basic System outputs (for example couts to the console).
Those outputs can be redirected to your java app using something like this:
InputStream is = process.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line = "";
while ((line = br.readLine()) != null) {
if(line.equals("something")) {
// do something
}
}
br.close();
This is how i do things like that and it works very well in general. But i must admit, that i can not say/garuantee, that this is THE way to do it. A more advanced approach might be the use of StreamGobbler (see Listing 4.5) to handle the outputs of the .exe.
Let me know if it helped you or not.