We Keep Coding

Java: Write to and read from same process multiple times

I've gone through so many related StackOverflow questions for this that I'm getting lost in them, and I've coded this multiple ways, but none seem to solve this problem in a way that works for me: How can I send output to the same command and process multiple times while at the same time receiving input from this same process?
(See Input various strings to same process in Java for a similar question, but this ended with only a theoretical answer.)
The command (command line, from a C++ executable) loads a large file, and then I want to send input to it very quickly, get back the answer, do other stuff in between, then send different input and get the corresponding answer. Multiply this by thousands or millions of times.
One implementation, with threads:
ProcessBuilder pb = new ProcessBuilder(command.split(" "));
kenLMProcess = pb.start();
KenLMInThread lmInput = new KenLMInThread(kenLMProcess.getInputStream());
KenLMInThread lmError = new KenLMInThread(kenLMProcess.getErrorStream());
KenLMOutThread lmOutput = new KenLMOutThread(kenLMProcess.getOutputStream());
lmOutput.inStr = "Test . \n";
lmInput.start();
lmOutput.start();
lmError.start();
lmOutput.join();
lmInput.join();
lmError.join();
outStr = lmInput.newStr;
But join waits until the thread ends. What if I don't want to wait for it to end? I can't seem to figure out how to use wait() for that purpose. For one I'd prefer to not have to keep opening and closing a new output stream and input stream every time I query the command. But at least that's better than starting a new ProcessBuilder every time.
Here's what run() looks like for KenLMOutThread:
public void run() {
try {
pw.write(inStr+"\n");
pw.write('\n');
} catch (Exception e) {
System.out.println("Error while inputting to KenLM.");
e.printStackTrace();
} finally {
pw.flush();
try {
pw.flush();
bw.close();
} catch (IOException e) {
e.printStackTrace();
}
}
Supposedly flush is supposed to let it move on, and "\n" at the end is supposed to help, but it just hangs unless I use close. And if I use close, I can't use the OutputStream anymore. I'm also then unable to make a new OutputStream from the Process.
If it helps, here's a more simple implementation with everything together (taken from How to send EOF to a process in Java?):
Note that close() is used, and using flush() without close() causes the program to hang.
public static String pipe(String str, String command2) throws IOException, InterruptedException {
Process p2 = Runtime.getRuntime().exec(command2);
OutputStream out = p2.getOutputStream();
out.write(str.getBytes());
out.close();
p2.waitFor();
BufferedReader reader
= new BufferedReader(new InputStreamReader(p2.getInputStream()));
StringBuilder sb = new StringBuilder();
String line;
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
return sb.toString();
}
Other things I've tried:
Using exec(): Process kenLMProcess=Runtime.getRuntime().exec(command);
Putting the command process in its own thread: KenLMProcessThread procThread = new KenLMProcessThread(pb.start());

If the target process is hanging unless you close the output stream, the problem is at that end: it is reading until end of stream before doing anything. Nothing you can do about that at the sending end.

Java - Use Input and OutputStream of ProcessBuilder continuously

I want to use an external tool while extracting some data (loop through lines).
For that I first used Runtime.getRuntime().exec() to execute it.
But then my extraction got really slow. So I am searching for a possibility to exec the external tool in each instance of the loop, using the same instance of shell.
I found out, that I should use ProcessBuilder. But it's not working yet.
Here is my code to test the execution (with input from the answers here in the forum already):
public class ExecuteShell {
ProcessBuilder builder;
Process process = null;
BufferedWriter process_stdin;
BufferedReader reader, errReader;
public ExecuteShell() {
String command;
command = getShellCommandForOperatingSystem();
if(command.equals("")) {
return; //Fehler! No error handling yet
}
//init shell
builder = new ProcessBuilder( command);
builder.redirectErrorStream(true);
try {
process = builder.start();
} catch (IOException e) {
System.out.println(e);
}
//get stdout of shell
reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
errReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
//get stdin of shell
process_stdin = new BufferedWriter(new OutputStreamWriter(process.getOutputStream()));
System.out.println("ExecuteShell: Constructor successfully finished");
}
public String executeCommand(String commands) {
StringBuffer output;
String line;
try {
//single execution
process_stdin.write(commands);
process_stdin.newLine();
process_stdin.flush();
} catch (IOException e) {
System.out.println(e);
}
output = new StringBuffer();
line = "";
try {
if (!reader.ready()) {
output.append("Reader empty \n");
return output.toString();
}
while ((line = reader.readLine())!= null) {
output.append(line + "\n");
return output.toString();
}
if (!reader.ready()) {
output.append("errReader empty \n");
return output.toString();
}
while ((line = errReader.readLine())!= null) {
output.append(line + "\n");
}
} catch (Exception e) {
System.out.println("ExecuteShell: error in executeShell2File");
e.printStackTrace();
return "";
}
return output.toString();
}
public int close() {
// finally close the shell by execution exit command
try {
process_stdin.write("exit");
process_stdin.newLine();
process_stdin.flush();
}
catch (IOException e) {
System.out.println(e);
return 1;
}
return 0;
}
private static String getShellCommandForOperatingSystem() {
Properties prop = System.getProperties( );
String os = prop.getProperty( "os.name" );
if ( os.startsWith("Windows") ) {
//System.out.println("WINDOWS!");
return "C:/cygwin64/bin/bash";
} else if (os.startsWith("Linux") ) {
//System.out.println("Linux!");
return"/bin/sh";
}
return "";
}
}
I want to call it in another Class like this Testclass:
public class TestExec{
public static void main(String[] args) {
String result = "";
ExecuteShell es = new ExecuteShell();
for (int i=0; i<5; i++) {
// do something
result = es.executeCommand("date"); //execute some command
System.out.println("result:\n" + result); //do something with result
// do something
}
es.close();
}
}
My Problem is, that the output stream is always empty:
ExecuteShell: Constructor successfully finished
result:
Reader empty
result:
Reader empty
result:
Reader empty
result:
Reader empty
result:
Reader empty
I read the thread here: Java Process with Input/Output Stream
But the code snippets were not enough to get me going, I am missing something. I have not really worked with different threads much. And I am not sure if/how a Scanner is of any help to me. I would really appreciate some help.
Ultimatively, my goal is to call an external command repeatetly and make it fast.
EDIT:
I changed the loop, so that the es.close() is outside. And I wanted to add, that I do not want only this inside the loop.
EDIT:
The problem with the time was, that the command I called caused an error. When the command does not cause an error, the time is acceptable.
Thank you for your answers

You are probably experiencing a race condition: after writing the command to the shell, your Java program continues to run, and almost immediately calls reader.ready(). The command you wanted to execute has probably not yet output anything, so the reader has no data available. An alternative explanation would be that the command does not write anything to stdout, but only to stderr (or the shell, maybe it has failed to start the command?). You are however not reading from stderr in practice.
To properly handle output and error streams, you cannot check reader.ready() but need to call readLine() (which waits until data is available) in a loop. With your code, even if the program would come to that point, you would read only exactly one line from the output. If the program would output more than one line, this data would get interpreted as the output of the next command. The typical solution is to read in a loop until readLine() returns null, but this does not work here because this would mean your program would wait in this loop until the shell terminates (which would never happen, so it would just hang infinitely).
Fixing this would be pretty much impossible, if you do not know exactly how many lines each command will write to stdout and stderr.
However, your complicated approach of using a shell and sending commands to it is probably completely unnecessary. Starting a command from within your Java program and from within the shell is equally fast, and much easier to write. Similarly, there is no performance difference between Runtime.exec() and ProcessBuilder (the former just calls the latter), you only need ProcessBuilder if you need its advanced features.
If you are experiencing performance problems when calling external programs, you should find out where they are exactly and try to solve them, but not with this approach. For example, normally one starts a thread for reading from both the output and the error stream (if you do not start separate threads and the command produces large output, everything might hang). This could be slow, so you could use a thread pool to avoid repeated spawning of processes.

Java Process/IO Execution Hang

Here is my workflow:
I get job from DB, I run a few tasks, I run an external program that reads a file and produces another one (this usually takes under 10 seconds). Here is the code:
Process p = Runtime.getRuntime().exec(prog, null, new File(path));
BufferedReader stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));
BufferedReader stdError = new BufferedReader(new InputStreamReader(p.getErrorStream()));
String s;
String errorString = "";
while((s = stdInput.readLine()) != null) {
if(s.toLowerCase().contains("error")) {
Log.writeLog("error: " + s);
errorString += s + "\r\n";
}
}
if(errorString.length() > 1) {
Emailer.email(name + "*" + errorString, "ERROR");
}
while((s = stdError.readLine()) != null) {
Log.writeLog("ERROR: " + s);
}
However, the snippet hanged. I control the server that the code runs on through LogMeIn, once I logged in, the process unblocked (total running time around 280 seconds) and continued. The process did not produce an ERROR results. This happens from time to time more often than we would like to. We do quite a bit of small IOs operation in the program and the harddrive gets pretty full from time to time.
Any idea what could be happening?
Thanks!
EDIT: the server is a just a regular computer that is connected to LogMeIn. My fear is that since it is a regular computer, it may powerdown the CPU/hard drive when not in use (not sure the correct terminology). This would somewhat explain why it would continue once I logged in to LogMeIn and had access to a computer.
EDIT2: directly following the process, I run this. And this hangs for an absurd amount of time as well (usually 5 seconds, took 200+ seconds). Makes me thing that the hard drive is decided to take a nap?
private void cleanup(String path) {
File srcPath = new File(path);
File[] files = srcPath.listFiles();
if(files != null) {
for(File file : files) {
if(file.isDirectory()) {
cleanup(file.getAbsolutePath());
} else {
if(file.getAbsolutePath().endsWith(".original")) {
String fileName = file.getAbsolutePath().substring(0, file.getAbsolutePath().lastIndexOf(".original"));
IO.delete(fileName);
if(!IO.renameFile(file.getAbsolutePath(), new File(fileName).getAbsolutePath())) {
Log.writeLogSevere("Failed to rename file, this could be a problem..." + fileName);
} else {
Log.writeLog("Cleaned up: " + fileName);
}
}
}
}
}
}

You are not draining the error stream. You do it at the end, which may often be too late. The output buffer of the process fills up and the process blocks waiting to get more space in the stderr output buffer.
You must either use a separate thread for that, or (much simpler) redirectErrorStream using ProcessBuilder.

The most likely thing is that the thread running p didn't die and p.getInputStream() is not null, and but not data on it.
While it's hanging, I would check the current running processes (ps command on Unix) or Task manager on windows. This will tell you if p is done or not. If it is not, then whatever that program is, has issues and it's holding up the rest of your code.

How to detect java process exit?

In a java program, I am generating an sh script for use on a centOS machine, which will use sox and lame to decode an MP3 audio file, then apply some gain to the file respectively. Im having some issues getting the Process.waitFor() method to do anything other than hang indefinitely. Here is the code:
try
{
// TODO code application logic here
String reviewPath = "/SomeDirectory/";
String fileName = "FileName";
String extension = ".mp3";
StringBuilder sb = new StringBuilder();
sb.append("#!/bin/bash\n");
sb.append("cd " + reviewPath + "\n");
sb.append("lame --decode " + fileName + extension + "\n");
File script = new File(reviewPath + fileName + ".sh");
script.createNewFile();
script.setExecutable(true);
FileWriter writer = new FileWriter(script);
writer.write(sb.toString());
writer.close();
Process p = Runtime.getRuntime().exec(script.getAbsolutePath());
String line;
BufferedReader bri = new BufferedReader
(new InputStreamReader(p.getInputStream()));
BufferedReader bre = new BufferedReader
(new InputStreamReader(p.getErrorStream()));
while ((line = bri.readLine()) != null) {
System.out.println(line);
}
bri.close();
while ((line = bre.readLine()) != null) {
System.out.println(line);
}
bre.close();
p.waitFor();
System.out.println("Done.");
}
catch (Exception e)
{
System.out.println(e.getMessage());
}
The odd part is that when I run the .sh file it generates by hand, it runs and exits nicely, but when I execute it from a process object in java, it never exits. The exitValue of the process is always "Process has not exited". Ive tried adding set -e to the script, and exit to the end of the script. Short of using the kill command (which I dont really think I can do here) Im at a loss as to what is going on here. Any suggestions?

Add something like while(p.getInputStream().read() != -1); after starting the process. The buffer will get filled and the process will stop waiting for something (in this case, your program) to read from it to free up space.

I figured it out! The problem here was indeed that the output streams needed to be flushed for the application to exit, but simply reading from the streams is not enough. I used Suresh Koya's suggestion and used the processBuilder api, and redirected the error stream on the process before starting it, and read from the streams. This fixed the issues I was having :D

How to run command line within java using thread async

I am new to java. I am tasked to write java program to run the command lines. I tested the command line under the DOS prompt since i do not have have access to Linux box yet. it worked fine. See the PROGRAM below for full command line syntax. the job will take 6 input files and generate some output files. Next i tried to create a class to and using getruntime and process to process this job. Even it compiled without error but when i run it just show the cursor blinking... i thought i need to use Thread async technique. please provide some advices since i do not have enough time for the projects. I also would like to implement a call back or return values when the job is done. an example would be greatly appreciated. Thanks
import java.io.*;
public class RunJob {
// public static final String PROGRAM = "c:\\wrk\\java.exe Hello";
//one command line below
public static final String PROGRAM = "c:/java.exe -cp \"wrk/jmp.jar;wrk/colt.jar\" gov.lanl.yadas.reliability.UltimateMissileReliabilityModel 10000 \"wrk/\" x1.dat x2c.dat x3.dat x4.dat x5.dat x6.dat true";
// Set to true to end the loop
static boolean done = false;
public static void main(String argv[]) throws IOException {
BufferedReader is;
String line;
String returnMsg = "Start ";
final Process p = Runtime.getRuntime().exec(PROGRAM);
System.out.println("start");
Thread waiter = new Thread() {
public void run() {
try {
p.waitFor();
} catch (InterruptedException ex) {
System.out.println("InterruptedException");
return;
}
System.out.println("Program terminated!");
done = true;
}
};
waiter.start();
is = new BufferedReader(new InputStreamReader(p.getInputStream()));
while (!done && ((line = is.readLine()) != null))
{
System.out.println(line);
returnMsg = returnMsg + line;
}
System.out.println(returnMsg);
System.out.println("End");
return;
}// main
}

I assume that there is a good reason why you want to run a java program from another java program and not just from a shell script, or by invoking an API - but if not - please reconsider.
As to your problem - if your application produces a lot of output (the one you are running as a process) - your application will hang. The p.waitFor() will halt until the process ends. But if you don't read the information from the InputStream - it will overflow and hang!
Advice #1: put the p.waitFor() at the end.
Advice #2: read this article. If I remember correctly it is the one I read when I had a similar problem. You can also google for "StreamGobbler" - it is a common name for a separate thread that "gobbles" your streams.
Advice #3: Don't forget the ErrorStream - if your application will produce too many errors - that stream will cause the process to hang as well.