We Keep Coding

Java: Write to and read from same process multiple times

I've gone through so many related StackOverflow questions for this that I'm getting lost in them, and I've coded this multiple ways, but none seem to solve this problem in a way that works for me: How can I send output to the same command and process multiple times while at the same time receiving input from this same process?
(See Input various strings to same process in Java for a similar question, but this ended with only a theoretical answer.)
The command (command line, from a C++ executable) loads a large file, and then I want to send input to it very quickly, get back the answer, do other stuff in between, then send different input and get the corresponding answer. Multiply this by thousands or millions of times.
One implementation, with threads:
ProcessBuilder pb = new ProcessBuilder(command.split(" "));
kenLMProcess = pb.start();
KenLMInThread lmInput = new KenLMInThread(kenLMProcess.getInputStream());
KenLMInThread lmError = new KenLMInThread(kenLMProcess.getErrorStream());
KenLMOutThread lmOutput = new KenLMOutThread(kenLMProcess.getOutputStream());
lmOutput.inStr = "Test . \n";
lmInput.start();
lmOutput.start();
lmError.start();
lmOutput.join();
lmInput.join();
lmError.join();
outStr = lmInput.newStr;
But join waits until the thread ends. What if I don't want to wait for it to end? I can't seem to figure out how to use wait() for that purpose. For one I'd prefer to not have to keep opening and closing a new output stream and input stream every time I query the command. But at least that's better than starting a new ProcessBuilder every time.
Here's what run() looks like for KenLMOutThread:
public void run() {
try {
pw.write(inStr+"\n");
pw.write('\n');
} catch (Exception e) {
System.out.println("Error while inputting to KenLM.");
e.printStackTrace();
} finally {
pw.flush();
try {
pw.flush();
bw.close();
} catch (IOException e) {
e.printStackTrace();
}
}
Supposedly flush is supposed to let it move on, and "\n" at the end is supposed to help, but it just hangs unless I use close. And if I use close, I can't use the OutputStream anymore. I'm also then unable to make a new OutputStream from the Process.
If it helps, here's a more simple implementation with everything together (taken from How to send EOF to a process in Java?):
Note that close() is used, and using flush() without close() causes the program to hang.
public static String pipe(String str, String command2) throws IOException, InterruptedException {
Process p2 = Runtime.getRuntime().exec(command2);
OutputStream out = p2.getOutputStream();
out.write(str.getBytes());
out.close();
p2.waitFor();
BufferedReader reader
= new BufferedReader(new InputStreamReader(p2.getInputStream()));
StringBuilder sb = new StringBuilder();
String line;
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
return sb.toString();
}
Other things I've tried:
Using exec(): Process kenLMProcess=Runtime.getRuntime().exec(command);
Putting the command process in its own thread: KenLMProcessThread procThread = new KenLMProcessThread(pb.start());

If the target process is hanging unless you close the output stream, the problem is at that end: it is reading until end of stream before doing anything. Nothing you can do about that at the sending end.

sshj: how to read InputStream from long-running command while command is executing

Currently I am executing command over ssh using:
val sshCmd = session.exec(command)
println(IOUtils.readFully(sshCmd.inputStream).toString())
sshCmd.join()
However, to see the output I need to wait until the command is finished.
How can I get "live" response?
I guess I can read the input stream until end of the line occurs and then print the line; however, is there already some method in the library that can help me with this?

It blocks and waits for the whole thing because that's what IOUtils.readFully is meant to do, it reads fully.
Instead, to read line-by-line, you can do something as simple as:
try (BufferedReader reader = new BufferedReader(new InputStreamReader(sshCmd.inputStream))) {
String line;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
} catch (IOException e) {
System.out.println(e);
}

java println blocked after communicating with exe if i do not close the stream

I am currently interfacing a GUI to an UCI chess engine. For this purpose i am creating the engine process using:
try {
process = Runtime.getRuntime().exec(enginePath);
} catch (IOException e) {
System.err.println("ENGINE NOT FOUND");
e.printStackTrace();
}
and i am sure that i am able to open the engine.
When engine is opened, there is no significant stream outputted from engine. In order to initiate communication, I have to send specific commands to the engine. The engine will respond then... Therefore it is working in a command/response approach(not immediately streaming data when opened or talking without spoken to). In order to communicate i have a send message block. In this block write a message to the engine using its outputstream and get input using its standart input stream as in the following send method:
private String sendCommand(String command) {
stdin = new PrintWriter(new OutputStreamWriter(process.getOutputStream()));
BufferedReader inputReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
BufferedReader errReader = new BufferedReader(newInputStreamReader(process.getErrorStream()));
String answer = "";
stdin.println(command);
stdin.flush();
stdin.close();
try {
String line = "";
while ((line = inputReader.readLine()) != null) {
System.out.println(line);
answer = line;
}
inputReader.close();
} catch (IOException ioe) {
System.err.println("READ ERROR");
ioe.printStackTrace();
}
try {
String line = "";
while ((line = errReader.readLine()) != null) {
System.err.println(line);
}
errReader.close();
} catch (IOException ioe) {
System.err.println("READ ERROR");
ioe.printStackTrace();
}
return answer;
}
By the way the whole engine is working in a single seperate thread other than the main GUI thread. Therefore no overloading for the GUI. However i didnot seperate the stdin stderr and output methods in individual threads since i suppose there will be only stdin and the one that i will write to the engine. For the time being i assume they will not collide.
With these codes i am able to communicate with the engine and i can see the outputs in the console. However due to stdin.close(); i can only use this method once (In the seconds time i have a STREAM CLOSED error). The chess engine is needed to be communicated in command/response approach many times when opened, without restarting the exe in each time. The problem is that if i remove the line : stdin.close(); yes the communication continues, but my console is blocked by this communication. I.e. i cannot println to the console for debugging purposes anymore. Which is very critical because my main debugging weapon is system.out.println. If i do not remove stdin.close(); i have to restart executable each time i want to send message and i do not want that. **The strange part is that i can see the messages coming from the engine in my console due to "System.out.println(line);" line in the send method; however i cannot print anything on the console once the code exits the method. **
EDIT: Actually after this block:
while ((line = inputReader.readLine()) != null) {
System.out.println(line);
answer = line;
}
System.out.println starts not to work.
EDIT: The problem seems not to be with println but any statement after while loop.
Why statements after while loop is not getting executed?
according to this, it seems that since the stream is never closed, "while loop" is stuck(?). Actually when printing inside while loop, at some point(after a stream is finished), the prints stop, which must indicate while loop is finished. Therefore if it is finished, it should continue on the next statements, shouldn't it? Anyway, the messages from the engine had a set of strings at end of each stream; therefore when i encounter one of them i am breaking the while loop.
Note: it is also interesting that : accumulating data as: "receivedString += line"; doesnot work,i.e. data is not accumulated. In order to fix it i luckily made it "receivedString = receivedString + line + "\n""; and it worked.. I dont know why.

You're reading the input until end of stream, so it won't stop reading until end of stream occurs. End of stream means that the peer has closed the connection in this case. That won't happen for a process's output or error stream until the process exits. You'll have to find some other loop termination condition, or else consume the stdout and stderr in separate threads.

BufferedReader blocking when reading from process.getErrorStream()

When I run this code and the call graph is really large, the program prints to the last line that opt outputs and is blocked at readLine, even though there is nothing left. Anyone know what the problem is? opt -print-callgraph file sends the call graph to the error stream. I tried executing opt -print-callgraph file 2> callgraph so that I can read from a file instead but it complains that there are too many positional arguments.
Oddly enough, the code runs fine for call graphs that are small in size.
I tried using ProcessBuilder as well but I get the same problem.
Runtime runtime = Runtime.getRuntime();
Process process = runtime.exec("opt -print-callgraph " + file);
BufferedReader in = new BufferedReader(new InputStreamReader(process.getErrorStream()));
String s = null;
try {
// Gets stuck at readLine after printing out the last line.
while ((s = in.readLine()) != null) {
System.out.println(s);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
in.close();
}

You need to read both streams, in separate threads, or else merge them so you're reading them both at the same time. Otherwise the process can block if output is unconsumed. In this case there must be unconsumed output in stdout which is blocking the process, which means it won't finish, which means it won't close stderr, which means reading stderr will block.

Stuck in reader.readLine() [duplicate]

I am trying to connect to the POP server through Sockets in Java. I did the following code to run a LIST command to list all the emails from the server. But I don't know why on the second readLine() to read the second line and onwards, my application hangs at there.
popSock = new Socket(mailHost, pop_PORT);
inn = popSock.getInputStream();
outt = popSock.getOutputStream();
in = new BufferedReader(new InputStreamReader(inn));
out = new PrintWriter(new OutputStreamWriter(outt), true);
//USER and PASS commands to auth the server are ok
out.println("LIST");
String response = in.readLine();
System.out.println(response);
//Attempt to read the second line from the buffer but it hangs at here.
response = in.readLine();
System.out.println(response);
On the second in.readLine(), the application gets stuck at here and doesn't proceed from here. When I run the LIST command on telnet, I get the whole list of emails. So I should get the same response from the socket but I am not. How should I read the whole response line by line from the server?

readLine() won't return until it's read a carriage return or a line feed, which is what you normally get when you read from a terminal or a text file.
I wouldn't be surprised if the POP server doesn't actually tack \r\n on the end of its messages. Try read() instead.

You should be sending \r\n after each command, also, try not using a BufferedInputStream, try reading directly from the InputStream byte by byte to see at which point it actually hangs. The BufferedInputStream may be hanging waiting to read more before returning what it has already read.

Try reading it one character at a time using in.read and printing it. Perhaps, there's an issue with the newline character that the server is sending.

You can try following--
try {
String line = inn.readLine();
while(***input.ready()***)
{
System.out.println(line);
line=inn.readLine();
}
inn.close();
} catch (IOException e) {
e.printStackTrace();
}
where inn is your bufferedReader object whih stores the inputstreamdata