We Keep Coding

Java Process Builder always giving data in Error Stream [duplicate]

Can anyone clarify me if the below procedure is correct way to handle streams of process without any stream buffer full and blocking problems
I'm invoking external program from java program, I'm using ProcessBuilder to build the process and after I perform
Process gpgProcess = processBuilder.start();
I'm handling the process using a method
String executionResult = verifyExecution(gpgProcess);
and in my method i'm trying to handle the process streams
private String verifyExecution(Process gpgProcess) throws IOException, InterruptedException {
String gpgResult = null;
BufferedReader stdOut = new BufferedReader(new InputStreamReader(gpgProcess.getInputStream()));
BufferedReader stdErr = new BufferedReader(new InputStreamReader(gpgProcess.getErrorStream()));
gpgProcess.waitFor();
if(stdErr.ready()) {
gpgResult = "Exit code: " + gpgProcess.exitValue() + "\n" + readStream(stdErr);
} else if(stdOut.ready()) {
gpgResult = "Exit code: " + gpgProcess.exitValue() + "\n" + readStream(stdOut);
} else {
gpgResult = "Exit code: " + gpgProcess.exitValue();
}
int exitCode = gpgProcess.exitValue();
this.setExitCode(exitCode);
stdOut.close();
stdErr.close();
if(exitCode != 0) {
throw new RuntimeException("Pgp Exception: " + gpgResult);
}
return gpgResult;
}
The readStream method is used to read my stream text.
private String readStream(BufferedReader reader) throws IOException {
StringBuilder result = new StringBuilder();
try {
while(reader.ready()) {
result.append(reader.readLine());
if(reader.ready()) {
result.append("\n");
}
}
} catch(IOException ioe) {
System.err.println("Error while reading the stream: " + ioe.getMessage());
throw ioe;
}
return result.toString();
}

No, that is not the correct way to do it.
First, on some systems, your code will be stuck on the gpgProcess.waitFor() call forever, because the process cannot finish until its standard out and standard error have been fully read and consumed.
Second, you are not using the ready() method of Reader correctly. The documentation states that the method returns true only if reading a character is guaranteed not to block. Returning false does not mean that the end of the stream has been reached; it just means the next read might block (meaning, it might not return immediately).
The only ways to know when you have reached the end of a Reader’s data stream are:
check whether any of its read methods return a negative number
check whether the readLine method of BufferedReader returns null
So your readStream method should look like this:
String line;
while ((line = reader.readLine()) != null) {
result.append(line).append("\n");
}
As of Java 8, you can make it even shorter:
return reader.lines().collect(Collectors.joining("\n"));
Similarly, you should not be calling stdErr.ready() or stdOut.ready(). Either or both methods might or might not return true, even when there are no characters available; the only guarantee for the ready() method is that returning true means the next read will not block. It is possible for ready() to return true even at the end of the character stream, when the next read would immediately return -1, as long as that read does not block.
In summary, don't use ready() at all. Consume all of both streams, and check whether the error stream is empty:
String output = readStream(stdErr);
if (output.isEmpty()) {
String output = readStream(stdOut);
}
gpgResult = "Exit code: " + gpgProcess.exitValue() + "\n" + output;
That would address the case your question appears to present: Either the Process produces standard error and no lines on standard output, or the other way around. However, this will not properly handle Processes in general.
For the general case, the easiest solution is to have the process merge its standard error with standard output using redirectErrorStream, so there is only one stream to consume:
processBuilder.redirectErrorStream(true);
Process gpgProcess = processBuilder.start();
The verifyExecution method could then contain:
String output;
try (BufferedReader stdOut = new BufferedReader(new InputStreamReader(gpgProcess.getInputStream()))) {
output = readStream(stdOut);
}
if (output.isEmpty()) {
gpgResult = "Exit code: " + gpgProcess.waitFor();
} else {
gpgResult = "Exit code: " + gpgProcess.waitFor() + "\n" + output;
}
If you absolutely must have separate standard error and standard output, you need at least one background thread. I find an ExecutorService makes passing a value from a background thread easier:
ExecutorService background = Executors.newSingleThreadExecutor();
Future<String> stdOutReader = background.submit(() -> readStream(stdOut));
String output = readStream(stdErr);
if (output.isEmpty()) {
output = stdOutReader.get();
}
background.shutdown();
if (output.isEmpty()) {
gpgResult = "Exit code: " + gpgProcess.waitFor();
} else {
gpgResult = "Exit code: " + gpgProcess.waitFor() + "\n" + output;
}
Finally, you should not catch and re-throw IOException just to print it out. Whatever code calls verifyExecution will have to catch IOException anyway; it is that code’s job to print, log, or otherwise handle the IOException. Intercepting it like that will probably result in its being printed twice.

There's no reliable way to tell whether an stream has data available without a call to read()—but that call will block if there are no data available. Methods like available() and ready() aren't reliable, because they can give false negatives; they can report that no data are available, even when there are.
A general-purpose facility that will work with any process requires a separate thread to consume each InputStream. This is because, in general, processes could interleave output to stdout and stderr, and unblocking one could cause the other to block, and so on. The process might write partial standard output, then block on a write to standard error. If your master process uses just one thread, it will hang, regardless which stream it reads first. Independent threads consuming both streams will make sure the process runs smoothly.
If you are running a specific process, and you can guarantee it has certain output in every case, you could take some shortcuts… keeping in mind that, "Short cuts make long delays."

java println blocked after communicating with exe if i do not close the stream

I am currently interfacing a GUI to an UCI chess engine. For this purpose i am creating the engine process using:
try {
process = Runtime.getRuntime().exec(enginePath);
} catch (IOException e) {
System.err.println("ENGINE NOT FOUND");
e.printStackTrace();
}
and i am sure that i am able to open the engine.
When engine is opened, there is no significant stream outputted from engine. In order to initiate communication, I have to send specific commands to the engine. The engine will respond then... Therefore it is working in a command/response approach(not immediately streaming data when opened or talking without spoken to). In order to communicate i have a send message block. In this block write a message to the engine using its outputstream and get input using its standart input stream as in the following send method:
private String sendCommand(String command) {
stdin = new PrintWriter(new OutputStreamWriter(process.getOutputStream()));
BufferedReader inputReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
BufferedReader errReader = new BufferedReader(newInputStreamReader(process.getErrorStream()));
String answer = "";
stdin.println(command);
stdin.flush();
stdin.close();
try {
String line = "";
while ((line = inputReader.readLine()) != null) {
System.out.println(line);
answer = line;
}
inputReader.close();
} catch (IOException ioe) {
System.err.println("READ ERROR");
ioe.printStackTrace();
}
try {
String line = "";
while ((line = errReader.readLine()) != null) {
System.err.println(line);
}
errReader.close();
} catch (IOException ioe) {
System.err.println("READ ERROR");
ioe.printStackTrace();
}
return answer;
}
By the way the whole engine is working in a single seperate thread other than the main GUI thread. Therefore no overloading for the GUI. However i didnot seperate the stdin stderr and output methods in individual threads since i suppose there will be only stdin and the one that i will write to the engine. For the time being i assume they will not collide.
With these codes i am able to communicate with the engine and i can see the outputs in the console. However due to stdin.close(); i can only use this method once (In the seconds time i have a STREAM CLOSED error). The chess engine is needed to be communicated in command/response approach many times when opened, without restarting the exe in each time. The problem is that if i remove the line : stdin.close(); yes the communication continues, but my console is blocked by this communication. I.e. i cannot println to the console for debugging purposes anymore. Which is very critical because my main debugging weapon is system.out.println. If i do not remove stdin.close(); i have to restart executable each time i want to send message and i do not want that. **The strange part is that i can see the messages coming from the engine in my console due to "System.out.println(line);" line in the send method; however i cannot print anything on the console once the code exits the method. **
EDIT: Actually after this block:
while ((line = inputReader.readLine()) != null) {
System.out.println(line);
answer = line;
}
System.out.println starts not to work.
EDIT: The problem seems not to be with println but any statement after while loop.
Why statements after while loop is not getting executed?
according to this, it seems that since the stream is never closed, "while loop" is stuck(?). Actually when printing inside while loop, at some point(after a stream is finished), the prints stop, which must indicate while loop is finished. Therefore if it is finished, it should continue on the next statements, shouldn't it? Anyway, the messages from the engine had a set of strings at end of each stream; therefore when i encounter one of them i am breaking the while loop.
Note: it is also interesting that : accumulating data as: "receivedString += line"; doesnot work,i.e. data is not accumulated. In order to fix it i luckily made it "receivedString = receivedString + line + "\n""; and it worked.. I dont know why.

You're reading the input until end of stream, so it won't stop reading until end of stream occurs. End of stream means that the peer has closed the connection in this case. That won't happen for a process's output or error stream until the process exits. You'll have to find some other loop termination condition, or else consume the stdout and stderr in separate threads.

Can't detect disconnect without extra readLine() loop

I am developing a program that uses sockets and currently I have a function in my code that checks for a heartbeat from the client every second.
private void userLoop() { // checks for incoming data from client
Timer t = new Timer();
t.schedule(new TimerTask() {
#Override
public void run() {
try {
socketIn.read(); // check for heartbeat from client
String userInput;
while ((userInput = br.readLine()) != null) {
}
} catch (Exception e) {
ControlPanel.model.removeElement(getUsername());
ControlPanel.append(getUsername() + " has disconnected.");
}
}
}, 1000);
}
When a client closes the game via the X button, shutting off their computer, logging out, whatever it may be, I get the message "'username' has disconnected". This is exactly what I want, however, it only works with the while loop in the code. The while loop essentially does nothing and I have no idea why it doesn't work with out.
If I remove the while loop and I disconnect using my client nothing gets printed out server sided.
String userInput;
while ((userInput = br.readLine()) != null) {
}
The above is essentially the dead code that does nothing but without it my program doesn't work the way it should..
Why is the code needed and how can I remove it and still make my program work correctly?

In this case, your while loop is essentially stalling your program until you no longer receive an input string. It's not dead code; it is just your way of installing a wait.
Otherwise, based on my understanding in the Timer class, it only waits one second, which might be too short of a timespan for what you're waiting to capture.

I fixed my problem by changing everything in the try block with
br.readLine();

There's a saying I've heard about exception handling: "Exceptions should only be used for exceptional situations." A client disconnecting from a server is not exceptional.
Now that I have that off my chest, let's move on. According to this other question,
socket.getInputSteam.read() does not throw when I close the socket from the client
it sounds like the read call won't throw if you're closing things properly on the client side.

The problem is that when the remote socket is closed, read() does not throw an Exception, it just returns -1 to signal the end of the stream.
The following should work without needing to call readLine():
try {
int ret = socketIn.read(); // check for heartbeat from client
if (ret == -1) {
// Remote side closed gracefully
clientDisconnected();
}
} catch (SocketTimeoutException e) {
// Timeout -- handle as required
handleTimeout();
} catch (IOException e) {
// Connection lost due to I/O error
clientDisconnected()
}

Problems with InputStream

Following is a part of the code snippet that I will be using for my project.
public String fetchFromStream()
{
try
{
int charVal;
StringBuffer sb = new StringBuffer();
while((charVal = inputStream.read()) > 0) {
sb.append((char)charVal);
}
return sb.toString();
} catch (Exception e)
{
m_log.error("readUntil(..) : " + e.getMessage());
return null;
} finally {
System.out.println("<<<<<<<<<<<<<<<<<<<<<< Called >>>>>>>>>>>>>>>>>>>>>>>>>>>");
}
}
Initially the while loop start working pretty fine. But after the probable last character is read from the stream I was expecting to get -1 return value. But this is where my problem starts. The code gets hanged, even the finally block is not executed.
I was debugging this code in Eclipse to see what is actually happening during the run-time. I set a pointer (debug) inside the while loop and was constantly monitoring the StringBuffer getting populated with char values one by one. But suddenly while checking the condition inside the while loop, the debugging control is getting lost and this is the point where the code goes to hangup state !! No exception is thrown as well !!
What is happening here ?
Edit::
This is how I'm getting my InputStream. Basically I'm using Apache Commons Net for Telnet.
private TelnetClient getTelnetSession(String hostname, int port)
{
TelnetClient tc = new TelnetClient();
try
{
tc.connect(hostname, port != 0 ? port : 23);
//These are instance variables
inputStream = tc.getInputStream();
outputStream = new PrintStream(tc.getOutputStream());
//More codes...
return tc;
} catch (SocketException se)
{
m_log.error("getTelnetSession(..) : " + se.getMessage());
return null;
} catch (IOException ioe)
{
m_log.error("getTelnetSession(..) : " + ioe.getMessage());
return null;
} catch (Exception e)
{
m_log.error("getTelnetSession(..) : " + e.getMessage());
return null;
}
}

Look at the JavaDocs:
Reads the next byte of data from the input stream. The value byte is returned as an int in the range 0 to 255. If no byte is available because the end of the stream has been reached, the value -1 is returned. This method blocks until input data is available, the end of the stream is detected, or an exception is thrown.
In simple turns: if your stream ended (e.g. end of file), read() returns -1 immediately. However if the stream is still open but JVM is waiting for data (slow disk, socket connection), read() will block (not really hung).
Where are you getting the stream from? Check out the available() - but please do not call it in a loop exhausting CPU.
Finally: casting int/byte to char will only work for ASCII characters, consider using Reader on top of InputStream.

read the docs
read() will wait until there is more data on the InputStream if the InputStream is not closed.
I suspect you are doing this with sockets? This is the most common area where this comes up.
"Reads the next byte of data from the input stream. The value byte is returned as an int in the range 0 to 255. If no byte is available because the end of the stream has been reached, the value -1 is returned. This method blocks until input data is available, the end of the stream is detected, or an exception is thrown"

I have the same issue with the Apache Commons on Android ...
the read() command on the inputstream hangs forever for some reason. And no, it is not just blocking "until data is available" ...
my debugging information shows that there are several 100 chars available() ... yet it just randomly blocks at some read. However, whenever I send something to the telnet server the block is suddenly released and it will continue reading for several chars until it suddenly stops/blocks again at some arbitrary point!
I believe there is some bug within the Apache Commons library! This is really annoying because there isn't a lot that can be done ... no timeout for the read command or anything else ...
EDIT: I was able to get around it ... by setting the TelNetClient.setReaderThread(false) ... obviously there is a bug within the Library that exists as long as a thread handles the input data ... when dispabled it works just fine for me!

Java TelnetClient hangs at “press any key to continue”

I have a Java program that runs on Linux and telnets into a remote server using org.apache.commons.net.telnet.TelnetClient and performs a few commands. The problem is that it hangs intermittently when it gets to an output display that asks the users to “press any key to continue…” The program hangs on this about 1 out of every 10 tims it runs and out of the 7 servers I run it on only 3 of the servers have problems. Also, when I run the same program on a windows box it works all the time.
I was wondering if anyone else has encountered a problem like this?
On a test server I can get it to hang every time to test with. I have tried to send in other commands that won’t cause it to hang but no luck. I have tried all the carrage return, line feed, adding a character and putting in a line feed. Nothing seems to make it client continue.
Forgot to mention that flushing the buffer what the first thing I thought of. I put the flush command anywhere I thought it might hlep.
I will also mention that when I run it and watch the output from the write line it does find the "press any key" and keeps going but hangs the terminal does not continue.
CODE WHERE I MAKE THE CALL:
readUntil("X) Exit (no report)");
write("C", false);
out.flush();
readUntil("continue....");
// write this for all servers.
write("", true);
out.flush();
readUntil("X) Exit");
write("X", false);
/*
* This method is used to read the command line until the pattern that was
* passed in is found.
*/
public String readUntil(String pattern) throws Exception {
try {
String tempString;
char lastChar = pattern.charAt(pattern.length() - 1);
StringBuffer sb = new StringBuffer();
//boolean found = false;
char ch = (char) in.read();
while (true)
{
// NOTE: Turn line below on to watch the program perform the telnet
System.out.print(ch);
sb.append(ch);
tempString = sb.toString();
if (ch == lastChar) {
if (tempString.endsWith(pattern))
{
// log to file
logFileWriter.write(tempString);
logFileWriter.flush();
return tempString;
}
}
ch = (char) in.read();
}
}
catch (Exception e) {
e.printStackTrace();
throw e;
}
}
/*
* writes the String passed in to the command line.
* boolean userWriteln: true - use the return key after the command, false - just type the
* command with NO enter key
*/
public void write(String value, boolean useWriteln)
{
System.out.println("WRITTING '" + value + "'");
try {
if (useWriteln)
{
out.println(value);
}
else
{
out.print(value);
}
out.flush();
System.out.println(value);
}
catch (Exception e) {
e.printStackTrace();
}
}
StackTrace: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at java.io.PushbackInputStream.read(PushbackInputStream.java:122)
at org.apache.commons.net.io.FromNetASCIIInputStream.__read(FromNetASCIIInputStream.java:77)
at org.apache.commons.net.io.FromNetASCIIInputStream.read(FromNetASCIIInputStream.java:175)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at org.apache.commons.net.telnet.TelnetInputStream.__read(TelnetInputStream.java:122)
at org.apache.commons.net.telnet.TelnetInputStream.run(TelnetInputStream.java:564)
at java.lang.Thread.run(Thread.java:619)
WHERE IT HANGS:
english 1 6000 4462 26 % 13826 11056 20 %
Calls answered since Thu Jun 4, 2009 3:11 am: 41245
Press any key to continue....

There might be several reasons:
You're not flushing your output (the input of the remote command), so the "any key" is never sent.
The program tries to send you some data and you never read your input (the output of the remote command). Note that you must do this in a second thread since the I/O usually happens "at the same time" and one side will block if you don't handle the other side timely enough.
Maybe you experience a problem because the app turns the terminal into "RAW mode". But flushing your output should fix that :/