We Keep Coding

BufferedReader.readLine blocks my program but BufferedReader.read() reads properly

I have a snippet as follows:
Process proc = Runtime.getRuntime().exec(command);
BufferedReader br = new BufferedReader(new InputStreamReader(proc.getErrorStream()));
String line = br.readLine();
Now in the above code I am sure that process will always have on line input, so I did not use any kind of while loop or any null check. The problem is readLine blocks. The one reason I was aware of is, the stream having no data to read and hence readLine keeps waiting. To check this, I removed readLine and used read() function as follows:
Process proc = Runtime.getRuntime().exec( command );
BufferedReader br = new BufferedReader(new InputStreamReader(proc.getErrorStream()));
int a;
while((a=br.read())!=-1){
char ch = (char) a;
if(ch == '\n')
System.out.print("New line "+ch);
if(ch == '\r')
System.out.print("Carriage return "+ch);
System.out.print(ch);
}
To my surprise this code worked and printed the messags New line and Carriage return. Now I am wondering why did the readLine block? The data is available it is terminated by newline. What else could be the reason??
Note: The above worked once in a while! Maybe once out of 15times.
Note: I tried using ProcessBuilder too, but same behaviour.
UPDATE:
So I switched to ProcessBuilder and then I redirected the errorStream and now I get both input stream and error stream at once when I do process.getInputStream and this works fine. Below is the snippet.
ProcessBuilder pb = new ProcessBuilder(command.split(" "));
pb..redirectErrorStream(true);
Process proc = pb.start();
BufferedReader br = new BufferedReader(new InputStreamReader(proc.getInputStream()));
String line = br.readLine();
//Now I get both input and error stream.
I would like to differentiate my error stream from input stream but with this method they are all jumbled up! Any ideas on this?

You can use threads, in order to avoid it.
Like One slave thread which will be responsible for reading. This will not halt your progress of program.

I think the problem is not that the standard error is blocking, but that the standard output is blocking causing the application you are calling to block.
Standard output is normally buffered. If the process you are calling writes less to standard output than the buffer size all is well and it can reach the code that writes to standard error. If the process fills the buffer, its attempt to write to standard output will block and it will never reach the point where it writes to standard error.
This could be why you see it working occasionally - sometimes the standard output does not fill the buffer. It could also be why it works after a long time: eventually the write to standard output times out.
As a demonstration, this simple process always block like you describe on my Windows 8 machine:
public class Proc {
public static void main(String[] args) {
for(int i=0;i<1000;i++) {
System.out.print("More data ");
}
System.out.println();
System.err.println("An error line");
}
}

So to avoid getting the error stream and input stream merged just remove the line
pb.redirectErrorStream(true);
Because as said in the java doc :
if this property is true, then any error output generated by subprocesses subsequently started by this object's start() method will be merged with the standard output, so that both can be read using the Process.getInputStream() method. This makes it easier to correlate error messages with the corresponding output. The initial value is false.
By calling pb.redirectErrorStream(true); you are merging both output.

getErrorStream states the following
Returns the input stream connected to the error output of the subprocess. The stream obtains data piped from the error output of the process represented by this Process object.
If the standard error of the subprocess has been redirected using ProcessBuilder.redirectError or ProcessBuilder.redirectErrorStream then this method will return a null input stream.
and the ReadLine states the following
Reads a line of text. A line is considered to be terminated by any one of a line feed ('\n'), a carriage return ('\r'), or a carriage return followed immediately by a linefeed.
Based on the explanation provided to the API. readline waits indefinitely to get line feed or carriage return where as processbuilder returns NULL that is why readLine terminates for the same.

Error stream blocking when running external command with Java

Working on SEAndroid, I call Setools commands from my Java application.
It works perfectly with small SEAndroid policy and now I need to test my tool with real
SEAndroid policy. But unfortunately, I face a problem with an error stream.
Here my code I used to call external commands :
public static BufferedReader runCommand(final String[] args)
throws IOException {
BufferedReader stdInput = null;
BufferedReader stdError = null;
try {
Process p = Runtime.getRuntime().exec(args);
stdInput = new BufferedReader(new
InputStreamReader(p.getInputStream()));
stdError = new BufferedReader(new
InputStreamReader(p.getErrorStream()));
// read any errors from the attempted command
String s = null;
StringBuilder err = new StringBuilder();
while ((s = stdError.readLine()) != null) {
err.append(s + "\n");
}
if (err.length() != 0) {
throw new IOException(err.toString());
}
return stdInput;
} finally {
if (stdError != null) {
stdError.close();
}
}
}
So, as you can see, I call an external command. Then read the error stream and throw an exception if there is any errors, otherwise I return the InputStream, so I can parse it later.
With a real SEAndroid policy, the error stream seems to block (even if I read a single char) and I can't parse the result of the command. If I close the error stream without reading anything, the application works fine, but I want to handle errors if any.
If I type the command in a console, it works fine too.
In the first case (with small SEAndroid policy), the output of the command is small ( ~350 lines).
In the second case (with a real SEAndroid policy), the output of the command is larger ( >1500 lines).
Is it possible that the size of the output stream influences the error stream? The two streams are two distinctive resources, isn't it?
The fact that I do not read the output stream immediately have an importance?
I fear that its not a "programming" problem but more a system problem...
Any suggestion?
Thanks in advance for your help=)
Edit:
I try to read the output stream before the error stream and it works. But I need to check the error stream before perform any parsing on the output stream, so the problem is still topical.

First, it's probably better to use the newer ProcessBuilder class as opposed to Runtime exec. If you want to go a step further, you can even use Apache commons-exec which takes care of stream handling and other things for you.
Next, as you've discovered, process control is a tricky thing in Java and you've run into one of its tricky issues. From the documentation for java's Process class:
The parent process uses these streams to feed input to and get output from the subprocess. Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.
You need to have something consuming both (Error and Output) streams or you risk deadlock - these should each be read on their own threads. Using something like a StreamGobbler (google it, there are plenty out there) would be a good step, or you can roll your own if you're so inclined. It isn't too hard to get it right but if you're unfamiliar with multithreading you may want to look at someone else's implementation or go the Apache commons-exec route.

The processing of output is so annoying, that I wrote little library called jproc that deals with the problem of consuming stdout and stderr. It can simply filter strings through external programs likes this:
ProcBuilder.filter("x y z","sed" ,"s/y/a/")
It also lets you specify a timeout for the completion and will convert non-zero exit codes into exception.

Why does java.lang.process's readline() behave differently for reading inputstream on different boxes with the same os

I tested this code(below) on several different linux boxes(4+) and it worked fine. However, on one linux box I ran into an issue with readline() hanging for the error inputStream(errorStream). This stream should be empty so I suspected that box was not writing out a line terminator to the errorStream for the error. I changed my code to use read() instead of readline()...but read() also hung.
I tried retrieving the input inputStream first, and that worked and there was no hangs with readline()/read() for the error inputstream. I could not do this since I needed to obtain possible errors first. Appearing to be a deadlock, I was able to resolve this by having each inputstream read from it's own thread. Why did I only see this issue on one box? Is there a kernel setting or some other setting specific to this box that could have caused this?
ProcessBuilder processBuilder = new ProcessBuilder()
try
{
Process processA = null;
synchronized (processBuilder)
{
processBuilder.command("/bin/sh","-c"," . /Home/SomeScript.ksh");
processA = processBuilder.start();
}
inputStream = processA.getInputStream();
reader = new BufferedReader(new InputStreamReader(inputStream));
errorStream = processA.getErrorStream();
errorReader = new BufferedReader(new InputStreamReader(errorStream));
String driverError;
while ((driverError = errorReader.readLine()) != null)
{
//some code
}

Why did I only see this issue on one box?
Most likely because of something in the script that is being run ... and its interactions with its environment (e.g. files, environment variables, etc)
Is there a kernel setting or some other setting specific to this box that could have caused this?
It is possible but unlikely that it is a kernel setting. It might be "something else". Indeed, it has to be "something" outside of the Java application that is to blame, at least in part.
I suggest you do the following temporarily (at least):
ProcessBuilder processBuilder = new ProcessBuilder();
processBuilder.command("/bin/sh","-c"," . /Home/SomeScript.ksh");
processBuilder.redirectErrorStream(true);
processA = processBuilder.start();
inputStream = processA.getInputStream();
reader = new BufferedReader(new InputStreamReader(inputStream));
String line;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
System.out.println("Return code is " + processA.exitValue());
That way you can see what all of the output is.
There should not be a problem if the external process fails to put a newline at the end of the last line. The Java process will see an EOF on the input stream, and the BufferedReader will return what characters it has ... and return null on the next call.
Another possibility is that the external process is blocking because it is trying to read from its standard input.
UPDATE
The redirectErrorStream also resolved the issue, but I need the error stream separate.
OK so if it did (reliably) solve the problem then that (most likely) means that you have to read the external processes stdout and stderr streams in parallel. The simple way to do is to create 2 threads to read and buffer the two streams separately. For example: Capturing stdout when calling Runtime.exec
(Your problem is due to the fact that pipes have a finite buffering capacity. The external problem is most likely alternating between writing stuff to stdout and stderr. If it tries to write to one of the pipes when that pipe is "full", it will block. But if your application is reading all of the other pipe (to EOF) before it reads the blocked pipe, then everything will deadlock. The fact that the external process is stuck in PIPE_W state is more evidence for this explanation.
One possible reason that you are seeing different behaviour on different systems is that the amount of buffering in a pipe is system dependent. But it could also be due to differences in what the external process is doing; e.g. its inputs.)

You are running OS specific commands in a script, any one could be holding the error output. You can avoid this by discarding the errors, but that is unlikely to be a good idea.
I would check the version of the OS are the same and whether you have any significant differences in the command you run in the script. If this doesn't help, take out commands from the script until it starts working. I assume an empty script doesn't do this.

Reading error stream from a process

I am writing a java program to read the error stream from a process . Below is the structure of my code --
ProcessBuilder probuilder = new ProcessBuilder( command );
Process process = probuilder.start();
InputStream error = process.getErrorStream();
InputStreamReader isrerror = new InputStreamReader(error);
BufferedReader bre = new BufferedReader(isrerror);
while ((linee = bre.readLine()) != null) {
System.out.println(linee);
}
The above code works fine if anything is actually written to the error stream of the invoked process. However, if anything is not written to the error stream, then the call to readLine actually hangs indefinitely. However, I want to make my code generic so that it works for all scenarios. How can I modify my code to achieve the same.
Regards,
Dev

readline() is a blocking call. It will block until there's a line to be read (terminated by an end of line character) or the underlying stream is closed (returning EOF).
You need to have logic that is checking BufferedReader.ready() or just using BufferedReader.read() and bailing out if you decide you're waiting long enough (or want to do something else then check again).
Edit to add: That being said, it shouldn't hang "indefinitely" as-is; it should return once the invoked process terminates. By any chance is your invoked process also outputting something to stdout? If that's the case ... you need to be reading from that as well or the buffer will fill and will block the external process which will prevent it from exiting which ... leads to your problem.

This is a late reply, but the issue hasn't really solved and it's on the first page for some searches. I had the same issue, and BufferedReader.ready() would still set up a situation where it would lock.
The following workaround will not work if you need to get a persistent stream. However, if you're just running a program and waiting for it to close, this should be fine.
The workaround I'm using is to call ProcessBuilder.redirectError(File). Then I'd read the file and use that to present the error stream to the user. It worked fine, didn't lock. I did call Process.destroyForcibly() after Process.waitFor() but this is likely unnecessary.
Some pseudocode below:
File thisFile = new File("somefile.ext");
ProcessBuilder pb = new ProcessBuilder(yourStringList);
pb.redirectError(thisFile);
Process p = pb.start();
p.waitFor();
p.destroyForcibly();
ArrayList fileContents = getFileContents(thisFile);
I hope this helps with at least some of your use cases.

Something like this might also work and avoid the blocking behaviour (without requiring to create a File)
InputStream error = process.getErrorStream();
// Read from InputStream
for (int k = 0; k < error.available(); ++k)
System.out.println("Error stream = " + error.read());
From the Javadoc of InputStream.available
Returns an estimate of the number of bytes that can be read (orskipped over) from this input stream without blocking by the nextinvocation of a method for this input stream. The next invocationmight be the same thread or another thread. A single read or skip of thismany bytes will not block, but may read or skip fewer bytes.

The simplest answer would be to simply redirect the error stream to stdout:
process.getErrorStream().transferTo(System.out);

Read output from external process

I am trying to run a .csh script and read it's output into a StringBuffer.
the output sometime returns empty although running the script from console returns some output. the same running flow can sometimes returns output and sometimes not, although nothing is changed in the way the process starts (same script, path , args) and the script isn't changed as well.
I'm not getting any exceptions thrown.
what might cause output now to be read correctly/successfully ?
the code segment is
public static String getOutpoutScript(Process p) {
InputStream outpout = p.getInputStream();
logger.info("Retrived script output stream");
BufferedReader buf = new BufferedReader(new InputStreamReader(outpout));
String line = "";
StringBuffer write = new StringBuffer();
try {
while ((line = buf.readLine()) != null) {
write.append(line);
}
} catch (IOException e) {
// do something
}
return write.toString().trim();
}
beside the fact not closing the streams is not good, could this or something else in the code might prevent output from being read correctly under some circumstances ?
thanks,

If you launch it with ProcessBuilder, you can combine the error stream into the output stream. This way if the program prints to stderr you'll capture this too. Alternatively you could just read both. Additionally, you may not want to use readLine, you could be stuck for awhile if the program does not print end of line character at the end.

Maybe you must replace p.getInputStream() with p.getOutputStream()
Besides this sometimes processes can block waiting on input, so you must read and write asynchronously - one possible solution is to use different threads - e.g. one thread is reading, other is writing and one that is monitoring the process.

If you have an error, this will write to getErrorStream() by default. If you have a problem, I would ensure you are reading this somewhere.
If the buffer for this stream fills, your program will stop, waiting for you to read it.
A simple way around these issues is to use ProcessBuilder.redirectErrorStream(true)