I am using Java to run a program and reading the program's text output into a buffered reader. I then want to print out the results by doing this:
while((line=input.readLine()) != null){
System.out.println(line);
}
For some reason, when doing this the readLine() method does not return at all about halfway through the text, and the program waits indefinitely. Yet when running this program from the command line directly, its output is exactly as expected, just 100 lines or so of simple text. What could cause readLine() to not return halfway through reading simple text?
This may happen if nothing has been written to the stream.
You may often encounter the situation in which readLine() hangs is when you invoke a process that is supposed to write errors to stderr. When the process finishes you may try to read all the errors by doing:
BufferedReader brCleanUp = new BufferedReader(new InputStreamReader(stderr));
while ((line = brCleanUp.readLine()) != null) {
//process the error line
}
brCleanUp.close();
But if nothing at all was written to stderr, the above readLine() would hang as you described.
The way to avoid it is to check if there is any point of reading the stream (I don't know if it's a bug or not, but this worked for me:)
boolean available = stderr.available() >= 1;
if (available) {
BufferedReader brCleanUp = new BufferedReader(new InputStreamReader(stderr));
for( ; available && (line = brCleanUp.readLine()) != null;
available = stderr.available() >= 1) {
System.out.println ("[Stderr] " + line);
}
brCleanUp.close();
}
You need to encode the text document into UTF-8 like this, then use Scanner.
// specify file path
String filePath = "C:/path/path/file-location/file.txt";
// make a scanner with the UTF-8 param
Scanner scanner = new Scanner(new File(filePath),"UTF-8");
// loop through text
while (scanner.hasNextLine()){
System.out.println(scanner.nextLine());
}
There are actually a few ways to encode text like this, but the importance lies within the text encoding itself. Once you have the document in the correct format, the objects and classes you chose to read input are at your own discretion.
What could cause readLine() to not return halfway through reading simple text?
If you are reading from a file in a file system, this is possibly due to a problem with the file system; e.g. a remote file server is not responding.
If you are reading from a socket, either the remote server hasn't (yet) written / flushed the data you are trying to read, or a network problem is stopping the data from getting through.
If you are reading from a pipe (e.g. the output of another program), then the other program hasn't written / flushed the data you are trying to read.
(Another cases are reading from a "device file" or a file in the "/proc" file system, or a loopback file system. In these cases, the possible causes are difficult to enumerate.)
Related
Can someone say that code below is good way to read all file? Maybe the first code block is using the ready method in a wrong way.
try (var br = new BufferedReader(new FileReader("/some-file.txt"))) {
while (br.ready()) {
System.out.println(br.readLine());
}
}
Or maybe is better approach to read file without method ready?
try (var br = new BufferedReader(new FileReader("/some-file.txt"))) {
while (true) {
var line = br.readLine();
if (line == null) break;
System.out.println(line);
}
}
I tested that two blocks and all blocks print all file content, but I've never saw the first way over internet.
Here's the documentation of BufferedReader#ready():
Tells whether this stream is ready to be read. A buffered character stream is ready if the buffer is not empty, or if the underlying character stream is ready.
[...]
Returns:
True if the next read() is guaranteed not to block for input, false otherwise. Note that returning false does not guarantee that the next read will block.
So, this method is about whether or not the next read will block. You're trying to read the whole file in one go, which means you don't really care if the next read will block. Worse, what if the reader is not ready? Your loop will break, you'll close the file, and the code will continue on without having read the whole source.
A typical way to code what you're doing is:
try (var reader = new BufferedReader(new FileReader("/some-file.txt"))) {
String line;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
}
Since readLine() is contracted to return null only when the end of the stream is reached.
Note of caution though. The above did not specify a charset, which means the default charset is used (see documentation for Java 17 and before and Java 18+). You can use FileReader(String,Charset) to specify a charset.
There's also the java.nio.file.* API that you can use to do the same thing. For example:
try (var stream = Files.lines(Path.of("/some-file.txt"))) {
stream.forEachOrdered(System.out::println);
}
The above uses the UTF-8 charset. You can use Files#lines(Path,Charset) to use a specific charset.
I think everyone does it in a different way (you could also read each byte using loops) but there is a way to read the whole file at once without loops
String file = new String(Files.readAllBytes(Paths.get("file")));
You can also use Files.readAllLines depending on what you want to do with data in file.
edit. First comment under your question also shows a better way
My code looks like this:
public List<String> linux(String... commands) throws Exception{
Process p = null;
ProcessBuilder builder = new ProcessBuilder(commands);
p = builder.start();
BufferedReader reader= new BufferedReader(new InputStreamReader(p.getInputStream(), StandardCharsets.UTF_8));
String line;
List<String> lines = new ArrayList<>();
while((line = reader.readLine()) != null) lines.add(line);
return lines;
}
But it didn't return all the outputs. I tried in the actual remote machine, there were about 100+ lines, but I only got 30 lines. Am I missing something ?
The code you have shown us will read all lines written by the process to its standard output. The readLine() call will only return a null when it has reached the end-of-stream for the pipe that it is reading from; i.e. after all reading all lines.
The readLine() call can only see the end-of-stream if the other end of the pipe (i.e. the external process) has closed it. You cannot get null by reading "too quickly". And conversely, if readLine() does return null for a given stream, any / all subsequent calls will return null. There is no point in retrying the readLine() if it returned null.
Also, you shouldn't use isAlive() to decide when to stop reading ... as suggested by someone else. The availability of unread output, and process termination are orthogonal issues.
Given that your code for reading the output is correct (I think so), there are just two possible explanations for the missing output:
Some of the process output was written to the processes standard error stream rather than its standard output.
Solution: You can get the contents of the standard error stream by reading from the stream returned by getErrorStream(). But a simpler alternative is to get ProcessBuilder to redirect the standard error output to standard output.
The external process terminated without flushing its output buffers. That could mean that some lines were never actually output by the process.
Solution: You need to fix the external application. This cannot be remedied from the Java application side.
It sounds like the Charset is not UTF-8 if it errors as soon as 'é' occurs. It's probably Extended ASCII, but if that doesn't work I'd try to figure out what Charset it really is.
It would fail on the entire line because of the BufferedReader.
I have a snippet as follows:
Process proc = Runtime.getRuntime().exec(command);
BufferedReader br = new BufferedReader(new InputStreamReader(proc.getErrorStream()));
String line = br.readLine();
Now in the above code I am sure that process will always have on line input, so I did not use any kind of while loop or any null check. The problem is readLine blocks. The one reason I was aware of is, the stream having no data to read and hence readLine keeps waiting. To check this, I removed readLine and used read() function as follows:
Process proc = Runtime.getRuntime().exec( command );
BufferedReader br = new BufferedReader(new InputStreamReader(proc.getErrorStream()));
int a;
while((a=br.read())!=-1){
char ch = (char) a;
if(ch == '\n')
System.out.print("New line "+ch);
if(ch == '\r')
System.out.print("Carriage return "+ch);
System.out.print(ch);
}
To my surprise this code worked and printed the messags New line and Carriage return. Now I am wondering why did the readLine block? The data is available it is terminated by newline. What else could be the reason??
Note: The above worked once in a while! Maybe once out of 15times.
Note: I tried using ProcessBuilder too, but same behaviour.
UPDATE:
So I switched to ProcessBuilder and then I redirected the errorStream and now I get both input stream and error stream at once when I do process.getInputStream and this works fine. Below is the snippet.
ProcessBuilder pb = new ProcessBuilder(command.split(" "));
pb..redirectErrorStream(true);
Process proc = pb.start();
BufferedReader br = new BufferedReader(new InputStreamReader(proc.getInputStream()));
String line = br.readLine();
//Now I get both input and error stream.
I would like to differentiate my error stream from input stream but with this method they are all jumbled up! Any ideas on this?
You can use threads, in order to avoid it.
Like One slave thread which will be responsible for reading. This will not halt your progress of program.
I think the problem is not that the standard error is blocking, but that the standard output is blocking causing the application you are calling to block.
Standard output is normally buffered. If the process you are calling writes less to standard output than the buffer size all is well and it can reach the code that writes to standard error. If the process fills the buffer, its attempt to write to standard output will block and it will never reach the point where it writes to standard error.
This could be why you see it working occasionally - sometimes the standard output does not fill the buffer. It could also be why it works after a long time: eventually the write to standard output times out.
As a demonstration, this simple process always block like you describe on my Windows 8 machine:
public class Proc {
public static void main(String[] args) {
for(int i=0;i<1000;i++) {
System.out.print("More data ");
}
System.out.println();
System.err.println("An error line");
}
}
So to avoid getting the error stream and input stream merged just remove the line
pb.redirectErrorStream(true);
Because as said in the java doc :
if this property is true, then any error output generated by subprocesses subsequently started by this object's start() method will be merged with the standard output, so that both can be read using the Process.getInputStream() method. This makes it easier to correlate error messages with the corresponding output. The initial value is false.
By calling pb.redirectErrorStream(true); you are merging both output.
getErrorStream states the following
Returns the input stream connected to the error output of the subprocess. The stream obtains data piped from the error output of the process represented by this Process object.
If the standard error of the subprocess has been redirected using ProcessBuilder.redirectError or ProcessBuilder.redirectErrorStream then this method will return a null input stream.
and the ReadLine states the following
Reads a line of text. A line is considered to be terminated by any one of a line feed ('\n'), a carriage return ('\r'), or a carriage return followed immediately by a linefeed.
Based on the explanation provided to the API. readline waits indefinitely to get line feed or carriage return where as processbuilder returns NULL that is why readLine terminates for the same.
Working on SEAndroid, I call Setools commands from my Java application.
It works perfectly with small SEAndroid policy and now I need to test my tool with real
SEAndroid policy. But unfortunately, I face a problem with an error stream.
Here my code I used to call external commands :
public static BufferedReader runCommand(final String[] args)
throws IOException {
BufferedReader stdInput = null;
BufferedReader stdError = null;
try {
Process p = Runtime.getRuntime().exec(args);
stdInput = new BufferedReader(new
InputStreamReader(p.getInputStream()));
stdError = new BufferedReader(new
InputStreamReader(p.getErrorStream()));
// read any errors from the attempted command
String s = null;
StringBuilder err = new StringBuilder();
while ((s = stdError.readLine()) != null) {
err.append(s + "\n");
}
if (err.length() != 0) {
throw new IOException(err.toString());
}
return stdInput;
} finally {
if (stdError != null) {
stdError.close();
}
}
}
So, as you can see, I call an external command. Then read the error stream and throw an exception if there is any errors, otherwise I return the InputStream, so I can parse it later.
With a real SEAndroid policy, the error stream seems to block (even if I read a single char) and I can't parse the result of the command. If I close the error stream without reading anything, the application works fine, but I want to handle errors if any.
If I type the command in a console, it works fine too.
In the first case (with small SEAndroid policy), the output of the command is small ( ~350 lines).
In the second case (with a real SEAndroid policy), the output of the command is larger ( >1500 lines).
Is it possible that the size of the output stream influences the error stream? The two streams are two distinctive resources, isn't it?
The fact that I do not read the output stream immediately have an importance?
I fear that its not a "programming" problem but more a system problem...
Any suggestion?
Thanks in advance for your help=)
Edit:
I try to read the output stream before the error stream and it works. But I need to check the error stream before perform any parsing on the output stream, so the problem is still topical.
First, it's probably better to use the newer ProcessBuilder class as opposed to Runtime exec. If you want to go a step further, you can even use Apache commons-exec which takes care of stream handling and other things for you.
Next, as you've discovered, process control is a tricky thing in Java and you've run into one of its tricky issues. From the documentation for java's Process class:
The parent process uses these streams to feed input to and get output from the subprocess. Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.
You need to have something consuming both (Error and Output) streams or you risk deadlock - these should each be read on their own threads. Using something like a StreamGobbler (google it, there are plenty out there) would be a good step, or you can roll your own if you're so inclined. It isn't too hard to get it right but if you're unfamiliar with multithreading you may want to look at someone else's implementation or go the Apache commons-exec route.
The processing of output is so annoying, that I wrote little library called jproc that deals with the problem of consuming stdout and stderr. It can simply filter strings through external programs likes this:
ProcBuilder.filter("x y z","sed" ,"s/y/a/")
It also lets you specify a timeout for the completion and will convert non-zero exit codes into exception.
I am trying to run a .csh script and read it's output into a StringBuffer.
the output sometime returns empty although running the script from console returns some output. the same running flow can sometimes returns output and sometimes not, although nothing is changed in the way the process starts (same script, path , args) and the script isn't changed as well.
I'm not getting any exceptions thrown.
what might cause output now to be read correctly/successfully ?
the code segment is
public static String getOutpoutScript(Process p) {
InputStream outpout = p.getInputStream();
logger.info("Retrived script output stream");
BufferedReader buf = new BufferedReader(new InputStreamReader(outpout));
String line = "";
StringBuffer write = new StringBuffer();
try {
while ((line = buf.readLine()) != null) {
write.append(line);
}
} catch (IOException e) {
// do something
}
return write.toString().trim();
}
beside the fact not closing the streams is not good, could this or something else in the code might prevent output from being read correctly under some circumstances ?
thanks,
If you launch it with ProcessBuilder, you can combine the error stream into the output stream. This way if the program prints to stderr you'll capture this too. Alternatively you could just read both. Additionally, you may not want to use readLine, you could be stuck for awhile if the program does not print end of line character at the end.
Maybe you must replace p.getInputStream() with p.getOutputStream()
Besides this sometimes processes can block waiting on input, so you must read and write asynchronously - one possible solution is to use different threads - e.g. one thread is reading, other is writing and one that is monitoring the process.
If you have an error, this will write to getErrorStream() by default. If you have a problem, I would ensure you are reading this somewhere.
If the buffer for this stream fills, your program will stop, waiting for you to read it.
A simple way around these issues is to use ProcessBuilder.redirectErrorStream(true)