We Keep Coding

Capturing stdout from java.lang.reflect method invocation

I have an application that, among other things, runs Java methods via java.lang.reflect. It normally functions as normal; however, a user used it with one of their JARs, and it broke somewhat.
As you can see in the below code, I attempt to capture both stdout and stdin from the method. However, when the method is invoked, only the first line of what the method streams to stdout is actually captured.
Here's the relevant code. If you need to see more of the code, let me know, and I'll add some more:
String retVal = "";
ByteArrayOutputStream out = new ByteArrayOutputStream();
ByteArrayOutputStream err = new ByteArrayOutputStream();
PrintStream origOut = System.out;
PrintStream origErr = System.err;
System.setOut(new PrintStream(out));
System.setErr(new PrintStream(err));
Exception myException = null;
try {
Object myRetVal = null;
myRetVal = m.invoke(obj, convertedMethodArguments);
if (myRetVal != null)
retVal = myRetVal.toString();
} catch (Exception e) {
myException = e;
}
returnObj.addProperty("stdout", out.toString());
returnObj.addProperty("stderr", err.toString());
returnObj.addProperty("rv", retVal);
returnObj.addProperty("rt", m.getReturnType().toString());
if (myException != null && myException.getCause() != null)
returnObj.addProperty("exception", myException.getCause().toString());
else
returnObj.addProperty("exception", "");
System.setOut(origOut);
System.setErr(origErr);
System.out.print(new Gson().toJson(returnObj));
// TODO: remove, debug purposes only
// Should use normal stdout
try {
System.out.println();
m.invoke(obj, convertedMethodArguments);
} catch (Exception e) {
System.out.println(e.toString());
}
When I execute the above code, it only prints out the first line of stdout. However, at the bottom of the code block, I invoke the method again, but this time without any redirection, and I get all of the stdout from the method.
Any help would be greatly appreciated!
EDIT #1: OK, get this. For fun, I commented-out the two lines where I redirect the default System streams (e.g. System.setOut and System.setErr). With these gone, I now expect all stdout to be written to the console directly when I run the app.
I added a message (e.g. System.out.println("Testing...");) at the very end of my code, so that it's the last thing that is executed. When I test the app, I get the first line of stdout, followed by my testing message, and THEN the rest of the stdout.
I have no clue what's going on here.
EDIT #2: Per #Titus's suggestion, I looked into whether or not the method I'm invoking is spinning off its own threads. Turns out, it is. Two threads are created, AWT-AppKit and AWT-Shutdown. The former thread seems to stay in RUNNABLE state, whereas the latter thread stays in the TIMED_WAITING state.
Over time, the AWT-Shutdown thread goes away, but the other one stays alive in its RUNNABLE state. Once my application exits, I believe the method I'm invoking also exits, and at that point the extra messages are displayed to the screen (which explains why I can't capture this bit of STDOUT).
What I don't understand is why this method won't terminate within my application.

Try to flush the streams after you call the method.
Here is an example:
PrintStream outPR = new PrintStream(out);
System.setOut(outPR);
....
outPR.flush();
returnObj.addProperty("stdout", out.toString());
You can even do this:
System.setOut(new PrintStream(out, true));
....
System.out.println();
returnObj.addProperty("stdout", out.toString());
The PrintStream is automatically flushed (if you use the constructor that I've used) when a \n (new line) is written to it.
Based on the edits to your question, it is possible that the method you're calling is creating new threads which means that it is possible that this new threads print to the console after the method returns.
If that is the case, you'll have to wait until this threads finish in order to get all the output.

System.out.print output to console seen immediately. So PrintStream flushes after each print, not only println?

From the PrintStream documentation:
Optionally, a PrintStream can be created so as to flush automatically;
this means that the flush method is automatically invoked after a byte
array is written, one of the println methods is invoked, or a newline
character or byte ('\n') is written.
Then given code
System.out.print("hi"); // gives console output: hi
System.out.print(7); // gives console output: 7
// prevents flushing when stream wiil be closed at app shutdown
for (;;) {
}
Why then I see output to my console? Nothing shall be written to console (PrintStream instance from System.out), because nothing shall be flushed so far!
This didn't answer this.
I guess, the answer is in the source code (private utility method BufferedWriter.flushBuffer()), but I don't understand the comment to code: "Flushes the output buffer to the underlying character stream, without flushing the stream itself": if PrintStream (which is tied to console output), which is "stream itself" is not flushed, output to console shall not be refreshed!...
Source for PrintStream.print(String):
private void write(String s) {
try {
synchronized (this) {
ensureOpen();
textOut.write(s);
textOut.flushBuffer();
charOut.flushBuffer();
if (autoFlush && (s.indexOf('\n') >= 0))
out.flush();
}
}
catch (InterruptedIOException x) {
Thread.currentThread().interrupt();
}
catch (IOException x) {
trouble = true;
}
}
Source for BufferedWriter.flushBuffer():
/**
* Flushes the output buffer to the underlying character stream, without
* flushing the stream itself. This method is non-private only so that it
* may be invoked by PrintStream.
*/
void flushBuffer() throws IOException {
synchronized (lock) {
ensureOpen();
if (nextChar == 0)
return;
out.write(cb, 0, nextChar);
nextChar = 0;
}
}
More details are also given here. It is very complicated, but seems like at some stage BufferedWriter is given to PrintStream constructor.

I went step by step using debugger and this is what I found:
String s is displayed in the console after 527th line, so it's before line 528 in which the check of having \n is done.
In charOut.flushBuffer() deep inside, there is the following method called:
In which, the check about \n is missing.
The flow is as it follows:
System.out#print(String s) calls PrintStream#print(String s).
PrintStream#print(String s) calls PrintStream#write(String s).
PrintStream#write(String s) calls OutputSteamWriter#flushBuffer().
OutputStreamWriter#flushBuffer() calls StreamEncoder#flushBuffer().
StreamEncoder#flushBuffer() calls StreamEncoder#implFlushBuffer().
StreamEncoder#implFlushBuffer() calls StreamEncoder#writeBytes().
StreamEncoder#writeBytes() calls PrintStream#write(byte buf[], int off, int len) which flushes the buffor if(autoFlush).
The most important snippets are above. The BufferedWriter seems not to be called in this flow.

https://bugs.openjdk.java.net/browse/JDK-8025883 describes this bug.
This bit me in a program that reads and parses a binary file, doing a lot of System.out.printf() calls, which took way longer that it should.
What I ended up doing was writing a helper class that violates the contract of Streams by not honoring every flush request:
class ForceBufferedOutputStream extends OutputStream {
OutputStream out;
byte[] buffer;
int buflen;
boolean haveNewline;
private static final int bufsize=16384;
public ForceBufferedOutputStream(OutputStream out) {
this.out=out;
this.buffer=new byte[bufsize];
this.buflen=0;
this.haveNewline=false;
}
#Override
public void flush() throws IOException {
if (this.haveNewline || this.buflen==bufsize) {
out.write(buffer, 0, buflen);
out.flush();
this.buflen=0;
this.haveNewline=false;
}
}
#Override
public void close() throws IOException {
out.close();
}
#Override
public void write(int b) throws IOException {
buffer[buflen++]=(byte)b;
if (b=='\n')
this.haveNewline=true;
if (buflen==bufsize)
this.flush();
}
}
then using a new PrintStream(new ForceBufferedOutputStream(System.out)) instead of System.out.
I consider this a horrible piece of software - as said, it violates the contract that flush() needs to make sure everything is written, and it could optimize array write calls. But in my case, runtime was cut from 17 minutes to 3:45, so if you need a copy/paste that speeds up a quick and dirty type of program, I hope it helps somewhat.

System.out .write() executes but doesn't print

public class BRRead {
public static void main(String[] args) {
int b;
b='A';
System.out .write(b);
System.out .write('\n');
}
}
So when i execute the above program i get the expected output - A
but when comment out the last line System.out.write('\n'); the program executes but doesn't print the output - A.
Can any one explain what's exactly happening here?
public class BRRead {
public static void main(String[] args) {
int b;
b='A';
System.out .write(b);
//System.out .write('\n');
}
}

public void write(int b)
Writes the specified byte to this stream. If the byte is a newline and
automatic flushing is enabled then the flush method will be invoked.
In your example, flush() is not called automatically, if you call it explicitly the character will be printed.

From here
Basically, Java doesn't guarantee that output is actually sent to a
file, socket, the screen, or any other output device until you call
flush() on your OutputStream or Writer, or until it is closed. The
OutputStream or Writer may buffer the output, meaning that it will be
saved up and sent in larger chunks for efficiency. You can't really
predict how this will be done.
PrintWriter.println() will automatically call flush(), by the way,
although that doesn't matter here. But in any case, the general rule
is that if you want output to appear somewhere immediately, and you're
not using PrintWriter.println() (or PrintStream.println()), then call
flush() yourself.

Call System.out.flush() at the end of your code

From the write(int bytevalue) method body, the arg seems buffered/stored temporarily and not actually written until you write a new line.

Does FileChannel.close() close the underlying stream?

Like the title; Does closing a FileChannel close the underlying file stream?
From the AbstractInterruptibleChannel.close() API docs you can read:
Closes this channel.
If the channel has already been closed then this method returns
immediately. Otherwise it marks the channel as closed and then invokes
the implCloseChannel method in order to complete the close operation.
Which invokes AbstractInterruptibleChannel.implCloseChannel:
Closes this channel.
This method is invoked by the close method in order to perform the
actual work of closing the channel. This method is only invoked if the
channel has not yet been closed, and it is never invoked more than
once.
An implementation of this method must arrange for any other thread
that is blocked in an I/O operation upon this channel to return
immediately, either by throwing an exception or by returning normally.
And that doesn't say anything about the stream. So in fact, when I do:
public static void copyFile(File from, File to)
throws IOException, FileNotFoundException {
FileChannel sc = null;
FileChannel dc = null;
try {
to.createNewFile();
sc = new FileInputStream(from).getChannel();
dc = new FileOutputStream(to).getChannel();
long pos = 0;
long total = sc.size();
while (pos < total)
pos += dc.transferFrom(sc, pos, total - pos);
} finally {
if (sc != null)
sc.close();
if (dc != null)
dc.close();
}
}
...I leave the streams open?

The answer is 'yes' but there's nothing in the Javadoc that actually says so. The reason is that FileChannel itself is an abstract class, and its concrete implementation provides the implCloseChannel() method, which closes the underlying FD. However due to that architecture and the fact that implCloseChannel() is protected, this doesn't get documented.

How to determine the exact state of a BufferedReader?

I have a BufferedReader (generated by new BufferedReader(new InputStreamReader(process.getInputStream()))). I'm quite new to the concept of a BufferedReader but as I see it, it has three states:
A line is waiting to be read; calling bufferedReader.readLine will return this string instantly.
The stream is open, but there is no line waiting to be read; calling bufferedReader.readLine will hang the thread until a line becomes available.
The stream is closed; calling bufferedReader.readLine will return null.
Now I want to determine the state of the BufferedReader, so that I can determine whether I can safely read from it without hanging my application. The underlying process (see above) is notoriously unreliable and so might have hung; in this case, I don't want my host application to hang. Therefore I'm implementing a kind of timeout. I tried to do this first with threading but it got horribly complicated.
Calling BufferedReader.ready() will not distinguish between cases (2) and (3) above. In other words, if ready() returns false, it might be that the stream just closed (in other words, my underlying process closed gracefully) or it might be that the underlying process hung.
So my question is: how do I determine which of these three states my BufferedReader is in without actually calling readLine? Unfortunately I can't just call readLine to check this, as it opens my app up to a hang.
I am using JDK version 1.5.

There is a state where some data may be in the buffer, but not necessarily enough to fill a line. In this case, ready() would return true, but calling readLine() would block.
You should easily be able to build your own ready() and readLine() methods. Your ready() would actually try to build up a line, and only when it has done so successfully would it return true. Then your readLine() could return the fully-formed line.

Finally I found a solution to this. Most of the answers here rely on threads, but as I specified earlier, I am looking for a solution which doesn't require threads. However, my basis was the process. What I found was that processes seem to exit if both the output (called "input") and error streams are empty and closed. This makes sense if you think about it.
So I just polled the output and error streams and also tried to determine if the process had exited or not. Below is a rough copy of my solution.
public String readLineWithTimeout(Process process, long timeout) throws IOException, TimeoutException {
BufferedReader output = new BufferedReader(new InputStreamReader(process.getInputStream()));
BufferedReader error = new BufferedReader(new InputStreamReader(process.getErrorStream()));
boolean finished = false;
long startTime = 0;
while (!finished) {
if (output.ready()) {
return output.readLine();
} else if (error.ready()) {
error.readLine();
} else {
try {
process.exitValue();
return null;
} catch (IllegalThreadStateException ex) {
//Expected behaviour
}
}
if (startTime == 0) {
startTime = System.currentTimeMills();
} else if (System.currentTimeMillis() > startTime + timeout) {
throw new TimeoutException();
}
}
}

This is a pretty fundamental issue with java's blocking I/O API.
I suspect you're going to want to pick one of:
(1) Re-visit the idea of using threading. This doesn't have to be complicated, done properly, and it would let your code escape a blocked I/O read fairly gracefully, for example:
final BufferedReader reader = ...
ExecutorService executor = // create an executor here, using the Executors factory class.
Callable<String> task = new Callable<String> {
public String call() throws IOException {
return reader.readLine();
}
};
Future<String> futureResult = executor.submit(task);
String line = futureResult.get(timeout); // throws a TimeoutException if the read doesn't return in time
(2) Use java.nio instead of java.io. This is a more complicated API, but it has non-blocking semantics.

Have you confirmed by experiment your assertion that ready() will return false even if the underlying stream is at end of file? Because I would not expect that assertion to be correct (although I haven't done the experiment).

You could use InputStream.available() to see if there is new output from the process. This should work the way you want it if the process outputs only full lines, but it's not really reliable.
A more reliable approach to the problem would be to have a seperate thread dedicated to reading from the process and pushing every line it reads to some queue or consumer.

In general, you have to implement this with multiple threads. There are special cases, like reading from a socket, where the underlying stream has a timeout facility built-in.
However, it shouldn't be horribly complicated to do this with multiple threads. This is a pattern I use:
private static final ExecutorService worker =
Executors.newSingleThreadExecutor();
private static class Timeout implements Callable<Void> {
private final Closeable target;
private Timeout(Closeable target) {
this.target = target;
}
public Void call() throws Exception {
target.close();
return null;
}
}
...
InputStream stream = process.getInputStream();
Future<?> task = worker.schedule(new Timeout(stream), 5, TimeUnit.SECONDS);
/* Use the stream as you wish. If it hangs for more than 5 seconds,
the underlying stream is closed, raising an IOException here. */
...
/* If you get here without timing out, cancel the asynchronous timeout
and close the stream explicitly. */
if(task.cancel(false))
stream.close();

You could make your own wrapper around InputStream or InputStreamReader that works on a byte-by-byte level, for which ready() returns accurate values.
Your other options are threading which could be done simply (look into some of the concurrent data structures Java offers) and NIO, which is very complex and probably overkill.

If you just want the timeout then the other methods here are possibly better. If you want a non-blocking buffered reader, here's how I would do it, with threads: (please note I haven't tested this and at the very least it needs some exception handling added)
public class MyReader implements Runnable {
private final BufferedReader reader;
private ConcurrentLinkedQueue<String> queue = new ConcurrentLinkedQueue<String>();
private boolean closed = false;
public MyReader(BufferedReader reader) {
this.reader = reader;
}
public void run() {
String line;
while((line = reader.readLine()) != null) {
queue.add(line);
}
closed = true;
}
// Returns true iff there is at least one line on the queue
public boolean ready() {
return(queue.peek() != null);
}
// Returns true if the underlying connection has closed
// Note that there may still be data on the queue!
public boolean isClosed() {
return closed;
}
// Get next line
// Returns null if there is none
// Never blocks
public String readLine() {
return(queue.poll());
}
}
Here's how to use it:
BufferedReader b; // Initialise however you normally do
MyReader reader = new MyReader(b);
new Thread(reader).start();
// True if there is data to be read regardless of connection state
reader.ready();
// True if the connection is closed
reader.closed();
// Gets the next line, never blocks
// Returns null if there is no data
// This doesn't necessarily mean the connection is closed, it might be waiting!
String line = reader.readLine(); // Gets the next line
There are four possible states:
Connection is open, no data is available
Connection is open, data is available
Connection is closed, data is available
Connection is closed, no data is available
You can distinguish between them with the isClosed() and ready() methods.