We Keep Coding

Java Process Builder always giving data in Error Stream [duplicate]

Can anyone clarify me if the below procedure is correct way to handle streams of process without any stream buffer full and blocking problems
I'm invoking external program from java program, I'm using ProcessBuilder to build the process and after I perform
Process gpgProcess = processBuilder.start();
I'm handling the process using a method
String executionResult = verifyExecution(gpgProcess);
and in my method i'm trying to handle the process streams
private String verifyExecution(Process gpgProcess) throws IOException, InterruptedException {
String gpgResult = null;
BufferedReader stdOut = new BufferedReader(new InputStreamReader(gpgProcess.getInputStream()));
BufferedReader stdErr = new BufferedReader(new InputStreamReader(gpgProcess.getErrorStream()));
gpgProcess.waitFor();
if(stdErr.ready()) {
gpgResult = "Exit code: " + gpgProcess.exitValue() + "\n" + readStream(stdErr);
} else if(stdOut.ready()) {
gpgResult = "Exit code: " + gpgProcess.exitValue() + "\n" + readStream(stdOut);
} else {
gpgResult = "Exit code: " + gpgProcess.exitValue();
}
int exitCode = gpgProcess.exitValue();
this.setExitCode(exitCode);
stdOut.close();
stdErr.close();
if(exitCode != 0) {
throw new RuntimeException("Pgp Exception: " + gpgResult);
}
return gpgResult;
}
The readStream method is used to read my stream text.
private String readStream(BufferedReader reader) throws IOException {
StringBuilder result = new StringBuilder();
try {
while(reader.ready()) {
result.append(reader.readLine());
if(reader.ready()) {
result.append("\n");
}
}
} catch(IOException ioe) {
System.err.println("Error while reading the stream: " + ioe.getMessage());
throw ioe;
}
return result.toString();
}

No, that is not the correct way to do it.
First, on some systems, your code will be stuck on the gpgProcess.waitFor() call forever, because the process cannot finish until its standard out and standard error have been fully read and consumed.
Second, you are not using the ready() method of Reader correctly. The documentation states that the method returns true only if reading a character is guaranteed not to block. Returning false does not mean that the end of the stream has been reached; it just means the next read might block (meaning, it might not return immediately).
The only ways to know when you have reached the end of a Reader’s data stream are:
check whether any of its read methods return a negative number
check whether the readLine method of BufferedReader returns null
So your readStream method should look like this:
String line;
while ((line = reader.readLine()) != null) {
result.append(line).append("\n");
}
As of Java 8, you can make it even shorter:
return reader.lines().collect(Collectors.joining("\n"));
Similarly, you should not be calling stdErr.ready() or stdOut.ready(). Either or both methods might or might not return true, even when there are no characters available; the only guarantee for the ready() method is that returning true means the next read will not block. It is possible for ready() to return true even at the end of the character stream, when the next read would immediately return -1, as long as that read does not block.
In summary, don't use ready() at all. Consume all of both streams, and check whether the error stream is empty:
String output = readStream(stdErr);
if (output.isEmpty()) {
String output = readStream(stdOut);
}
gpgResult = "Exit code: " + gpgProcess.exitValue() + "\n" + output;
That would address the case your question appears to present: Either the Process produces standard error and no lines on standard output, or the other way around. However, this will not properly handle Processes in general.
For the general case, the easiest solution is to have the process merge its standard error with standard output using redirectErrorStream, so there is only one stream to consume:
processBuilder.redirectErrorStream(true);
Process gpgProcess = processBuilder.start();
The verifyExecution method could then contain:
String output;
try (BufferedReader stdOut = new BufferedReader(new InputStreamReader(gpgProcess.getInputStream()))) {
output = readStream(stdOut);
}
if (output.isEmpty()) {
gpgResult = "Exit code: " + gpgProcess.waitFor();
} else {
gpgResult = "Exit code: " + gpgProcess.waitFor() + "\n" + output;
}
If you absolutely must have separate standard error and standard output, you need at least one background thread. I find an ExecutorService makes passing a value from a background thread easier:
ExecutorService background = Executors.newSingleThreadExecutor();
Future<String> stdOutReader = background.submit(() -> readStream(stdOut));
String output = readStream(stdErr);
if (output.isEmpty()) {
output = stdOutReader.get();
}
background.shutdown();
if (output.isEmpty()) {
gpgResult = "Exit code: " + gpgProcess.waitFor();
} else {
gpgResult = "Exit code: " + gpgProcess.waitFor() + "\n" + output;
}
Finally, you should not catch and re-throw IOException just to print it out. Whatever code calls verifyExecution will have to catch IOException anyway; it is that code’s job to print, log, or otherwise handle the IOException. Intercepting it like that will probably result in its being printed twice.

There's no reliable way to tell whether an stream has data available without a call to read()—but that call will block if there are no data available. Methods like available() and ready() aren't reliable, because they can give false negatives; they can report that no data are available, even when there are.
A general-purpose facility that will work with any process requires a separate thread to consume each InputStream. This is because, in general, processes could interleave output to stdout and stderr, and unblocking one could cause the other to block, and so on. The process might write partial standard output, then block on a write to standard error. If your master process uses just one thread, it will hang, regardless which stream it reads first. Independent threads consuming both streams will make sure the process runs smoothly.
If you are running a specific process, and you can guarantee it has certain output in every case, you could take some shortcuts… keeping in mind that, "Short cuts make long delays."

wkhtmltopdf called from java getting hanged

We are the using the following code to generate PDFs using wkhtmltopdf
public class SystemUtils{
public String executeCommand(String... command) {
Process process = null;
try {
// Using redirectErrorStream as true. Otherwise we have to read both process.getInputStream() and
// process.getErrorStream() in order to not exhaust the stream buffer.
process = new ProcessBuilder(command).redirectErrorStream(true).start();
process.waitFor();
StringBuilder outputBuilder = new StringBuilder();
try(BufferedReader stdError = new BufferedReader(new InputStreamReader(process.getInputStream()))) {
String line;
while ((line = stdError.readLine()) != null) {
outputBuilder.append(line).append(StringConstants.CARRIAGE_RETURN);
}
}
return outputBuilder.toString();
} catch (IOException | InterruptedException e) {
String exceptionMsg = "Error while executing command '"+command+"' : ";
LOGGER.error(exceptionMsg, e);
throw new AppException(exceptionMsg, e);
} finally {
if(process != null){
process.destroy();
}
}
}
public static void main(String[] args){
SystemUtils systemUtils = new SystemUtils();
String[] array = {"wkhtmltopdf", "/home/pgullapalli/Desktop/testsimilar1.html", "/home/pgullapalli/Desktop/test.pdf"};
systemUtils.executeCommand(array);
}
}
This works absolutely fine for smaller size files. But when we try to process a larger file, it is indefinitely waiting without any response. I am not sure what is going wrong? Can someone please suggest?

I moved process.waitFor() before the return statement and it started working. This probably could be happening as the output buffer has filled and we are not reading from it. After moving the process.waitFor after the stream reading, things are working fine.

Exec'ing multiple processes from Java: outputs mixed up?

I have a servlet which creates a new Action object inside doGet(), and this object uses exec() to run external processes. There may be several requests to the servlet at the same time, so I might have several Action objects where each one is running an external process at the same time. Occasionally when this happens I find that the output from one process gets mixed up with the output from one of the others.
Each Action creates a unique temporary directory and runs the process with this as its current directory. Separate threads then read the output streams from the process into a string buffer. The code that the Action object executes looks basically like this:
Process proc = null;
ReaderThread stdout = null;
ReaderThread stderr = null;
StringBuffer buff = new StringBuffer();
int exit = -1;
try {
//
// Run the process
//
proc = Runtime.getRuntime().exec(command,null,directory);
//
// Read the output from the process
//
stdout = new ReaderThread(proc.getInputStream(),buff);
stderr = new ReaderThread(proc.getErrorStream(),buff);
stdout.start();
stderr.start();
//
// Get the exit code
//
exit = proc.waitFor();
//
// Wait for all the output to be read
//
stdout.join();
stderr.join();
}
catch (InterruptedException e) {
if (proc != null) {
proc.destroy();
}
}
catch (Exception e) {
buff.append(e.getClass() + ": " + e.getMessage());
if (proc != null) {
proc.destroy();
}
}
So each request uses a separate Action object to run a process, and this has its own StringBuffer "buff" that the output of the process is accumulated into by the two ReaderThreads. But what I find is that, when two requests are running two processes at the same time, the output of one will sometimes end up in the StringBuffer of the thread that is running the other one, and one of the two servlet requests will see output intended for the other one. It basically behaves as if Runtime.exec() provides a single global pipe to which the output streams of all the processes are connected.
The ReaderThread looks like this:
public class ReaderThread extends Thread {
private BufferedReader reader;
private StringBuffer buffer;
public ReaderThread (InputStream stream, StringBuffer buffer) {
this.reader = new BufferedReader(new InputStreamReader(stream));
this.buffer = buffer;
}
#Override
public void run () {
try {
String line;
while ((line = reader.readLine()) != null) {
synchronized (buffer) {
buffer.append(line + "\n");
}
}
}
catch (IOException e) {
synchronized (buffer) {
buffer.append(e.getMessage() + "\n");
}
}
}
}
Can anyone suggest what I can do to fix this?

Use a ThreadLocal variable to store the output of each thread.
When and how should I use a ThreadLocal variable?

Here's the explanation, partly as a cautionary tale:
the output was being added to an ArrayList in the XML node that
accessed it
the XML nodes were created by cloning prototypes
the ArrayList was initialised in the declaration, not in the initialise()
method of the class, so every instance ended up referring to the same
object.
Duh.
Another two days of my life down the drain!
Happy new year...

Reading Input and Error Streams Concurrently using BufferedReaders Hangs

First off let me apologize to the SO community for coming to you with something that ought to be so trivial. But I've been at this all day and I'm at the end of my rope.
There is a section of my program that needs pull text from an input stream and an error stream from a process that is launched using Runtime.getrunTime().exec() and pass it through to standard input and output in an orderly manner. I have a function that near as I can tell should work. But it seems to be getting caught in a catch-22 where it's waiting for the stream to report ready - but the stream has finished and is not reporting. I'm baffled. I can't think of another way to do this that fits my constraints and I'm rather skeptical that such a catch-22 can exist.
Here is my code:
private void forwardStreamtoStd(InputStream in, InputStream err)
throws IOException {
int c = -1;
BufferedReader inReader = new BufferedReader(
new InputStreamReader(in, "US-ASCII"));
BufferedReader errReader = new BufferedReader(
new InputStreamReader(err, "US-ASCII"));
boolean inFinished = false, errFinished = false;
try {
System.out.println("Begin stream read loop...");
while (!inFinished && !errFinished) {
if (!inFinished) {
while (inReader.ready()) {
if ((c = inReader.read()) == -1) {
inFinished = true;
}
else {
System.out.print((char) c);
}
}
}
if (!errFinished) {
while (errReader.ready()) {
if ((c = errReader.read()) == -1) {
errFinished = true;
}
else {
System.err.print((char) c);
}
}
}
}
System.out.println("End stream read loop.");
}
catch (IOException e) {
throw e;
}
finally {
errReader.close();
inReader.close();
}
}
The problem seems to be that the reading loops are waiting for the streams to report ready, and as a result aren't seeing the -1 returned by read telling them that it's time to quit. I'm trying to avoid having either stream blocking, so that I can pull from both in turn when they are prepared. However, how can I catch the process's end of stream? Am I missing something? Shouldn't read report that it's read when it has an end of stream -1? The processes are finishing, and so their streams should be dying. What am I doing wrong here?

There are two more possibilities:
Use the ProcessBuilder and invoke redirectErrorStream(true) to join the two streams and you need to read one stream. I have an example here.
In JDK7, you could call the inheritIO() to automatically forward everything
Edit On the second guess, it seems the ready() call is misleading your program. Try this:
private void forwardStreamtoStd(InputStream in, InputStream err)
throws IOException {
int c = -1;
BufferedReader inReader = new BufferedReader(
new InputStreamReader(in, "US-ASCII"));
BufferedReader errReader = new BufferedReader(
new InputStreamReader(err, "US-ASCII"));
boolean inFinished = false, errFinished = false;
try {
System.out.println("Begin stream read loop...");
if (!inFinished) {
while ((c = inReader.read()) != -1) {
System.out.print((char) c);
}
inFinished = true;
}
if (!errFinished) {
while ((c = errReader.read()) != -1) {
System.err.print((char) c);
}
errFinished = true;
}
System.out.println("End stream read loop.");
}
catch (IOException e) {
throw e;
}
finally {
errReader.close();
inReader.close();
}
}
Or better yet, leave off the BufferedReader if you don't plan any extra transformation:
private void createReader(final InputStream in, final OutputStream out) {
new Thread() {
public void run() {
try {
int c = 0;
while ((c = in.read()) != -1) {
out.write(c);
}
} catch (IOException ex) {
ex.printStackTrace();
} finally {
in.close();
}
}
}.start();
}
private void forwardStreamtoStd(InputStream in, InputStream err)
throws IOException {
createReader(in, System.out);
createReader(err, System.err);
}

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4090471
The solution I've always used is to create a separate thread to read one of the streams, join on the thread when the main thread finishes reading, and then waitFor the process.

It's essential to consume the 2 streams concurrently, to prevent blocking. See this article for more info, and in particular note the StreamGobbler mechanism that captures stdout/err in separate threads.

If I remember correctly, the spawned process will never close the stream - so you would need to have the readers in their own threads, sleep on the main thread until the process is done, and then close the readers.

How do I handle multiple streams in Java?

I'm trying to run a process and do stuff with its input, output and error streams. The obvious way to do this is to use something like select(), but the only thing I can find in Java that does that is Selector.select(), which takes a Channel. It doesn't appear to be possible to get a Channel from an InputStream or OutputStream (FileStream has a getChannel() method but that doesn't help here)
So, instead I wrote some code to poll all the streams:
while( !out_eof || !err_eof )
{
while( out_str.available() )
{
if( (bytes = out_str.read(buf)) != -1 )
{
// Do something with output stream
}
else
out_eof = true;
}
while( err_str.available() )
{
if( (bytes = err_str.read(buf)) != -1 )
{
// Do something with error stream
}
else
err_eof = true;
}
sleep(100);
}
which works, except that it never terminates. When one of the streams reaches end of file, available() returns zero so read() isn't called and we never get the -1 return that would indicate EOF.
One solution would be a non-blocking way to detect EOF. I can't see one in the docs anywhere. Alternatively is there a better way of doing what I want to do?
I see this question here:
link text
and although it doesn't exactly do what I want, I can probably use that idea, of spawning separate threads for each stream, for the particular problem I have now. But surely that isn't the only way to do it? Surely there must be a way to read from multiple streams without using a thread for each?

As you said, the solution outlined in this Answer is the traditional way of reading both stdout and stderr from a Process. A thread-per-stream is the way to go, even though it is slightly annoying.

You will indeed have to go the route of spawning a Thread for each stream you want to monitor. If your use case allows for combining both stdout and stderr of the process in question you need only one thread, otherwise two are needed.
It took me quite some time to get it right in one of our projects where I have to launch an external process, take its output and do something with it while at the same time looking for errors and process termination and also being able to terminate it when the java app's user cancels the operation.
I created a rather simple class to encapsulate the watching part whose run() method looks something like this:
public void run() {
BufferedReader tStreamReader = null;
try {
while (externalCommand == null && !shouldHalt) {
logger.warning("ExtProcMonitor("
+ (watchStdErr ? "err" : "out")
+ ") Sleeping until external command is found");
Thread.sleep(500);
}
if (externalCommand == null) {
return;
}
tStreamReader =
new BufferedReader(new InputStreamReader(watchStdErr ? externalCommand.getErrorStream()
: externalCommand.getInputStream()));
String tLine;
while ((tLine = tStreamReader.readLine()) != null) {
logger.severe(tLine);
if (filter != null) {
if (filter.matches(tLine)) {
informFilterListeners(tLine);
return;
}
}
}
} catch (IOException e) {
logger.logExceptionMessage(e, "IOException stderr");
} catch (InterruptedException e) {
logger.logExceptionMessage(e, "InterruptedException waiting for external process");
} finally {
if (tStreamReader != null) {
try {
tStreamReader.close();
} catch (IOException e) {
// ignore
}
}
}
}
On the calling side it looks like this:
Thread tExtMonitorThread = new Thread(new Runnable() {
public void run() {
try {
while (externalCommand == null) {
getLogger().warning("Monitor: Sleeping until external command is found");
Thread.sleep(500);
if (isStopRequested()) {
getLogger()
.warning("Terminating external process on user request");
if (externalCommand != null) {
externalCommand.destroy();
}
return;
}
}
int tReturnCode = externalCommand.waitFor();
getLogger().warning("External command exited with code " + tReturnCode);
} catch (InterruptedException e) {
getLogger().logExceptionMessage(e, "Interrupted while waiting for external command to exit");
}
}
}, "ExtCommandWaiter");
ExternalProcessOutputHandlerThread tExtErrThread =
new ExternalProcessOutputHandlerThread("ExtCommandStdErr", getLogger(), true);
ExternalProcessOutputHandlerThread tExtOutThread =
new ExternalProcessOutputHandlerThread("ExtCommandStdOut", getLogger(), true);
tExtMonitorThread.start();
tExtOutThread.start();
tExtErrThread.start();
tExtErrThread.setFilter(new FilterFunctor() {
public boolean matches(Object o) {
String tLine = (String)o;
return tLine.indexOf("Error") > -1;
}
});
FilterListener tListener = new FilterListener() {
private boolean abortFlag = false;
public boolean shouldAbort() {
return abortFlag;
}
public void matched(String aLine) {
abortFlag = abortFlag || (aLine.indexOf("Error") > -1);
}
};
tExtErrThread.addFilterListener(tListener);
externalCommand = new ProcessBuilder(aCommand).start();
tExtErrThread.setProcess(externalCommand);
try {
tExtMonitorThread.join();
tExtErrThread.join();
tExtOutThread.join();
} catch (InterruptedException e) {
// when this happens try to bring the external process down
getLogger().severe("Aborted because auf InterruptedException.");
getLogger().severe("Killing external command...");
externalCommand.destroy();
getLogger().severe("External command killed.");
externalCommand = null;
return -42;
}
int tRetVal = tListener.shouldAbort() ? -44 : externalCommand.exitValue();
externalCommand = null;
try {
getLogger().warning("command exit code: " + tRetVal);
} catch (IllegalThreadStateException ex) {
getLogger().warning("command exit code: unknown");
}
return tRetVal;
Unfortunately I don't have to for a self-contained runnable example, but maybe this helps.
If I had to do it again I would have another look at using the Thread.interrupt() method instead of a self-made stop flag (mind to declare it volatile!), but I leave that for another time. :)