We Keep Coding

ProcessBuilder: process doesn't finish without "redirectOutput"

I'm automating a gradle build using Java. I execute "gradlew.bat" in a Process created from a ProcessBuilder. Here's the code:
ProcessBuilder gradlewProcessBuilder = new ProcessBuilder(mainDirPath.concat("\\android\\gradlew.bat"), "assembleDebug");
gradlewProcessBuilder.directory(new File(mainDirPath.concat("/android")));
gradlewProcessBuilder.redirectOutput(ProcessBuilder.Redirect.INHERIT); //This is the line
Process gradlewProcess = gradlewProcessBuilder.start();
gradlewProcess.waitFor();
Now this code works flawlessly, but it outputs the gradle console through the application console and I don't want that. If I delete this line:
gradlewProcessBuilder.redirectOutput(ProcessBuilder.Redirect.INHERIT);
The process hangs in waitFor() indefinitely.
I have absolutely no idea how could redirectOutput have effect over this, any light you can shed is welcome.

If the output is not redirected or handled, then a default output buffer is allocated. However, on Windows the output buffer is very small; when it fills up, the app will stop executing and will block until the output buffer is drained.
Therefore, you need to drain the output somehow; either by using INHERIT or by having a thread which drains the output. I recommend using https://github.com/zeroturnaround/zt-exec since it has much better default handling.

Run external program concurrently and communicate with it through stdin / stdout

I want to be able to run an external program concurrently with my Java code, i.e. I want to start the program, then return control to the calling method while keeping the external program running at the same time. The Java code will then keep generating input and send it to the external program and receive output back.
I don't want to keep loading the external program as it has very high overhead. What is the best way to accomplish this? Thanks!

Have a look at ProcessBuilder. Once you've set up the ProcessBuilder and executed start you'll have a handle to a Process to which you can feed input and read output.
Here's a snippet to get you started:
ProcessBuilder pb = new ProcessBuilder("/bin/bash");
Process proc = pb.start();
// Start reading from the program
final Scanner in = new Scanner(proc.getInputStream());
new Thread() {
public void run() {
while (in.hasNextLine())
System.out.println(in.nextLine());
}
}.start();
// Write a few commands to the program.
PrintWriter out = new PrintWriter(proc.getOutputStream());
out.println("touch hello1");
out.flush();
out.println("touch hello2");
out.flush();
out.println("ls -la hel*");
out.flush();
out.close();
Output:
-rw-r--r-- 1 aioobe aioobe 0 2011-04-08 08:29 hello1
-rw-r--r-- 1 aioobe aioobe 0 2011-04-08 08:29 hello2

YOu can launch the external app with Runtime.getRuntime().exec(...)
To send data to the external program, you can either send data on the Processes output stream (You get a Process object back from exec) or you can open sockets and communicate that way.

I think you will find the Javadoc for class java.lang.Process helpful. Of note, you can get the input and output streams from a Process to communicate with it while it is running.

I second the answer about using ProcessBuilder. If you want to know more details about this, and why you should prefer it to Runtime.exec(), see this entry in the Java glossary. It also shows how to use threads to communicate with the external process.

I had issues trying to achieve bidirectional communication with the external process through stdin/stdout, because of blocking. In the end I found a github gist which allowed me solve the issue simply and elegantly; that gist is actually based on a stackoverflow answer.
See that other answer for sample code, but the core of the idea is to set up an event loop for reading and writing (while loop with 10ms sleeping), and using low-level stream operations so that no caching and blocking is going on -- only try to read if you know the other process in fact wrote something (through InputStream.available()).
It leads to a bit strange programming style, but the code is much simpler than it would be if using threads, and does the job pretty well.

Java - (android) Reuse a process after flushing its OutputStream

im trying to do this on Android:
Process p = Runtime.getRuntime().exec("sh");
DataOutputStream out = new DataOutputStream(p.getOutputStream());
out.writeBytes("something useful\n");
out.close();
p.waitFor();
out = new DataOutputStream(p.getOutputStream());
out.writeBytes("something useful\n");
out.close();
p.waitFor();
The second time I execute out.writeBytes(); , I get a java IOException: "Bad file number".
My app has to execute several native programs, but always use the same process.
Anyone know why this does not work?

Note that the shell is not part of the public SDK (note it is not documented anywhere in the SDK documentation), so this code is in effect relying on private APIs.
Also this puts you outside of the normal application model -- we have no guarantee what will happen to a process you have forked and is not being managed by the platform. It may get killed at any time.
This is also a very inefficient way to do things, compared to doing whatever the command is doing in your own process. And starting a separate process for a command won't let it do anything more than you can, because it still runs as your uid.
So basically... for 99.99% of apps please don't do this. If you are writing a terminal app... well, okay, only geeks are going to care about that anyway, and it isn't going to be of much use since it runs as your uid, but okay. But otherwise, please no. :)

When you call out.close(), it will automatically call close() on the ouputstream of your process.
Each time you call p.getOutputStream() you get the same OutputStream, on your second use of out, p.getOutputStream() returns an already closed OutputStream.
Basically with your code, you don't really need to close the first DataOutputStream.
Sources :
Sources of DataOutputStream extends FilterOutputStream
Sources of FilterOutputStream.close()

The best way to monitor output of process along with its execution

I have started a process in my Java code, this process take a very long time to run and could generate some output from time to time. I need to react to every output when they are generated, what is the best way to do this?

What kind of reaction are you talking about? Is the process writing to its standard output and/or standard error? If so, I suspect Process.getInputStream and Process.getErrorStream are what you're looking for. Read from both of those and react accordingly. Note that you may want to read from both of them from different threads, to avoid the individual buffer for either stream from filling up.
Alternatively, if you don't need the two separately, just leave redirectErrorStream in ProcessBuilder as false, so the error and output streams are merged.

You should start a thread which reads from the Process.getInputStream() and getErrorStream() (or alternatively use ProcessBuilder.redirectErrorStream(true)) and handle it when something shows up in the stream. There are many ways that how to handle it - the right way depends on how the data is being used. Please tell more details.
Here is one real-life example: SbtRunner uses ProcessRunner to send commands to a command line application and wait for the command to finish execution (the application will print "> " when a command finishes execution). There is some indirection happening to make it easier to read from the process' output (the output is written to a MulticastPipe from where it is then read by an OutputReader).

Why can't my Java program read Perl's STDERR?

We have a Perl program to validate XML which is invoked from a Java program. It is not able to write to standard error and hanging in the print location.
Perl is writing to STDERR and a java program is reading the STDERR using getErrorStream() function. But the Perl program is hanging to write to STDERR. I suspect Java function is blocking the STDERR stream completely and Perl is waiting for this stream to be released.
Is there a way in Perl to overcome this blockage and write to standard error forcefully? Since Java is doing only a read the API should not be locking the STDERR stream as per java doc.
Perl Code snippet is:
sub print_error
{
print STDERR shift;
}
Java code snippet is:
while ( getErrorStream() != null )
{
SOP errorMessage;
}
Appreciate the help in advance.
Thanks,
Mathew Liju

getErrorStream does not read the error stream, it just obtains a handle to it. As it's a pipe, if you never actually read it, it will fill up and force the Perl program to block.
You need something like:
Inputstream errors = getErrorStream();
while (errors.read(buffer) > 0) {
SOP buffer;
}

Ideally, I think that to avoid deadlock, in Java you need to spawn separate threads to read the STDERR and the STDOUT. It sounds like Perl is blocking when writing to STDERR because for one reason or another you are never reading from it in Java.

An additional factor to consider is the buffering that occurs with piped processes.
There is by default, about a 30-line-ish buffer that is maintained by the shell creating the inter-process pipe, so if the Perl app has not created enough data, it won't have been sent to the Java application yet to process.

May be this thread has a possible cause for your problem:
Add 3 lines to the top of the Perl script:
use IO::Handle;
STDOUT->autoflush(1);
STDERR->autoflush(1);
The problem in the mentioned thread was related to "the way Perl is buffering its output".
However here, Adrian Pronk mentions in the comments that "Perl is hanging because Java is never reading its output".

STDOUT->autoflush(1);
STDERR->autoflush(1);
This is the information I needed!
I have a Java app running some Perl scripts and I'd only get the output after it was finished.
By adding the autoflush(1) I get it right away.
BTW, I do have separate threads for reading STDERR and STDOUT, and that's the way to go.
Thanks.