We Keep Coding

Error stream blocking when running external command with Java

Working on SEAndroid, I call Setools commands from my Java application.
It works perfectly with small SEAndroid policy and now I need to test my tool with real
SEAndroid policy. But unfortunately, I face a problem with an error stream.
Here my code I used to call external commands :
public static BufferedReader runCommand(final String[] args)
throws IOException {
BufferedReader stdInput = null;
BufferedReader stdError = null;
try {
Process p = Runtime.getRuntime().exec(args);
stdInput = new BufferedReader(new
InputStreamReader(p.getInputStream()));
stdError = new BufferedReader(new
InputStreamReader(p.getErrorStream()));
// read any errors from the attempted command
String s = null;
StringBuilder err = new StringBuilder();
while ((s = stdError.readLine()) != null) {
err.append(s + "\n");
}
if (err.length() != 0) {
throw new IOException(err.toString());
}
return stdInput;
} finally {
if (stdError != null) {
stdError.close();
}
}
}
So, as you can see, I call an external command. Then read the error stream and throw an exception if there is any errors, otherwise I return the InputStream, so I can parse it later.
With a real SEAndroid policy, the error stream seems to block (even if I read a single char) and I can't parse the result of the command. If I close the error stream without reading anything, the application works fine, but I want to handle errors if any.
If I type the command in a console, it works fine too.
In the first case (with small SEAndroid policy), the output of the command is small ( ~350 lines).
In the second case (with a real SEAndroid policy), the output of the command is larger ( >1500 lines).
Is it possible that the size of the output stream influences the error stream? The two streams are two distinctive resources, isn't it?
The fact that I do not read the output stream immediately have an importance?
I fear that its not a "programming" problem but more a system problem...
Any suggestion?
Thanks in advance for your help=)
Edit:
I try to read the output stream before the error stream and it works. But I need to check the error stream before perform any parsing on the output stream, so the problem is still topical.

First, it's probably better to use the newer ProcessBuilder class as opposed to Runtime exec. If you want to go a step further, you can even use Apache commons-exec which takes care of stream handling and other things for you.
Next, as you've discovered, process control is a tricky thing in Java and you've run into one of its tricky issues. From the documentation for java's Process class:
The parent process uses these streams to feed input to and get output from the subprocess. Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.
You need to have something consuming both (Error and Output) streams or you risk deadlock - these should each be read on their own threads. Using something like a StreamGobbler (google it, there are plenty out there) would be a good step, or you can roll your own if you're so inclined. It isn't too hard to get it right but if you're unfamiliar with multithreading you may want to look at someone else's implementation or go the Apache commons-exec route.

The processing of output is so annoying, that I wrote little library called jproc that deals with the problem of consuming stdout and stderr. It can simply filter strings through external programs likes this:
ProcBuilder.filter("x y z","sed" ,"s/y/a/")
It also lets you specify a timeout for the completion and will convert non-zero exit codes into exception.

Why does java.lang.process's readline() behave differently for reading inputstream on different boxes with the same os

I tested this code(below) on several different linux boxes(4+) and it worked fine. However, on one linux box I ran into an issue with readline() hanging for the error inputStream(errorStream). This stream should be empty so I suspected that box was not writing out a line terminator to the errorStream for the error. I changed my code to use read() instead of readline()...but read() also hung.
I tried retrieving the input inputStream first, and that worked and there was no hangs with readline()/read() for the error inputstream. I could not do this since I needed to obtain possible errors first. Appearing to be a deadlock, I was able to resolve this by having each inputstream read from it's own thread. Why did I only see this issue on one box? Is there a kernel setting or some other setting specific to this box that could have caused this?
ProcessBuilder processBuilder = new ProcessBuilder()
try
{
Process processA = null;
synchronized (processBuilder)
{
processBuilder.command("/bin/sh","-c"," . /Home/SomeScript.ksh");
processA = processBuilder.start();
}
inputStream = processA.getInputStream();
reader = new BufferedReader(new InputStreamReader(inputStream));
errorStream = processA.getErrorStream();
errorReader = new BufferedReader(new InputStreamReader(errorStream));
String driverError;
while ((driverError = errorReader.readLine()) != null)
{
//some code
}

Why did I only see this issue on one box?
Most likely because of something in the script that is being run ... and its interactions with its environment (e.g. files, environment variables, etc)
Is there a kernel setting or some other setting specific to this box that could have caused this?
It is possible but unlikely that it is a kernel setting. It might be "something else". Indeed, it has to be "something" outside of the Java application that is to blame, at least in part.
I suggest you do the following temporarily (at least):
ProcessBuilder processBuilder = new ProcessBuilder();
processBuilder.command("/bin/sh","-c"," . /Home/SomeScript.ksh");
processBuilder.redirectErrorStream(true);
processA = processBuilder.start();
inputStream = processA.getInputStream();
reader = new BufferedReader(new InputStreamReader(inputStream));
String line;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
System.out.println("Return code is " + processA.exitValue());
That way you can see what all of the output is.
There should not be a problem if the external process fails to put a newline at the end of the last line. The Java process will see an EOF on the input stream, and the BufferedReader will return what characters it has ... and return null on the next call.
Another possibility is that the external process is blocking because it is trying to read from its standard input.
UPDATE
The redirectErrorStream also resolved the issue, but I need the error stream separate.
OK so if it did (reliably) solve the problem then that (most likely) means that you have to read the external processes stdout and stderr streams in parallel. The simple way to do is to create 2 threads to read and buffer the two streams separately. For example: Capturing stdout when calling Runtime.exec
(Your problem is due to the fact that pipes have a finite buffering capacity. The external problem is most likely alternating between writing stuff to stdout and stderr. If it tries to write to one of the pipes when that pipe is "full", it will block. But if your application is reading all of the other pipe (to EOF) before it reads the blocked pipe, then everything will deadlock. The fact that the external process is stuck in PIPE_W state is more evidence for this explanation.
One possible reason that you are seeing different behaviour on different systems is that the amount of buffering in a pipe is system dependent. But it could also be due to differences in what the external process is doing; e.g. its inputs.)

You are running OS specific commands in a script, any one could be holding the error output. You can avoid this by discarding the errors, but that is unlikely to be a good idea.
I would check the version of the OS are the same and whether you have any significant differences in the command you run in the script. If this doesn't help, take out commands from the script until it starts working. I assume an empty script doesn't do this.

Calling Python from Java (Tomcat6) as sub-process

I am trying to call a python script from a java/tomcat6 webapp. I am currently using the following code:
Process p = Runtime.getRuntime().exec("python <file.py>");
InputStream in = p.getInputStream();
InputStreamReader isr = new InputStreamReader(in);
BufferedReader b = new BufferedReader(isr);
logger.info("PYTHON OUTPUT");
String line = null;
while ( (line = b.readLine()) != null){
logger.info(line);
}
p.waitFor();
logger.info("COMPLETE PYTHON OUTPUT");
logger.info("EXIT VALUE: "+p.exitValue());
I can't really see any output in the catalinia.out file from the python script and using an adapter library like jython is not possible as the script relies on several machine learning libraries that need python's Numpy module to work.
Help?

The explanation is probably one (or more) of following:
The command is failing and writing error messages to its "stderr" fd ... which you are not looking at.
The command is failing to launch because the command name is incorrect; e.g. it can't be found on $PATH.
The command is trying to read from its stdin fd ... but you haven't provided any input (yet).
It could be a problem with command-line splitting; e.g if you are using pathnames with embedded spaces, or other things that would normally be handled by the shell.
Also, since this is python, this could be a problem with python-specific environment variables, the current directory and/or the effective user that is executing the command.
How to proceed:
Determine if the python command is actually starting. For instance. "hack" the "" to write something to a temporary file on startup.
Change to using ProcessBuilder to create the Process object. This will give you more control over the streams and how they are handled.
Find out what is going to the child processes "stderr". (ProcessBuilder allows you to redirect it to "stdout" ...)

Send signal from C to Java

So I have this Java program that creates processes that run a certain C program on the terminal (unix), and I need to notify the Java program when something happens in the C program (before the termination). How can I do this? I know that I'm gonna need signals, but I don't have much experience on the subject.
Thanks in advance!
EDIT: This the changes I made to the java, after calling the process:
InputStream stdout = p.getInputStream();
InputStreamReader isr = new InputStreamReader(stdout);
BufferedReader br = new BufferedReader(isr);
String line = null;
System.out.println("<INPUT>");
while ( (line = br.readLine()) != null)
System.out.println(line);
System.out.println("</INPUT>");
int exitVal = p.waitFor();
System.out.println("Process exitValue: " + exitVal);
I what I did in the C when I want to notify the java:
char buff[20];
size_t nbytes;
ssize_t bytes_written;
int fd;
strcpy(buf, "This is a test\n");
nbytes = strlen(buf);
bytes_written = write(1, buff, nbytes);
But after running it I only get:
INPUT
/INPUT
Process exitValue: 0

One of the ways is reading your C program's stdout
Process p = Runtime.getRuntime().exec("c.exe");
InputStream stdout = p.getInputStream();
now C program can talk to Java program

You can create a socket using c program and send it to java socket server. You can search on goole to find out sample.

Easiest way to communicate between a Java program and another program spawned by it, is using the FIFO streams that are set up when you launch a subprocess. So, for example, if you launched your subprocess using Runtime.exec() or using ProcessBuilder, you have an object of type Process. By calling its getInputStream() you can have access to the process' stdout.
In the native code, just print your signals out to stdout and they can be read by the Java process.

You can use Signals with Java, but this is not the best mechanism. I would look at using JNI, JMS, Socket, RPC or some other solution first. The problem with signals is that it doesn't allow you to transmit much information and it's the easiest to get right or debug.
For more details on signal handling and Java
http://www.oracle.com/technetwork/java/javase/signals-139944.html
http://www.ibm.com/developerworks/java/library/i-signalhandling/ (link dead)
http://ringlord.com/dl/Signals-in-Java.pdf

Detecting Graphical interface after calling Runtime.exec()

I'm trying to integrate with a legacy system. The legacy system uses dialog windows to report errors. It have no return codes at all except for the dialog windows. I start the legacy system with Runtime.exec().
Is there a way to detect if the executed program has spawned dialog windows or any other graphical interface? This solution is done in Windows and the executed program is an exe.

If the legacy system report errors in console, is possible get your erros.
Simply take the inputstream of error and do your reading.
Like this:
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec(cmd);
// any error message?
InputStream error = proc.getErrorStream();
InputStreamReader isr = new InputStreamReader(error);
BufferedReader br = new BufferedReader(isr);
String line=null;
while ( (line = br.readLine()) != null)
System.out.println(" ERROR >" + line);
I recommend to read: Runtime.exec() quirks
Hope this help.

You can use this JNA snippet to poll for windows started by your process.
AFAIK, you can only get the standard and error output streams from a process using the Java Process API.

So the solution i did, is to use the snippet code that #Gerrett Hall linked to. That snippet code check the active window before i run the command and save the name.
Then after a vile if the command have not returned check if the active window have changed. if it have, kill the process(Alt. send a return key global).
To get the info from the dialog i could use Ctrl + C to copy the content of the dialog and reading the paste buffer to copy the message in to the log. Have not figured out that part yet.
And yes this is a ugly hack but so is the legacy system to.