We have a Perl program to validate XML which is invoked from a Java program. It is not able to write to standard error and hanging in the print location.
Perl is writing to STDERR and a java program is reading the STDERR using getErrorStream() function. But the Perl program is hanging to write to STDERR. I suspect Java function is blocking the STDERR stream completely and Perl is waiting for this stream to be released.
Is there a way in Perl to overcome this blockage and write to standard error forcefully? Since Java is doing only a read the API should not be locking the STDERR stream as per java doc.
Perl Code snippet is:
sub print_error
{
print STDERR shift;
}
Java code snippet is:
while ( getErrorStream() != null )
{
SOP errorMessage;
}
Appreciate the help in advance.
Thanks,
Mathew Liju
getErrorStream does not read the error stream, it just obtains a handle to it. As it's a pipe, if you never actually read it, it will fill up and force the Perl program to block.
You need something like:
Inputstream errors = getErrorStream();
while (errors.read(buffer) > 0) {
SOP buffer;
}
Ideally, I think that to avoid deadlock, in Java you need to spawn separate threads to read the STDERR and the STDOUT. It sounds like Perl is blocking when writing to STDERR because for one reason or another you are never reading from it in Java.
An additional factor to consider is the buffering that occurs with piped processes.
There is by default, about a 30-line-ish buffer that is maintained by the shell creating the inter-process pipe, so if the Perl app has not created enough data, it won't have been sent to the Java application yet to process.
May be this thread has a possible cause for your problem:
Add 3 lines to the top of the Perl script:
use IO::Handle;
STDOUT->autoflush(1);
STDERR->autoflush(1);
The problem in the mentioned thread was related to "the way Perl is buffering its output".
However here, Adrian Pronk mentions in the comments that "Perl is hanging because Java is never reading its output".
STDOUT->autoflush(1);
STDERR->autoflush(1);
This is the information I needed!
I have a Java app running some Perl scripts and I'd only get the output after it was finished.
By adding the autoflush(1) I get it right away.
BTW, I do have separate threads for reading STDERR and STDOUT, and that's the way to go.
Thanks.
Related
In my Java code I have found quite significant performance differences between two similar commands:
execString=new String[]{"CMD.EXE","/C", path_to_executable };
Runtime.getRuntime().exec(command)
runs my executable almost twice as quickly (6-7mins vs 3-4mins) as:
execString=new String[]{" path_to_executable };
Runtime.getRuntime().exec(command)
Please can someone educate me as to why? One seems to be telling the executable to run directly, whereas the other is telling cmd.exe to run the executable...?
Thanks in advance :-)
EDIT:
The same performance discrepancies were noted when using ProcessBuilder:
ProcessBuilder myPB = new ProcessBuilder(execString);
Process myProcess = myPB.start();
I have discovered the answer here:
https://stackoverflow.com/a/24676491/1961025
From the API doc of java.lang.Process:
Because some native platforms only provide limited buffer size for
standard input and output streams, failure to promptly write the input
stream or read the output stream of the subprocess may cause the
subprocess to block, or even deadlock.
Basically, you need to make sure that the process is handling the input, output and error streams. Mine wasn't. When using cmd.exe, I think it kind of wraps the executable so it's not an issue. Using the gobblers from https://www.infoworld.com/article/2071275/when-runtime-exec---won-t.html?page=2 works a treat!
Thanks!
Using zt-exec I would like to know how it can write/read to a process which is waiting for console input on an infinite loop, as well as responding using its console out?
I believe the easiest way to describe this is with a Python script:
while(True):
javaSaid = raw_input("Hey Java, Say Something: ") ##wait for input from java
print "Python Heard Java Say: " +str(javaSaid) ##java needs to be able to get this output
Note: Executing the python process multiple times is what I am trying to avoid as the initialization time on the real python script makes this unacceptable.
You need to call redirectInput as well as redirectOutput on ProcessExecutor.
Have a look at ProcessExecutorInputStreamTest.java. It's just an example. It writes data to the process input vie PipedOutputStream -> PipedInputStream -> ProcessExecutor and read data from the process via an OutputStream.
I want to invoke an external program in java code, then the Google tell me that the Runtime or ProcessBuilder can help me to do this work. I have tried it, and there come out a problem the java program can't exit, that means both the sub process and the father process wait for forever. they are hanging or deadlock.
Someone tell me the reason is that the sub process's cache is too small. when it try to give back data to the father process, but the father process don't read it in time, then both of them hang. So they advice me fork an thread to be in charge of read sub process's cache data. I do it as what they tell me, but there still some problem.
Then I close the output stream which get by the method getOutputStream(). Finally, the program success. But I don't know why it happen? Is there some relationship between the output steam and input stream?
You have provided very few details in your question, so I can only provide a general answer.
All processes have three standard streams: standard input, standard output and standard error. Standard input is used for reading in data, standard output for writing out data, and standard error for writing out error messages. When you start an external program using Runtime.getRuntime().exec() or ProcessBuilder, Java will create a Process object for the external program, and this Process object will have methods to access these streams.
These streams are accessed as follows:
process.getOutputStream(): return the standard input of the external program. This is an OutputStream as it is something your Java code will write to.
process.getInputStream(): return the standard output of the external program. This is an InputStream as it is something your Java code will read from.
process.getErrorStream(): return the standard error of the external program. This is an InputStream as, like standard output, it is something your Java code will read from.
Note that the names of getInputStream() and getOutputStream() can be confusing.
All streams between your Java code and the external program are buffered. This means each stream has a small amount of memory (a buffer) where the writer can write data that is yet to be read by the reader. The writer does not have to wait for the reader to read its data immediately; it can leave its output in the buffer and continue.
There are two ways in which writing to buffers and reading from them can hang:
attempting to write data to a buffer when there is not enough space left for the data,
attempting to read from an empty buffer.
In the first situation, the writer will wait until space is made in the buffer by reading data out of it. In the second, the reader will wait until data is written into the buffer.
You mention that closing the stream returned by getOutputStream() caused your program to complete successfully. This closes the standard input of the external program, telling it that there will be nothing more for it to read. If your program then completes successfully, this suggests that your program was waiting for more input to come when it was hanging.
It is perhaps arguable that if you do run an external program, you should close its standard input if you don't need to use it, as you have done. This tells the external program that there will be no more input, and so removes the possibility of it being stuck waiting for input. However, it doesn't answer the question of why your external program is waiting for input.
Most of the time, when you run external programs using Runtime.getRuntime().exec() or ProcessBuilder, you don't often use the standard input. Typically, you'd pass whatever inputs you'd need to the external program on the command line and then read its output (if it generates any at all).
Does your external program do what you need it to and then get stuck, apparently waiting for input? Do you ever need to send it data to its standard input? If you start a process on Windows using cmd.exe /k ..., the command interpreter will continue even after the program it started has exited. In this case, you should use /c instead of /k.
Finally, I'd like to emphasise that there are two output streams, standard output and standard error. There can be problems if you read from the wrong stream at the wrong time. If you attempt to read from the external program's standard output while its buffer is empty, your Java code will wait for the external program to generate output. However, if your external program is writing a lot of data to its standard error, it could fill the buffer and then find itself waiting for your Java code to make space in the buffer by reading from it. The end result of this is your Java code and the external program are both waiting for each other to do something, i.e. deadlock.
This problem can be eliminated simply by using a ProcessBuilder and ensuring that you call its redirectErrorStream() method with a true value. Calling this method redirects the standard error of the external program into its standard output, so you only have one stream to read from.
I want to be able to run an external program concurrently with my Java code, i.e. I want to start the program, then return control to the calling method while keeping the external program running at the same time. The Java code will then keep generating input and send it to the external program and receive output back.
I don't want to keep loading the external program as it has very high overhead. What is the best way to accomplish this? Thanks!
Have a look at ProcessBuilder. Once you've set up the ProcessBuilder and executed start you'll have a handle to a Process to which you can feed input and read output.
Here's a snippet to get you started:
ProcessBuilder pb = new ProcessBuilder("/bin/bash");
Process proc = pb.start();
// Start reading from the program
final Scanner in = new Scanner(proc.getInputStream());
new Thread() {
public void run() {
while (in.hasNextLine())
System.out.println(in.nextLine());
}
}.start();
// Write a few commands to the program.
PrintWriter out = new PrintWriter(proc.getOutputStream());
out.println("touch hello1");
out.flush();
out.println("touch hello2");
out.flush();
out.println("ls -la hel*");
out.flush();
out.close();
Output:
-rw-r--r-- 1 aioobe aioobe 0 2011-04-08 08:29 hello1
-rw-r--r-- 1 aioobe aioobe 0 2011-04-08 08:29 hello2
YOu can launch the external app with Runtime.getRuntime().exec(...)
To send data to the external program, you can either send data on the Processes output stream (You get a Process object back from exec) or you can open sockets and communicate that way.
I think you will find the Javadoc for class java.lang.Process helpful. Of note, you can get the input and output streams from a Process to communicate with it while it is running.
I second the answer about using ProcessBuilder. If you want to know more details about this, and why you should prefer it to Runtime.exec(), see this entry in the Java glossary. It also shows how to use threads to communicate with the external process.
I had issues trying to achieve bidirectional communication with the external process through stdin/stdout, because of blocking. In the end I found a github gist which allowed me solve the issue simply and elegantly; that gist is actually based on a stackoverflow answer.
See that other answer for sample code, but the core of the idea is to set up an event loop for reading and writing (while loop with 10ms sleeping), and using low-level stream operations so that no caching and blocking is going on -- only try to read if you know the other process in fact wrote something (through InputStream.available()).
It leads to a bit strange programming style, but the code is much simpler than it would be if using threads, and does the job pretty well.
I have started a process in my Java code, this process take a very long time to run and could generate some output from time to time. I need to react to every output when they are generated, what is the best way to do this?
What kind of reaction are you talking about? Is the process writing to its standard output and/or standard error? If so, I suspect Process.getInputStream and Process.getErrorStream are what you're looking for. Read from both of those and react accordingly. Note that you may want to read from both of them from different threads, to avoid the individual buffer for either stream from filling up.
Alternatively, if you don't need the two separately, just leave redirectErrorStream in ProcessBuilder as false, so the error and output streams are merged.
You should start a thread which reads from the Process.getInputStream() and getErrorStream() (or alternatively use ProcessBuilder.redirectErrorStream(true)) and handle it when something shows up in the stream. There are many ways that how to handle it - the right way depends on how the data is being used. Please tell more details.
Here is one real-life example: SbtRunner uses ProcessRunner to send commands to a command line application and wait for the command to finish execution (the application will print "> " when a command finishes execution). There is some indirection happening to make it easier to read from the process' output (the output is written to a MulticastPipe from where it is then read by an OutputReader).