I have a minor Problem with a small Project I'm trying to do.
I'm trying to use a Java-Program to call a Python-Script.
Java:
ProcessBuilder pb = new ProcessBuilder("python3", "tmp.py");
process = pb.start();
OutputStream stdin = process.getOutputStream();
InputStream stderr = process.getErrorStream();
InputStream stdout = process.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(stdout));
BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(stdin));
writer.write("example" + "\n");
String output = reader.readLine();
Python-Script tmp.py (example):
import sys
sys.stdin.readline()
print("Hello World")
It wont terminate, seemingly because sys.sdin.readline() isnt catching any input and if I remove this line it terminates just fine.
(stderror was empty too)
I tried different things to fix this but nothing seems to work.
I would really appreciate any advice. Thanks in advance.
(Update: I tried to modify the Java-Program to access a .jar File instead of a Python-Script but the same error occurs here to. Using the Python Method subprocess.Popen() to access the Script however works just fine.)
Make sure you start python unbuffered with -u flag:
Force the binary layer of the stdout and stderr streams (which is
available as their buffer attribute) to be unbuffered. The text I/O
layer will still be line-buffered if writing to the console, or
block-buffered if redirected to a non-interactive file.
Edit
I recommend reviewing the buffering all the same. The python unbuffered is a common one, but you possibly still have some buffering. Make sure you flush java writer, writer.flush().
Related
I want to parse content of some file by srcML parser which is an external windows program. I'm doing this in a following way:
String command = "src2srcml.exe --language java";
Process proc = Runtime.getRuntime().exec(command);
InputStream fileInput = Files.newInputStream(file)
OutputStream procOutput = proc.getOutputStream();
IOUtils.copy(fileInput, procOutput);
IOUtils.copy() is from Commons IO 2.4 library.
When my file is small (several KB) everything works fine. However, when I try to copy some relatively big file (~72 KB) my program hangs.
Moreover, when I execute the parser 'manually' in cmd:
src2srcml.exe --language Java < BigFile.java
everything works fine, too.
Any ideas why this is happening?
You should buffer the OutputStream:
OutputStream procOutput = proc.getOutputStream();
BufferedOutputStream bos = new BufferedOutputStream(procOutput);
IOUtils.copy(fileInput, bos);
Moreover, why don't you simply redirect fileInput as the process InputStream?
ProcessBuilder pb = new ProcessBuilder(command);
pb.redirectInput(file);
Process proc = pb.start();
proc.waitFor();
The problem is most likely that you are not consuming the output of the external program in a separate thread. you need to start a separate thread to consume the output so that the external program does not get blocked.
I want to run PianoBar from a Java GUI (PianoBar is a program that runs Pandora from command line). I thought this would be quick and dirty, but I guess I don't know enough about interaction between programs.
I use ProcessBuilder to launch an instance of PianoBar like so:
private Process createPianoBarProcess() throws IOException {
String[] command = {"CMD", "/C", "pianobar"};
ProcessBuilder probuilder = new ProcessBuilder( command );
probuilder.redirectErrorStream(true);
probuilder.directory(new File("~~location where pianobar.exe is~~"));
Process process = probuilder.start();
return process;
}
After I create the process, I create a BufferedReader to read in the PianoBar output:
Process pianoBar = createPianoBarProcess();
InputStream inS = pianoBar.getInputStream();
InputStreamReader isr = new InputStreamReader(inS);
BufferedReader br = new BufferedReader(isr);
But when I read the output from PianoBar via this reader, it spits out the first line of PianoBar ("Welcome to pianobar (2013.05.19-win32)! Press ? for a list of commands."), then it spits out the next line ("[?] Email:"). Then it just hangs.
Obviously, it is waiting for the user to input their email. But no matter what I try, I can't get my Java program to write the email to the PianoBar process when prompted - it just hangs as soon as it reads out the last character.
Is it possible to do what I am trying to do? I thought it would be an easy thing to look for on the internet, but I haven't been able to find anything. All I want is an easy way to write to the external process when prompted. Seems like this should be easy...
You may use the following code snippet to get working:
String s;
//s = email
BufferedWriter bufferedwriter = new BufferedWriter(new OutputStreamWriter(pianoBar.getOutputStream()));
bufferedwriter.write(s);
bufferedwriter.flush();
Done!
Remember to surround the code block with appropriate try/catch
I tested this code(below) on several different linux boxes(4+) and it worked fine. However, on one linux box I ran into an issue with readline() hanging for the error inputStream(errorStream). This stream should be empty so I suspected that box was not writing out a line terminator to the errorStream for the error. I changed my code to use read() instead of readline()...but read() also hung.
I tried retrieving the input inputStream first, and that worked and there was no hangs with readline()/read() for the error inputstream. I could not do this since I needed to obtain possible errors first. Appearing to be a deadlock, I was able to resolve this by having each inputstream read from it's own thread. Why did I only see this issue on one box? Is there a kernel setting or some other setting specific to this box that could have caused this?
ProcessBuilder processBuilder = new ProcessBuilder()
try
{
Process processA = null;
synchronized (processBuilder)
{
processBuilder.command("/bin/sh","-c"," . /Home/SomeScript.ksh");
processA = processBuilder.start();
}
inputStream = processA.getInputStream();
reader = new BufferedReader(new InputStreamReader(inputStream));
errorStream = processA.getErrorStream();
errorReader = new BufferedReader(new InputStreamReader(errorStream));
String driverError;
while ((driverError = errorReader.readLine()) != null)
{
//some code
}
Why did I only see this issue on one box?
Most likely because of something in the script that is being run ... and its interactions with its environment (e.g. files, environment variables, etc)
Is there a kernel setting or some other setting specific to this box that could have caused this?
It is possible but unlikely that it is a kernel setting. It might be "something else". Indeed, it has to be "something" outside of the Java application that is to blame, at least in part.
I suggest you do the following temporarily (at least):
ProcessBuilder processBuilder = new ProcessBuilder();
processBuilder.command("/bin/sh","-c"," . /Home/SomeScript.ksh");
processBuilder.redirectErrorStream(true);
processA = processBuilder.start();
inputStream = processA.getInputStream();
reader = new BufferedReader(new InputStreamReader(inputStream));
String line;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
System.out.println("Return code is " + processA.exitValue());
That way you can see what all of the output is.
There should not be a problem if the external process fails to put a newline at the end of the last line. The Java process will see an EOF on the input stream, and the BufferedReader will return what characters it has ... and return null on the next call.
Another possibility is that the external process is blocking because it is trying to read from its standard input.
UPDATE
The redirectErrorStream also resolved the issue, but I need the error stream separate.
OK so if it did (reliably) solve the problem then that (most likely) means that you have to read the external processes stdout and stderr streams in parallel. The simple way to do is to create 2 threads to read and buffer the two streams separately. For example: Capturing stdout when calling Runtime.exec
(Your problem is due to the fact that pipes have a finite buffering capacity. The external problem is most likely alternating between writing stuff to stdout and stderr. If it tries to write to one of the pipes when that pipe is "full", it will block. But if your application is reading all of the other pipe (to EOF) before it reads the blocked pipe, then everything will deadlock. The fact that the external process is stuck in PIPE_W state is more evidence for this explanation.
One possible reason that you are seeing different behaviour on different systems is that the amount of buffering in a pipe is system dependent. But it could also be due to differences in what the external process is doing; e.g. its inputs.)
You are running OS specific commands in a script, any one could be holding the error output. You can avoid this by discarding the errors, but that is unlikely to be a good idea.
I would check the version of the OS are the same and whether you have any significant differences in the command you run in the script. If this doesn't help, take out commands from the script until it starts working. I assume an empty script doesn't do this.
I'm working on a project that requires me to run an instance of scala using the Process class. To do this I execute the following code:
Process process = RunTime.getRunTime().exec("<path to scala binary> <scala file>");
This call gives me a process representative of the scala program that is executing. I am able to access the process's input stream and receive input from the process with:
BufferedReader stdin = new BufferedReader(new InputStreamReader(process.getInputStream()));
stdin.readLine();
The scala program I am trying to run involves a Scanner which picks up input and prints it out. When I try to register a BufferedWriter to the OutputStreamWriter that is generated from process.getOutputStream(), and then write to that stream with:
stdout.write("Hello");
The scala program does not receive this input, and I was wondering: why does this occur?
Turns out I forgot to flush the stream.
Everything works fine on the command line, but when I translate what I want into Java, the receiving process never gets anything on stdin.
Here's what I have:
private void deployWarFile(File warFile, String instanceId) throws IOException, InterruptedException {
Runtime runtime = Runtime.getRuntime();
// FIXME(nyap): Use Jsch.
Process deployWarFile = runtime.exec(new String[]{
"ssh",
"gateway",
"/path/to/count-the-bytes"});
OutputStream deployWarFileStdin = deployWarFile.getOutputStream();
InputStream deployWarFileStdout = new BufferedInputStream(deployWarFile.getInputStream());
InputStream warFileInputStream = new FileInputStream(warFile);
IOUtils.copy(warFileInputStream, deployWarFileStdin);
IOUtils.copy(deployWarFileStdout, System.out);
warFileInputStream.close();
deployWarFileStdout.close();
deployWarFileStdin.close();
int status = deployWarFile.waitFor();
System.out.println("************ Deployed with status " + status + " file handles. ************");
}
The script 'count-the-bytes' is simply:
#!/bin/bash
echo "************ counting stdin bytes ************"
wc -c
echo "************ counted stdin bytes ************"
The output indicates that the function hangs at the 'wc -c' line -- it never gets to the 'counted stdin bytes' line.
What's going on? Would using Jsch help?
You might try closing the output stream before you expect wc -c to return.
IOUtils.copy(warFileInputStream, deployWarFileStdin);
deployWarFileStdin.close();
IOUtils.copy(deployWarFileStdout, System.out);
warFileInputStream.close();
deployWarFileStdout.close();
Would using Jsch help?
Using JSch would only help if you would be using the setInputStream() and setOutputStream() methods of the channel instead of the IOUtils.copy method, since they manage the copying on a separate thread.
ChannelExec deployWarFile = (ChannelExec)session.openChannel("exec");
deployWarFile.setCommand("/path/to/count-the-bytes");
deployWarFile.setOutputStream(System.out);
deployWarFile.setInputStream(new BufferedInputStream(new FileInputStream(warFile)));
deployWarFile.connect();
(Here you somehow have to wait until the other side closes the channel.)
If you simply replaced the Runtime.exec with opening an ChannelExec (and starting it after getting the streams), the problem would be completely the same, and could be solved by the same solution mentioned by antlersoft, i.e. closing the input before reading the output:
ChannelExec deployWarFile = (ChannelExec)session.openChannel("exec");
deployWarFile.setCommand("/path/to/count-the-bytes");
OutputStream deployWarFileStdin = deployWarFile.getOutputStream();
InputStream deployWarFileStdout = new BufferedInputStream(deployWarFile.getInputStream());
InputStream warFileInputStream = new FileInputStream(warFile);
deployWarFile.connect();
IOUtils.copy(warFileInputStream, deployWarFileStdin);
deployWarFileStdin.close();
warFileInputStream.close();
IOUtils.copy(deployWarFileStdout, System.out);
deployWarFileStdout.close();
(Of course, if you have longer output, you will want to do input and output in parallel, or simply use the first method.)
You probably get an error, but the process hangs because you are not reading the error stream.
Taken from the Process JavaDoc
All its standard io (i.e. stdin, stdout, stderr) operations will be redirected to the parent process through three streams (Process.getOutputStream(), Process.getInputStream(), Process.getErrorStream()). The parent process uses these streams to feed input to and get output from the subprocess. Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.
So you need to read all of them. Using the ProcessBuilder is probably easier