We Keep Coding

Java Process Builder always giving data in Error Stream [duplicate]

Can anyone clarify me if the below procedure is correct way to handle streams of process without any stream buffer full and blocking problems
I'm invoking external program from java program, I'm using ProcessBuilder to build the process and after I perform
Process gpgProcess = processBuilder.start();
I'm handling the process using a method
String executionResult = verifyExecution(gpgProcess);
and in my method i'm trying to handle the process streams
private String verifyExecution(Process gpgProcess) throws IOException, InterruptedException {
String gpgResult = null;
BufferedReader stdOut = new BufferedReader(new InputStreamReader(gpgProcess.getInputStream()));
BufferedReader stdErr = new BufferedReader(new InputStreamReader(gpgProcess.getErrorStream()));
gpgProcess.waitFor();
if(stdErr.ready()) {
gpgResult = "Exit code: " + gpgProcess.exitValue() + "\n" + readStream(stdErr);
} else if(stdOut.ready()) {
gpgResult = "Exit code: " + gpgProcess.exitValue() + "\n" + readStream(stdOut);
} else {
gpgResult = "Exit code: " + gpgProcess.exitValue();
}
int exitCode = gpgProcess.exitValue();
this.setExitCode(exitCode);
stdOut.close();
stdErr.close();
if(exitCode != 0) {
throw new RuntimeException("Pgp Exception: " + gpgResult);
}
return gpgResult;
}
The readStream method is used to read my stream text.
private String readStream(BufferedReader reader) throws IOException {
StringBuilder result = new StringBuilder();
try {
while(reader.ready()) {
result.append(reader.readLine());
if(reader.ready()) {
result.append("\n");
}
}
} catch(IOException ioe) {
System.err.println("Error while reading the stream: " + ioe.getMessage());
throw ioe;
}
return result.toString();
}

No, that is not the correct way to do it.
First, on some systems, your code will be stuck on the gpgProcess.waitFor() call forever, because the process cannot finish until its standard out and standard error have been fully read and consumed.
Second, you are not using the ready() method of Reader correctly. The documentation states that the method returns true only if reading a character is guaranteed not to block. Returning false does not mean that the end of the stream has been reached; it just means the next read might block (meaning, it might not return immediately).
The only ways to know when you have reached the end of a Reader’s data stream are:
check whether any of its read methods return a negative number
check whether the readLine method of BufferedReader returns null
So your readStream method should look like this:
String line;
while ((line = reader.readLine()) != null) {
result.append(line).append("\n");
}
As of Java 8, you can make it even shorter:
return reader.lines().collect(Collectors.joining("\n"));
Similarly, you should not be calling stdErr.ready() or stdOut.ready(). Either or both methods might or might not return true, even when there are no characters available; the only guarantee for the ready() method is that returning true means the next read will not block. It is possible for ready() to return true even at the end of the character stream, when the next read would immediately return -1, as long as that read does not block.
In summary, don't use ready() at all. Consume all of both streams, and check whether the error stream is empty:
String output = readStream(stdErr);
if (output.isEmpty()) {
String output = readStream(stdOut);
}
gpgResult = "Exit code: " + gpgProcess.exitValue() + "\n" + output;
That would address the case your question appears to present: Either the Process produces standard error and no lines on standard output, or the other way around. However, this will not properly handle Processes in general.
For the general case, the easiest solution is to have the process merge its standard error with standard output using redirectErrorStream, so there is only one stream to consume:
processBuilder.redirectErrorStream(true);
Process gpgProcess = processBuilder.start();
The verifyExecution method could then contain:
String output;
try (BufferedReader stdOut = new BufferedReader(new InputStreamReader(gpgProcess.getInputStream()))) {
output = readStream(stdOut);
}
if (output.isEmpty()) {
gpgResult = "Exit code: " + gpgProcess.waitFor();
} else {
gpgResult = "Exit code: " + gpgProcess.waitFor() + "\n" + output;
}
If you absolutely must have separate standard error and standard output, you need at least one background thread. I find an ExecutorService makes passing a value from a background thread easier:
ExecutorService background = Executors.newSingleThreadExecutor();
Future<String> stdOutReader = background.submit(() -> readStream(stdOut));
String output = readStream(stdErr);
if (output.isEmpty()) {
output = stdOutReader.get();
}
background.shutdown();
if (output.isEmpty()) {
gpgResult = "Exit code: " + gpgProcess.waitFor();
} else {
gpgResult = "Exit code: " + gpgProcess.waitFor() + "\n" + output;
}
Finally, you should not catch and re-throw IOException just to print it out. Whatever code calls verifyExecution will have to catch IOException anyway; it is that code’s job to print, log, or otherwise handle the IOException. Intercepting it like that will probably result in its being printed twice.

There's no reliable way to tell whether an stream has data available without a call to read()—but that call will block if there are no data available. Methods like available() and ready() aren't reliable, because they can give false negatives; they can report that no data are available, even when there are.
A general-purpose facility that will work with any process requires a separate thread to consume each InputStream. This is because, in general, processes could interleave output to stdout and stderr, and unblocking one could cause the other to block, and so on. The process might write partial standard output, then block on a write to standard error. If your master process uses just one thread, it will hang, regardless which stream it reads first. Independent threads consuming both streams will make sure the process runs smoothly.
If you are running a specific process, and you can guarantee it has certain output in every case, you could take some shortcuts… keeping in mind that, "Short cuts make long delays."

Java reader does not start printing until closing the programm [duplicate]

I have the following code example below. Whereby you can enter a command to the bash shell i.e. echo test and have the result echo'd back. However, after the first read. Other output streams don't work?
Why is this or am I doing something wrong? My end goal is to created a Threaded scheduled task that executes a command periodically to /bash so the OutputStream and InputStream would have to work in tandem and not stop working. I have also been experiencing the error java.io.IOException: Broken pipe any ideas?
Thanks.
String line;
Scanner scan = new Scanner(System.in);
Process process = Runtime.getRuntime ().exec ("/bin/bash");
OutputStream stdin = process.getOutputStream ();
InputStream stderr = process.getErrorStream ();
InputStream stdout = process.getInputStream ();
BufferedReader reader = new BufferedReader (new InputStreamReader(stdout));
BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(stdin));
String input = scan.nextLine();
input += "\n";
writer.write(input);
writer.flush();
input = scan.nextLine();
input += "\n";
writer.write(input);
writer.flush();
while ((line = reader.readLine ()) != null) {
System.out.println ("Stdout: " + line);
}
input = scan.nextLine();
input += "\n";
writer.write(input);
writer.close();
while ((line = reader.readLine ()) != null) {
System.out.println ("Stdout: " + line);
}

Firstly, I would recommend replacing the line
Process process = Runtime.getRuntime ().exec ("/bin/bash");
with the lines
ProcessBuilder builder = new ProcessBuilder("/bin/bash");
builder.redirectErrorStream(true);
Process process = builder.start();
ProcessBuilder is new in Java 5 and makes running external processes easier. In my opinion, its most significant improvement over Runtime.getRuntime().exec() is that it allows you to redirect the standard error of the child process into its standard output. This means you only have one InputStream to read from. Before this, you needed to have two separate Threads, one reading from stdout and one reading from stderr, to avoid the standard error buffer filling while the standard output buffer was empty (causing the child process to hang), or vice versa.
Next, the loops (of which you have two)
while ((line = reader.readLine ()) != null) {
System.out.println ("Stdout: " + line);
}
only exit when the reader, which reads from the process's standard output, returns end-of-file. This only happens when the bash process exits. It will not return end-of-file if there happens at present to be no more output from the process. Instead, it will wait for the next line of output from the process and not return until it has this next line.
Since you're sending two lines of input to the process before reaching this loop, the first of these two loops will hang if the process hasn't exited after these two lines of input. It will sit there waiting for another line to be read, but there will never be another line for it to read.
I compiled your source code (I'm on Windows at the moment, so I replaced /bin/bash with cmd.exe, but the principles should be the same), and I found that:
after typing in two lines, the output from the first two commands appears, but then the program hangs,
if I type in, say, echo test, and then exit, the program makes it out of the first loop since the cmd.exe process has exited. The program then asks for another line of input (which gets ignored), skips straight over the second loop since the child process has already exited, and then exits itself.
if I type in exit and then echo test, I get an IOException complaining about a pipe being closed. This is to be expected - the first line of input caused the process to exit, and there's nowhere to send the second line.
I have seen a trick that does something similar to what you seem to want, in a program I used to work on. This program kept around a number of shells, ran commands in them and read the output from these commands. The trick used was to always write out a 'magic' line that marks the end of the shell command's output, and use that to determine when the output from the command sent to the shell had finished.
I took your code and I replaced everything after the line that assigns to writer with the following loop:
while (scan.hasNext()) {
String input = scan.nextLine();
if (input.trim().equals("exit")) {
// Putting 'exit' amongst the echo --EOF--s below doesn't work.
writer.write("exit\n");
} else {
writer.write("((" + input + ") && echo --EOF--) || echo --EOF--\n");
}
writer.flush();
line = reader.readLine();
while (line != null && ! line.trim().equals("--EOF--")) {
System.out.println ("Stdout: " + line);
line = reader.readLine();
}
if (line == null) {
break;
}
}
After doing this, I could reliably run a few commands and have the output from each come back to me individually.
The two echo --EOF-- commands in the line sent to the shell are there to ensure that output from the command is terminated with --EOF-- even in the result of an error from the command.
Of course, this approach has its limitations. These limitations include:
if I enter a command that waits for user input (e.g. another shell), the program appears to hang,
it assumes that each process run by the shell ends its output with a newline,
it gets a bit confused if the command being run by the shell happens to write out a line --EOF--.
bash reports a syntax error and exits if you enter some text with an unmatched ).
These points might not matter to you if whatever it is you're thinking of running as a scheduled task is going to be restricted to a command or a small set of commands which will never behave in such pathological ways.
EDIT: improve exit handling and other minor changes following running this on Linux.

I think you can use thread like demon-thread for reading your input and your output reader will already be in while loop in main thread so you can read and write at same time.You can modify your program like this:
Thread T=new Thread(new Runnable() {
#Override
public void run() {
while(true)
{
String input = scan.nextLine();
input += "\n";
try {
writer.write(input);
writer.flush();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
} );
T.start();
and you can reader will be same as above i.e.
while ((line = reader.readLine ()) != null) {
System.out.println ("Stdout: " + line);
}
make your writer as final otherwise it wont be able to accessible by inner class.

You have writer.close(); in your code. So bash receives EOF on its stdin and exits. Then you get Broken pipe when trying to read from the stdoutof the defunct bash.

Java exec stdin not flushing

For a school project I am trying to create a terminal in Java. The terminal works in the following way:
User types a command
Program grabs command and replaces <command> with the command in the string
/bin/bash -c "cd current/directory/; <command>; echo kjsfdjkadhlga; pwd
Program starts the process created via a ProcessBuilder object
Program spawns a thread that reads from stdout and stderr
Program continues looking for user input, and if the command is done running, then whatever the user entered is run as a command, otherwise it is fed to the currently running command as input.
As output is generated, program looks through the output for the kjsfdjkadhlga string so it knows when the user's command is done being run, and then grabs the remaining output and stores it as the current path that the user is at.
How this works/reasons for everything:
In order to avoid myself having to implement my own input parser to handle things like multiple commands on a line, IO redirection, and whatnot to work with the ProcessBuilder, I just essentially convert the command to a bash script and let bash execute it.
Since every process executes only a single command (or whatever it was given at the time of creation, which is a single user command in this case) then terminates, no process specific information is stored, such as the current working directory. To transfer that information, I call pwd after the user's command and then in the process of the next command, but before the user's command is run, I cd to that directory, effectively allowing the value of $PWD to persist between processes.
The Problem:
It all works well, except for when user interaction is required. If the user just types cat, it is supposed to wait for a line of user input, then print it, then wait for a line of user input, then print it, and repeat forever (I don't handle Crtl+C yet...). However, what actually happens is that the terminal waits for a line of user input, then prints it, then terminates without waiting for more input.
What I have tried:
Currently, I provide input to the command being run with:
BufferedWriter stdin = new BufferedWriter(new OutputStreamWriter(process.getOutputStream()));
stdin.write(input);
stdin.newLine();
stdin.close();
If instead of calling close(), I call flush(), then cat ends up waiting for user input and not doing anything until I terminate my Terminal program, at which point it then prints everything the user had input.
It appears that the flush() function doesn't actually do anything. A Stack Overflow question mentioned using the raw OutputStream and calling write() instead of using a BufferedWriter. However, that has the same effect. In the OutputStream documentation for flush(), it states that "The flush method of OutputStream does nothing."
I have also tried using a BufferedOutputStream, but the documentation says that its flush function simply forces the buffered data to be written to the underlying OutputStream, which doesn't change the fact that the OutputStream is not flushing its stream.
This question seems to be the most promising, but I couldn't get it to work when implementing it. It may be because I am on Mac OS instead of Windows.
Does anybody know how to do this if keeping stdin open long enough to submit multiple lines of input is possible, or if I am going about it wrong?
Code
main()
Terminal terminal = new Terminal();
Scanner in = new Scanner(System.in);
while (in.hasNextLine())
{
String line = in.nextLine();
terminal.sendInput(line, terminal);
}
terminal.sendInput() called by main
// ProcessReaderDelegate implements functions called when receiving output on stdout, stderr, and when the process terminates.
public int sendInput(String text, ProcessReaderDelegate delegate)
{
if (processes.size() > 0)
{
processes.get(0).sendInput(text); // Is a ProcessReader object
return 1;
}
run(text, delegate); // runs the given text as the <command> text described above
return 2;
}
ProcessReader's sendInput() called by terminal.sendInput()
public boolean sendInput(String input)
{
try
{
// stdin and process are a instance fields
// tried this and doesn't seem to work (with either flush or close)
stdin = new BufferedWriter(new OutputStreamWriter(process.getOutputStream()));
stdin.write(input);
stdin.newLine();
stdin.close();
// tried this and doesn't seem to work (with either flush or close)
//BufferedOutputStream os = new BufferedOutputStream(process.getOutputStream());
//os.write(input.getBytes());
//os.write("\n".getBytes());
//os.flush();
//os.close();
return true;
}
catch (IOException e)
{
System.out.println("ERROR: this should never happen: " + e.getMessage());
return false;
}
}
terminal.run() called by terminal.sendInput()
public void run(String command, ProcessReaderDelegate delegate)
{
// don't do anything with empty command since it screws up the command concatentaion later
if (command.equals(""))
{
delegate.receivedOutput(null, prompt);
return;
}
try
{
// create the command
List<String> list = new ArrayList<String>();
list.add(shellPath);
list.add(UNIX_BASED ? "-c" : "Command : ");
String cmd = (UNIX_BASED ? getUnixCommand(command) : getWindowsCommand(command));
list.add(cmd);
//System.out.println("command='" + list.get(0) + " " + list.get(1) + " " + list.get(2) + "'");
// create the process and run it
ProcessBuilder builder = new ProcessBuilder(list);
Process p = builder.start();
ProcessReader stdout = new ProcessReader(p, delegate, this);
new Thread(stdout).start();
processes.add(stdout);
}
catch (IOException e)
{
System.out.println(e.getMessage());
}
}
ProcessReader.run() executed in thread and reads stdout and stderr
public void run()
{
try
{
boolean hitend = false;
String buffer = "";
while (true)
{
int c;
String text;
// ======================================================
// read from stdout
// read the next character
c = stdout.read();
// build the string
while (c != -1) // while data available in the stream
{
buffer += (char)c;
c = stdout.read();
}
// send the string to the delegate
if ((!hitend) && (buffer.length() > 0))
{
// END_STRING is the "kjsfdjkadhlga" echoed after the command executes
int index = buffer.indexOf(END_STRING);
if (index >= 0)
{
hitend = true;
text = buffer.substring(0, index);
buffer = buffer.substring(index + END_STRING.length());
if (outputDelegate != null)
{
outputDelegate.receivedOutput(process, text);
}
}
else
{
for (int i = END_STRING.length() - 1; i >= 0; i--)
{
index = buffer.indexOf(END_STRING.substring(0, i));
if (i == 0)
{
index = buffer.length();
}
if (index >= 0)
{
text = buffer.substring(0, index);
buffer = buffer.substring(index + i);
if (outputDelegate != null)
{
outputDelegate.receivedOutput(process, text);
}
}
}
}
}
// ======================================================
// read from stderr
// read the next character
c = stderr.read();
text = ""; // slow method; make faster with array
// build the string
while (c != -1) // while data available in the stream
{
text += (char)c;
c = stderr.read();
}
// send the string to the delegate
if ((text.length() > 0) && (outputDelegate != null))
{
outputDelegate.receivedError(process, text);
}
// ======================================================
// check if the process is done (and hence no more output)
boolean done = false;
try
{
int value = process.exitValue();
done = true; // if got to this point, then process is done
// read the ending environment variables
Map<String, String> env = new HashMap<String, String>();
String[] words = buffer.split(" ");
env.put(ENV_WORKING_DIR, words[0]);
if (envDelegate != null)
{
envDelegate.processTerminatedWithEnvironment(process, env);
}
// end the process
outputDelegate.processEnded(process);
stdout.close();
stderr.close();
break;
}
catch (Exception e) {System.out.println(e.getMessage());} // no exit value --> process not done
if (done) // just on the off chance that closing the streams crashes everything
{
break;
}
}
}
catch (IOException e)
{
System.out.println("ERROR: ProcessReader: " + e.getMessage());
}
}

IF Condition is Met but skipped to the next IF

I'm not sure why, but while I am trying to debug, I find this is very weird:
As you see in the image, the value of in.readLine() is null and in.readLine() == null is true. But why it skips the if (in.readLine() == null) { ... line? But when I tried to place the breakpoint in line 266 and 267, it's entering the code on that condition.
The code:
private void startSOfficeService() throws InterruptedException, IOException {
if (System.getProperty("os.name").matches(("(?i).*Windows.*"))) {
try {
//Check if the soffice process is running
Process process = Runtime.getRuntime().exec("tasklist /FI \"IMAGENAME eq soffice.exe\"");
//Need to wait for this command to execute
int code = process.waitFor();
//If we get anything back from readLine, then we know the process is running
BufferedReader in = new BufferedReader(new InputStreamReader(process.getInputStream()));
if (in.readLine() == null) {
//Nothing back, then we should execute the process
String[] SOFFICE_CMD = { SOFFICE_SERVICE_PATH,
"-accept=socket,host=" + SOFFICE_SERVICE_HOST + ",port=" + SOFFICE_SERVICE_PORT + ";urp;",
"-invisible",
"-nologo"};
process = Runtime.getRuntime().exec(SOFFICE_CMD);
code = process.waitFor();
System.out.println("soffice script started");
} else {
System.out.println("soffice script is already running");
}
in.close();
in = null;
System.gc();
} catch(Exception e) {
e.printStackTrace();
}
}
}

When your debugger evaluates in.readLine(), it consumes from the reader. Therefore, if you were on the last line of whatever is being read, in.readLine() would be non-null, putting control in the else, but when you evaluate in.readLine() to display in the debugger, it reads again, finds that there are no more lines, and returns null as the value to show in the debugger.
To see the real picture, assign in.readLine() to a variable first, and watch the value of that variable, which won't change by simply reading it.

Getting command prompt output instantly?

Probably a misleading title, but thanks to people on my other question, I got my program somewhat working, though now not sure what to do.
Here is my method to run command prompt commands and return the output
public static String cmdExec(String cmdLine) {
String line;
String output = "";
try {
Process p = Runtime.getRuntime().exec(cmdLine);
BufferedReader input = new BufferedReader(new InputStreamReader(p.getInputStream()));
while ((line = input.readLine()) != null) {
output += (line + '\n');
}
input.close();
}
catch (Exception ex) {
ex.printStackTrace();
}
return output;
}
It works how I want it to work, though now when I run this method, it is going to wait until everything is done, and then return the output, which is not what I want.
I made a ping flooding method that uses the cmdExec method. Here is how I call cmdExec()
String[] input = ipTextField.getText().split(":");
if(input.length == 2) {
System.out.println(cmdExec("cmd /c ping " + input[0] + " -t -n " + Integer.parseInt(input[1])));
So now for instance, if I type "127.0.0.1:3" in my program, it's going to send 3 packets to localhost. Now the problem here is, instead of printing out the lines I get as output from command prompt 1 by 1, it's going to wait until all the 3 packets are sent, and then print the output full output.
If I type "ping 127.0.0.1 -t -n 3" in command prompt, it's going to print the reply 1 by 1, not all just at once, so how would I go about doing the same in my program?

Instead of
while ((line = input.readLine()) != null) {
output += (line + '\n');
}
To output the results to the command line immediately just print within the while loop
while ((line = input.readLine()) != null) {
System.out.println(line);
output += (line + '\n');
}

You can print out continuously with:
while ((line = input.readLine()) != null) {
System.out.println(line);
output += (line + '\n');
}
(The following is still useful for future viewers, but is not an applicable answer to the question.)
Simply insert a process.waitFor() before reading from the output stream. This command waits until a process dies before continuing. You then don't have to read line by line, either. You can simply dump the buffer all at once.
Either this, or you can find a string which only occurs on the last line (e.g. Minimum in Windows) and change the while condition to:
while((line == input.readline()).indexOf("Minimum") == -1)