I'm spawning a process using ProcessBuilder in Java and trying to read its output:
public static void main(String... args) throws IOException, InterruptedException {
ProcessBuilder builder = new ProcessBuilder("test.exe");
Process process = builder.start();
InputStream stdout = process.getInputStream();
final BufferedReader reader = new BufferedReader(new InputStreamReader(stdout));
final List<String> queue = new ArrayList<String>();
Thread ioThread = new Thread() {
public void run() {
System.out.println("STARTED THREAD");
String line = null;
try {
while ((line = reader.readLine()) != null) {
System.out.println("ADD: " + line);
queue.add(line);
Thread.yield();
}
} catch (IOException exception) {
System.err.println("Fatal Error: " + exception.getMessage());
}
}
};
ioThread.start();
Thread.sleep(10000);
System.out.println("GOT: " + queue);
}
This works perfectly fine on my Windows machine, and also on a local Linux installation.
On my Linux VM's (DigitalOcean and Linode), it does not work, however, meaning that the process gets started, but the running thread is unable to add a single line to the queue (after waiting ten seconds in the main program, queue is empty whereas in Windows, the queue contains the read lines).
Thinks I've tried so far:
redirecting the error stream to the output stream (even although I'm sure the test program is outputting lines on STDOUT
using stdbuf with -oL and -eL... doesn't work
trying to read byte by byte manually instead of using BufferedReader, same issue
converting thread to a FutureTask
switching between OpenJDK and Oracle VM
It definitely works in Windows and on some Linux machines.
It turned out to be a buffering issue after all: one stdbuf couldn't fix in this case.
Using a pseudo terminal with script did the trick, however, using:
ProcessBuilder builder = new ProcessBuilder();
builder.command().add("script");
builder.command().add("-c");
builder.command().add(command);
builder.command().add("-f");
builder.command().add("-q");
Related
Code below prints output of executing process (using different thread for output):
public static void main(String[] args) throws InterruptedException,
IOException {
ProcessBuilder builder = new ProcessBuilder("some_command");
final Process process = builder.start();
final Thread ioThread = new Thread() {
#Override
public void run() {
try {
final BufferedReader reader = new BufferedReader(
new InputStreamReader(process.getInputStream()));
String line;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
reader.close();
} catch (final Exception e) {
e.printStackTrace();
}
}
};
ioThread.start();
process.waitFor();
}
So, if I run this code with
ProcessBuilder builder = new ProcessBuilder("ping","8.8.8.8");
output will be like this:
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=205 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=57 time=202 ms
...
But if I run
ProcessBuilder builder = new ProcessBuilder("wvdial");
then command will be executed (wvdial will start) but there will be no output in BufferedReader. wvdial in terminal shows output text.
Why wvdial gives no output for ProcessBuilder?
Maybe it writes to stderr? process.getErrorStream()
Most UNIX programs avoid writing information messages for humans to stdout. Especially batch programs. stdout is for program results, for data output, not for informational messages.
You might also be able to make sense of exit code thus abolishing the need of output parsing.
I am trying to run plink in my own console window. I started by using Process.exec() and that worked fine. The I moved to using ProcessBuilder and now the output is not sent out until I kill the process.
My code looks like this:
class ConsoleOutputThread extends Thread {
public void start(String processName) {
// this was old code: Runtime r = Runtime.getRuntime();
try {
builder = new ProcessBuilder("plink", "-ssh", "192.168.3.21");
builder.redirectErrorStream(true);
process = builder.start();
//this was old code: process = r.exec (processName);
} catch (IOException ex) {
}
start();
}
#Override
public void run() {
try {
BufferedReader is = new BufferedReader(new InputStreamReader(process.getInputStream()));
writer = new BufferedWriter(new OutputStreamWriter(process.getOutputStream()));
try {
process.waitFor();
} catch (InterruptedException ex) {
}
char b[];
b = new char[1];
while(is.read(b, 0, 1)> 0) {
// this is for debug, normally sent to console
System.out.println("Got character: " + b[0]);
}
} catch (IOException ex) {
}
}
}
So, when using Runtime.exec() everything worked fine. Now, with ProcessBuilder, the read function blocks forever (actually until I kill the process, when everuthing is spitted out). However, the error stream works, i.e. if I put a bad option I get the messages in the console.
I am probably missing something here and looking for help.
Thank you
You've set the plink process to write its output to a pipe which is connected to the java process. Anything output by the plink process will be saved in an operating-system buffer until your process reads it. The OS buffer has a limited capacity,, and if plink writes too much data, then it will block until your process reads some data from the buffer.
Unfortunately, the java process waits for the plink process to complete before reading anything from the pipe. So, if the plink process writes too much output, it will block indefinitely.
You should change the java logic to read the plink process's output before calling waitfor().
First, this is my code :
import java.io.*;
import java.util.Date;
import com.banctecmtl.ca.vlp.shared.exceptions.*;
public class PowershellTest implements Runnable {
public static final String PATH_TO_SCRIPT = "C:\\Scripts\\ScriptTest.ps1";
public static final String SERVER_IP = "XX.XX.XX.XXX";
public static final String MACHINE_TO_MOD = "MachineTest";
/**
* #param args
* #throws OperationException
*/
public static void main(String[] args) throws OperationException {
new PowershellTest().run();
}
public PowershellTest(){}
#Override
public synchronized void run() {
String input = "";
String error = "";
boolean isHanging = false;
try {
Runtime runtime = Runtime.getRuntime();
Process proc = runtime.exec("powershell -file " + PATH_TO_SCRIPT +" "+ SERVER_IP +" "+ MACHINE_TO_MOD);
proc.getOutputStream().close();
InputStream inputstream = proc.getInputStream();
InputStreamReader inputstreamreader = new InputStreamReader(inputstream);
BufferedReader bufferedreader = new BufferedReader(inputstreamreader);
proc.waitFor();
String line;
while (!isHanging && (line = bufferedreader.readLine()) != null) {
input += (line + "\n");
Date date = new Date();
while(!bufferedreader.ready()){
this.wait(1000);
//if its been more then 1 minute since a line has been read, its hanging.
if(new Date().getTime() - date.getTime() >= 60000){
isHanging = true;
break;
}
}
}
inputstream.close();
inputstream = proc.getErrorStream();
inputstreamreader = new InputStreamReader(inputstream);
bufferedreader = new BufferedReader(inputstreamreader);
isHanging = false;
while (!isHanging && (line = bufferedreader.readLine()) != null) {
error += (line + "\n");
Date date = new Date();
while(!bufferedreader.ready()){
this.wait(1000);
//if its been more then 1 minute since a line has been read, its hanging.
if(new Date().getTime() - date.getTime() >= 60000){
isHanging = true;
break;
}
}
}
inputstream.close();
proc.destroy();
} catch (IOException e) {
//throw new OperationException("File IO problem.", e);
} catch (InterruptedException e) {
//throw new OperationException("Script thread problem.",e);
}
System.out.println("Error : " + error + "\nInput : " + input);
}
}
I'm currently trying to run a powershell script that will start/stop a vm (VMWARE) on a remote server. The script work from command line and so does this code. The thing is, I hate how I have to use a thread (and make it wait for the script to respond, as explained further) for such a job. I had to do it because both BufferedReader.readline() and proc.waitFor() hang forever.
The script, when ran from cmd, is long to execute. it stall for 30 sec to 1 min from validating authentification with the server and executing the actual script. From what I saw from debugging, the readline hang when it start receiving those delays from the script.
I'm also pretty sure it's not a memory problem since I never had any OOM error in any debugging session.
Now I understand that Process.waitFor() requires me to flush the buffer from both the error stream and the regular stream to work and so that's mainly why I don't use it (I need the output to manage VM specific errors, certificates issues, etc.).
I would like to know if someone could explain to me why it hangs and if there is a way to just use the typical readline() without having it to hang so hard. Even if the script should have ended since a while, it still hang (I tried to run both the java application and a cmd command using the exact same thing I use in the java application at the same time, left it runingfor 1h, nothing worked). It is not just stuck in the while loop, the readline() is where the hanging is.
Also this is a test version, nowhere close to the final code, so please spare me the : this should be a constant, this is useless, etc. I will clean the code later. Also the IP is not XX.XX.XX.XXX in my code, obviously.
Either explanation or suggestion on how to fix would be greatly appreciated.
Ho btw here is the script I currently use :
Add-PSSnapin vmware.vimautomation.core
Connect-VIServer -server $args[0]
Start-VM -VM "MachineTest"
If you need more details I will try to give as much as I can.
Thanks in advance for your help!
EDIT : I also previously tested the code with a less demanding script, which job was to get the content of a file and print it. Since no waiting was needed to get the information, the readline() worked well. I'm thus fairly certain that the problem reside on the wait time coming from the script execution.
Also, forgive my errors, English is not my main language.
Thanks in advance for your help!
EDIT2 : Since I cannot answer to my own Question :
Here is my "final" code, after using threads :
import java.io.*;
public class PowershellTest implements Runnable {
public InputStream is;
public PowershellTest(InputStream newIs){
this.is = newIs;
}
#Override
public synchronized void run() {
String input = "";
String error = "";
try {
InputStreamReader inputstreamreader = new InputStreamReader(is);
BufferedReader bufferedreader = new BufferedReader(inputstreamreader);
String line;
while ((line = bufferedreader.readLine()) != null) {
input += (line + "\n");
}
is.close();
} catch (IOException e) {
//throw new OperationException("File IO problem.", e);
}
System.out.println("Error : " + error + "\nInput : " + input);
}
}
And the main simply create and start 2 thread (PowerShellTest instances), 1 with the errorStream and 1 with the inputStream.
I believe I made a dumb error when I first coded the app and fixed it somehow as I reworked the code over and over. It still take a good 5-6 mins to run, which is somehow similar if not longer than my previous code (which is logical since the errorStream and inputStream get their information sequentially in my case).
Anyway, thanks to all your answer and especially Miserable Variable for his hint on threading.
First, don't call waitFor() until after you've finished reading the streams. I would highly recommend you look at ProcessBuilder instead of simply using Runtime.exec, and split the command up yourself rather than relying on Java to do it for you:
ProcessBuilder pb = new ProcessBuilder("powershell", "-file", PATH_TO_SCRIPT,
SERVER_IP, MACHINE_TO_MOD);
pb.redirectErrorStream(true); // merge stdout and stderr
Process proc = pb.start();
redirectErrorStream merges the error output into the normal output, so you only have to read proc.getInputStream(). You should then be able to just read that stream until EOF, then call proc.waitFor().
You are currently waiting to complete reading from inputStream before starting to read from errorStream. If the process writes to its stderr before stdout maybe you are getting into a deadlock situation.
Try reading from both streams from concurrently running threads. While you are at it, also remove proc.getOutputStream().close();. It shouldn't affect the behavior, but it is not required either.
To be more specific, what i really want to do is to open two new terminals. From terminal_1 i want to ssh #host1 and run a program1 to host1. From terminal_2 i want to ssh #host2 and run a program2 to host2. I need the output of program1 to be viewed on terminal_1 and output of program2 to be viewed on terminal_2.
(I have managed to open xterm and ssh #host.I tried to pass a second command "&&java echo_1" but it does nothing at all)
Here is the code:
import java.io.*;
public class multi1 implements Runnable {
public void run() {
try {
String ss = null;
Runtime obj = null;
String[] newcmd = new String[]{"/usr/bin/xterm","-hold","-e","ssh andreas#192.168.0.0&&java echo_1"};
Process p = Runtime.getRuntime().exec(newcmd);
BufferedReader stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));
BufferedReader stdError = new BufferedReader(new InputStreamReader(p.getErrorStream()));
while ((ss = stdInput.readLine()) != null) {
System.out.println(ss);
}
}catch (IOException e) {
System.out.println("FROM CATCH" + e.toString());
}
}
public static void main(String[] args) throws Exception {
Thread th = new Thread(new multi1());
th.start();
//Thread th2 = new Thread(new multi1());
//th2.start();
}
}
I rather use jsch or other ssh library and do it without terminals and exec.
One concern I have immediately is that you're not handling the stdout/stderr from the spawned process. If you don't do this, it's likely that you'll block the output from the process and the process itself. See this answer for more info.
BufferedReader stdError = ..
..would be flagged a warning by Eclipse as it does not seem to be referenced again. That in turn would indicate that the error stream is not being consumed. Do so.
For more tips on using a Process, see the article linked in the runtime.exec info. page. Also, better to use a ProcessBuilder, which can merge the System.out and System.err, making it easier to consume both with one loop.
This is my code to start a process in Windows via java (and gobble the output).
public static void main(String[] args) throws Exception {
String[] command = new String[3];
command[0] = "cmd";
command[1] = "/C";
command[2] = "test.exe";
final Process child = Runtime.getRuntime().exec(command);
new StreamGobbler(child.getInputStream(), "out").start();
new StreamGobbler(child.getErrorStream(), "err").start();
BufferedWriter out = new BufferedWriter(new OutputStreamWriter(
child.getOutputStream()));
out.write("exit\r\n");
out.flush();
child.waitFor();
}
private static class StreamGobbler extends Thread {
private final InputStream inputStream;
private final String name;
public StreamGobbler(InputStream inputStream, String name) {
this.inputStream = inputStream;
this.name = name;
}
public void run() {
try {
BufferedReader in = new BufferedReader(new InputStreamReader(
inputStream));
for (String s = in.readLine(); s != null; s = in.readLine()) {
System.out.println(name + ": " + s);
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
Somehow the program in question (process) is recieving an EOF right away (as in right after I step pas the "exec" line) and thus throwing an error ( detected, invalid) message immediately after runtime.exec is called. I can run this program manually via command prompt without this issue, but have confirmed that sending a ctrl-z on windows is what causes this message.
Anyone know what could be causing this?
If it matters, I have tried running the process directly as "test.exe" instead of cmd /c test.exe, but when I do that I can't see the output via the inputStream. And when I do cmd test.exe without the /c, there is no difference.
Your code looks like it should work (with one caveat, see below).
I took your code verbatim and replaced test.ext with sort, which can read from piped stdin.
If I run the code as-is, it starts the sort command, which waits for input. It hangs at child.waitFor() because you don't close the output stream to indicate EOF. When I add the close() call, everything works correctly.
I suggest you look at test.exe and determine if it is capable of reading from piped stdin, or is expecting console input.
Get rid of "cmd" and "/c". At present you are feeding output to cmd.exe, not to test.exe.