I'm really stuck here, I've read number of articles and answers on stackoverflow, but nothing solved my problem.
I've got the method which runs Selenium Server Hub from cmd.exe using batch file:
public static boolean startGrid() throws IOException {
ProcessBuilder builder = new ProcessBuilder("cmd", "/c", "start", START_GRID_BAT_PATH);
builder.redirectErrorStream(true);
Process process = builder.start();
String out = getCmdOutput(process);
return out.contains("Selenium Grid hub is up and running");
}
The server is started successfully and is running in open cmd.
To make sure that server is up I use the following method to get the output from the cmd:
protected static String getCmdOutput(Process proc) throws java.io.IOException {
String res = "";
BufferedReader stdInput = new BufferedReader(new
InputStreamReader(proc.getInputStream()));
String s = "";
while ((s = stdInput.readLine()) != null) {
res += s;
}
return res;
}
And here is where the problem starts - method hangs at the the line s = stdInput.readLine()) != null
It seems that it can't read any line from the cmd, although I can see that there is number of lines in the output produced by running server.
Any ideas what is wrong?
Maybe the output produced by the server doesn't contain any \r or \n characters but is simply line-wrapped.
Anyhow, you shouldn't rely on the line feed characters when reading the output.
The following example shows how to read binary data from the InputStream of the process non-blocking with java.nio, convert the bytes to chars and write the result to a StringWriter - which is preferrable to using string concatenation.
protected static String getCmdOutput(Process proc, Consumer<String> consumer)
final InputStream source = proc.getInputStream();
final StringWriter sink = new StringWriter();
final ByteBuffer buf = ByteBuffer.allocate(1024);
final CharBuffer chars = CharBuffer.allocate(1024);
final CharsetDecoder decoder = Charset.defaultCharset().newDecoder();
try(final ReadableByteChannel from = Channels.newChannel(source)) {
while (from.read(buf) != -1) {
buf.flip();
decoder.decode(buf, chars, false);
chars.flip();
sink.write(chars.array(), chars.position(), chars.remaining());
buf.compact();
chars.clear();
}
} catch (IOException e) {
e.printStackTrace();
}
return sink.toString();
}
Edit:
As you are starting a process that is continuously producing output, the loop won't finish as long as the process is running. So to continue your parent process, you have to start a thread that is capturing and processing the output of the process continuously
For example
Consumer<String> consumer = ...;
Thread t = new Thread(() -> {
final InputStream source = proc.getInputStream();
final StringWriter sink = new StringWriter();
final ByteBuffer buf = ByteBuffer.allocate(1024);
final CharBuffer chars = CharBuffer.allocate(1024);
final CharsetDecoder decoder = Charset.defaultCharset().newDecoder();
try(final ReadableByteChannel from = Channels.newChannel(source)) {
while (from.read(buf) != -1) {
buf.flip();
decoder.decode(buf, chars, false);
chars.flip();
sink.write(chars.array(), chars.position(), chars.remaining());
forward(sink, consumer); //forward the captured content to a consumer
buf.compact();
chars.clear();
}
} catch (IOException e) {
e.printStackTrace();
}
});
t.start();
private void forward(StringWriter sink, Consumer<String> consumer) {
StringBuffer buf = sink.getBuffer();
int pos = buf.lastIndexOf("\n");
//you may use other options for triggering the process here,
//i.e. on every invocation, on fixed length, etc
if(pos != -1){
consumer.accept(buf.substring(0, pos + 1));
buf.delete(0, pos + 1);
}
//or to send the entire buffer
/*
consumer.accept(sink.toString());
sink.getBuffer().delete(0, sink.getBuffer().length());
*/
}
Depending on your use case, you may not necessarily need a separate thread. It might be ok to process the child-process' output in the main thread and do what you want with the output in the consumer.
It doesn't hang, it just blocks waiting for a line to be read. And after the line has been read, it waits for the next. And so on.
Getting the output stream of a child process should be made in a separate thread, because the child process could in fact hang, than your main thread will hang, too.
An example:
given the instance methods
private static final long WAITTIME_FOR_CHILD = Duration.ofMinutes(5).toNanos();
private static final String START_SEQUENCE = "whatever";
private final Lock lock = new ReentrantLock();
private final Condition waitForStart = lock.newCondition();
private boolean started;
the code that starts the child process
// prepare child process
lock.lock();
try {
Process process = processBuilder.start();
// both standard and error output streams get redirected
pipe(process.getInputStream(), System.out, START_SEQUENCE);
pipe(process.getErrorStream(), System.err, START_SEQUENCE);
// loop because of 'spurious wakeups' - should happen only on Linux,
// but better take care of it
// wait until WAITTIME_FOR_CHILD (default 5 minutes, see above)
long waitTime = WAITTIME_FOR_CHILD;
while (!started) {
// we wait hier until the child process starts or timeout happens
// value <= 0 means timeout
waitTime = waitForStart.awaitNanos(waitTime);
if (waitTime <= 0) {
process.destroyForcibly();
throw new IOException("Prozess xxx couldn't be started");
}
}
} catch (IOException | InterruptedException e) {
throw new RuntimeException("Error during start of xxx", e);
} finally {
lock.unlock();
}
private void getCmdOutput(InputStream in, PrintStream out, final String startSeq) {
startDaemon(() -> {
try (Scanner sc = new Scanner(in)) {
while (sc.hasNextLine()) {
String line = sc.nextLine();
// check your know output sequence
if (line.contains(startSeq)) {
lock.lock();
try {
if (!started) {
started = true;
// waiting for process start finished
waitForStart.signal();
}
} finally {
lock.unlock();
}
}
out.println(line);
}
}
});
}
private void startDaemon(Runnable task) {
Thread thread = new Thread(task);
thread.setDaemon(true);
thread.start();
}
Try replacing while condition like this:
while ((s = stdInput.readLine()) != null && !s.equals("")) {
ProcessBuilder pb =
new ProcessBuilder("myCommand", "myArg1", "myArg2");
Map<String, String> env = pb.environment();
env.put("VAR1", "myValue");
env.remove("OTHERVAR");
env.put("VAR2", env.get("VAR1") + "suffix");
pb.directory(new File("myDir"));
File log = new File("log");
pb.redirectErrorStream(true);
pb.redirectOutput(Redirect.appendTo(log));
Process p = pb.start();
assert pb.redirectInput() == Redirect.PIPE;
assert pb.redirectOutput().file() == log;
assert p.getInputStream().read() == -1;
You can redirect output to a file as oracle documentation offers in basic example:
https://docs.oracle.com/javase/7/docs/api/java/lang/ProcessBuilder.html
Related
I would like to print live output of a process in java; but the output is (almost) only available when the process stops execution.
In other words, when I run the same process with the command line, I get more frequent feedback (output) than when I execute it with Java.
I tested the following code to print the output every 250ms:
private static String printStream(BufferedReader bufferedReader) {
try {
String line = bufferedReader.readLine();
if(line != null) {
System.out.println(line);
}
return line;
} catch (IOException ex) {
ex.printStackTrace();
}
return null;
}
public static void sendProcessStreams(Process process, SendMessage sendMessage) {
String[] command = { "/bin/bash", "-c", "custom_process"};
ProcessBuilder processBuilder = new ProcessBuilder(command);
processBuilder.directory(new File("."));
Process process = processBuilder.start();
BufferedReader inputBufferedReader = new BufferedReader(new InputStreamReader(process.getInputStream()), 1);
BufferedReader errorBufferedReader = new BufferedReader(new InputStreamReader(process.getErrorStream()), 1);
System.out.println("Start reading process");
Timer timer = new Timer();
timer.scheduleAtFixedRate(new TimerTask() {
#Override
public void run() {
String inputLine = printStream(inputBufferedReader);
String errorLine = printStream(errorBufferedReader);
if(inputLine == null && errorLine == null && !process.isAlive()) {
timer.cancel();
}
}
}, 0, 250);
}
Yet it only prints two lines, and then waits for the process to end before printing everything else.
Is it possible to get more frequent feedback from an external process? If not, Why not?
Your code basically (as far as reading from the output) works correctly, the problem is surely for another reason, please build a minimal problem that is reproducible (eg. what is custom_process, why you use Timer when you can use the current thread, ...).
Anyway, here's an example reading in realtime the output:
final Process proc = new ProcessBuilder(
"/bin/bash", "-c",
"for i in `seq 1 10`; do echo $i; sleep $((i % 2)); done")
.start();
try(InputStreamReader isr = new InputStreamReader(proc.getInputStream())) {
int c;
while((c = isr.read()) >= 0) {
System.out.print((char) c);
System.out.flush();
}
}
With output:
1 (and wait one second)
2
3 (and wait one second)
4
5 (and wait one second)
6
7 (and wait one second)
8
9 (and wait one second)
10
it seems like you are dealing with multithreading-problems and not with getting the output of the process.
I just made this demo class that you can use:
CommandExecTest.java
import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.PrintWriter;
import java.util.ArrayList;
public class CommandExecTest {
public static void main(String[] args) throws InterruptedException {
String executable = "cmd";
String[] commandParams = {"#ping -n 5 localhost","echo \"hello world\"","exit 123"};
boolean passCommandsAsLinesToShellExecutableAfterStartup = true;
AsyncExecutor asyncExecutor = new AsyncExecutor(executable, commandParams,passCommandsAsLinesToShellExecutableAfterStartup);
System.out.println("x"+"/x\tsecs in main thread \t\t status:"+asyncExecutor.runstate+" of async thread that monitors the process");
asyncExecutor.start();//start() invokes the run() method as a detached thread
for(int i=0;i<10;i++) {
// you can do whatever here and the other process is still running and printing its output inside detached thread
Thread.sleep(1000);
System.out.println(i+"/10\tsecs in main thread \t\t status:"+asyncExecutor.runstate+" of async thread that monitors the process");
}
asyncExecutor.join(); // main thread has nothing to do anymore, wait till other thread that monitor other process finishes as well
System.out.println("END OWN-PROGRAMM: 0 , END OTHER PROCESS:"+asyncExecutor.processExitcode);
System.exit(0);
}
}
Runstate.java
public static enum Runstate {
CREATED, RUNNING, STOPPED
}
AsyncExecutor.java
public static class AsyncExecutor extends Thread{
private String executable;
private String[] commandParams;
public ArrayList<String> linesSoFarStdout = new ArrayList<>();
public ArrayList<String> linesSoFarStderr = new ArrayList<>();
public Runstate runstate;
public int processExitcode=-1;
private boolean passCommandsAsLinesToShellExecutableAfterStartup = false;
public AsyncExecutor(String executable, String[] commandParams) {
this.executable=executable;
this.commandParams=commandParams;
this.runstate=Runstate.CREATED;
this.passCommandsAsLinesToShellExecutableAfterStartup=false;
}
/**
* if you want to run a single-process with arguments use <b>false</b> example executable="java" commandParams={"-jar","myjarfile.jar","arg0","arg1"}
* <p>
* if you want to run a shell-process and enter commands afterwards use <b>true</b> example executable="cmd" commandParams={"#ping -n 5 localhost","echo \"hello world\"","exit 123"}
* #param executable
* #param commandParams
* #param passCommandsAsLinesToShellExecutableAfterStartup
*/
public AsyncExecutor(String executable, String[] commandParams, boolean passCommandsAsLinesToShellExecutableAfterStartup) {
this.executable=executable;
this.commandParams=commandParams;
this.runstate=Runstate.CREATED;
this.passCommandsAsLinesToShellExecutableAfterStartup=passCommandsAsLinesToShellExecutableAfterStartup;
}
#Override
public void run() {
this.runstate=Runstate.RUNNING;
// 1 start the process
Process p = null;
try {
if(passCommandsAsLinesToShellExecutableAfterStartup) {
// open a shell-like process like cmd and pass the arguments/command after opening it
// * example:
// * open 'cmd' (shell)
// * write 'echo "hello world"' and press enter
p = Runtime.getRuntime().exec(new String[] {executable});
PrintWriter stdin = new PrintWriter( p.getOutputStream());
for( int i = 0; i < commandParams.length; i++) {
String commandstring = commandParams[i];
stdin.println( commandstring);
}
stdin.close();
}
else {
// pass the arguments directly during startup to the process
// * example:
// * run 'java -jar myexecutable.jar arg0 arg1 ...'
String[] execWithArgs = new String[commandParams.length+1];
execWithArgs[0] = executable;
for(int i=1;i<=commandParams.length;i++) {
execWithArgs[i]=commandParams[i-1];
}
p = Runtime.getRuntime().exec( execWithArgs);
}
// 2 print the output
InputStream is = p.getInputStream();
BufferedReader br = new BufferedReader( new InputStreamReader( is));
InputStream eis = p.getErrorStream();
BufferedReader ebr = new BufferedReader( new InputStreamReader( eis));
String lineStdout=null;
String lineStderr=null;
while(p.isAlive()) {
Thread.yield(); // *
// * free cpu clock for other tasks on your PC! maybe even add thread.sleep(milliseconds) to free some more
// * everytime this thread gets cpu clock it will try the following codeblock inside the while and yield afterwards for the next time it gets cpu-time from sheduler
while( (lineStdout = br.readLine()) != null || (lineStderr = ebr.readLine()) != null) {
if(lineStdout!=null) {
System.out.println(lineStdout);
linesSoFarStdout.add(lineStdout);
}
if(lineStderr!=null) {
System.out.println(lineStderr);
linesSoFarStderr.add(lineStderr);
}
}
}
// 3 when process ends
this.processExitcode = p.exitValue();
}
catch(Exception e) {
System.err.println("Something went wrong!");
e.printStackTrace();
}
if(processExitcode!=0) {
System.err.println("The other process stopped with unexpected existcode: " + processExitcode);
}
this.runstate=Runstate.STOPPED;
}
}
We are the using the following code to generate PDFs using wkhtmltopdf
public class SystemUtils{
public String executeCommand(String... command) {
Process process = null;
try {
// Using redirectErrorStream as true. Otherwise we have to read both process.getInputStream() and
// process.getErrorStream() in order to not exhaust the stream buffer.
process = new ProcessBuilder(command).redirectErrorStream(true).start();
process.waitFor();
StringBuilder outputBuilder = new StringBuilder();
try(BufferedReader stdError = new BufferedReader(new InputStreamReader(process.getInputStream()))) {
String line;
while ((line = stdError.readLine()) != null) {
outputBuilder.append(line).append(StringConstants.CARRIAGE_RETURN);
}
}
return outputBuilder.toString();
} catch (IOException | InterruptedException e) {
String exceptionMsg = "Error while executing command '"+command+"' : ";
LOGGER.error(exceptionMsg, e);
throw new AppException(exceptionMsg, e);
} finally {
if(process != null){
process.destroy();
}
}
}
public static void main(String[] args){
SystemUtils systemUtils = new SystemUtils();
String[] array = {"wkhtmltopdf", "/home/pgullapalli/Desktop/testsimilar1.html", "/home/pgullapalli/Desktop/test.pdf"};
systemUtils.executeCommand(array);
}
}
This works absolutely fine for smaller size files. But when we try to process a larger file, it is indefinitely waiting without any response. I am not sure what is going wrong? Can someone please suggest?
I moved process.waitFor() before the return statement and it started working. This probably could be happening as the output buffer has filled and we are not reading from it. After moving the process.waitFor after the stream reading, things are working fine.
I'm currently working on a project where a client receives shell/console commands from a server, and must execute them.
How do I get Java to run these commands from within either a shell or a command prompt? I'm hoping to be able to disregard the platform type - and not have to specify shell or command prompt - but if I can't, then that's okay.
I must be able to send a sequence of related commands, not just one command. This means that the shell/prompt cannot exit or close between commands.
My current code, as follows, allows for the execution of a sequence of programs, but these commands must somehow be piped into a shell/command prompt, from which the output must be read.
ArrayList<String> comDat = new ArrayList<>();
while(true) {
String input = con.recv();
System.out.println("> " + input);
if(!input.equals("EOF")) comDat.add(input); else {
String[] s = new String[comDat.size()];
for(int i = 0; i < comDat.size(); i++) s[i] = comDat.get(i);
System.out.println("---Command sequence executing---");
Process p = Runtime.getRuntime().exec(s);
p.waitFor();
System.out.println("---ErrorStream output---"); String line = "";
BufferedReader errStream = new BufferedReader(new InputStreamReader(p.getErrorStream()));
while((line = errStream.readLine()) != null) System.out.println("< " + line);
System.out.println("\n---OutputStream output---"); line = "";
BufferedReader outStream = new BufferedReader(new InputStreamReader(p.getInputStream()));
while((line = errStream.readLine()) != null) System.out.println("< " + line);
}
Thread.sleep(200);
}
Thanks for the help!
The basic premise revoles around the fact the dir isn't an external command but is function of cmd.
I would avoid BufferedReaders when reading the output of a process as not all processes use new lines when sending output (such as progress indicators), instead you should read char for char (IMHO).
You should us ProcessBuilder instead of Runtime#exec. It provides better management and allows you to redirect the error stream into the input stream, making it easier to read the input.
import java.io.IOException;
import java.io.InputStream;
public class TestProcessBuilder {
public static void main(String[] args) {
try {
ProcessBuilder pb = new ProcessBuilder("cmd", "/c", "dir");
pb.redirectError();
Process p = pb.start();
InputStreamConsumer isc = new InputStreamConsumer(p.getInputStream());
isc.start();
int exitCode = p.waitFor();
isc.join();
System.out.println("Process terminated with " + exitCode);
} catch (IOException | InterruptedException exp) {
exp.printStackTrace();
}
}
public static class InputStreamConsumer extends Thread {
private InputStream is;
public InputStreamConsumer(InputStream is) {
this.is = is;
}
#Override
public void run() {
try {
int value = -1;
while ((value = is.read()) != -1) {
System.out.print((char)value);
}
} catch (IOException exp) {
exp.printStackTrace();
}
}
}
}
I'd like to get the output from a long running shell command as it is available instead of waiting for the command to complete. My code is run in a new thread
Process proc = Runtime.getRuntime().exec("/opt/bin/longRunning");
InputStream in = proc.getInputStream();
int c;
while((c = in.read()) != -1) {
MyStaticClass.stringBuilder.append(c);
}
The problem with this is that my program in /opt/bin/longRunning has to complete before the InputStream gets assigned and read. Is there any good way to do this asynchronously? My goal is that an ajax request will return the current value MyStaticClass.stringBuilder.toString()
every second or so.
I'm stuck on Java 5, fyi.
Thanks!
W
Try with Apache Common Exec. It has the ability to asynchronously execute a process and then "pump" the output to a thread. Check the Javadoc for more info
Runtime.getRuntime().exec does not wait for the command to terminate, so you should be getting the output straight away. Maybe the output is being buffered because the command knows it is writing to a pipe rather than a terminal?
Put the reading in a new thread:
new Thread() {
public void run() {
InputStream in = proc.getInputStream();
int c;
while((c = in.read()) != -1) {
MyStaticClass.stringBuilder.append(c);
}
}
}.start();
Did you write the program you're calling? If so try flushing your output after writing. The text could be stuck in a buffer and not getting to your java program.
I use this code to do this and it works:
Runtime runtime = Runtime.getRuntime();
Process proc = runtime.exec(command);
BufferedReader br = new BufferedReader(new InputStreamReader(proc.getInputStream()));
while (true) {
// enter a loop where we read what the program has to say and wait for it to finish
// read all the program has to say
while (br.ready()) {
String line = br.readLine();
System.out.println("CMD: " + line);
}
try {
int exitCode = proc.exitValue();
System.out.println("exit code: " + exitCode);
// if we get here then the process finished executing
break;
} catch (IllegalThreadStateException ex) {
// ignore
}
// wait 200ms and try again
Thread.sleep(200);
}
Try :
try {
Process p = Runtime.getRuntime().exec("/opt/bin/longRunning");
BufferedReader in = new BufferedReader(
new InputStreamReader(p.getInputStream()));
String line = null;
while ((line = in.readLine()) != null) {
System.out.println(line); }
I would like to execute foo.bat from within a Groovy program and have the resulting process' output redirected to stdout. Either a Java or Groovy code example would be fine.
foo.bat can take several minutes to run and generates a lot of output, so I would like to see the output as soon as it is generated, rather than having to wait until the process has completed before seeing all the output at once.
It is simple to redirect all your stream to standard output using inheritIO() method. This will print the output to the stdout of the process from which you are running this command.
ProcessBuilder pb = new ProcessBuilder("command", "argument");
pb.directory(new File(<directory from where you want to run the command>));
pb.inheritIO();
Process p = pb.start();
p.waitFor();
There exist other methods too, like as mentioned below. These individual methods will help redirect only required stream.
pb.redirectInput(Redirect.INHERIT)
pb.redirectOutput(Redirect.INHERIT)
pb.redirectError(Redirect.INHERIT)
This uses a class which reads all output the executed program generates and displays it in it's own stdout.
class StreamGobbler extends Thread {
InputStream is;
// reads everything from is until empty.
StreamGobbler(InputStream is) {
this.is = is;
}
public void run() {
try {
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line=null;
while ( (line = br.readLine()) != null)
System.out.println(line);
} catch (IOException ioe) {
ioe.printStackTrace();
}
}
}
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec("javac");
//output both stdout and stderr data from proc to stdout of this process
StreamGobbler errorGobbler = new StreamGobbler(proc.getErrorStream());
StreamGobbler outputGobbler = new StreamGobbler(proc.getInputStream());
errorGobbler.start();
outputGobbler.start();
proc.waitFor();
If you're looking to do this with more Groovy and less java, this will print each line as it happens:
def cmd = "./longRunningProcess"
def process = cmd.execute()
process.in.eachLine { line -> println line }
Alternatively, if you want to see both stdout and stderr
def cmd = "./longRunningProcess"
def process = cmd.execute()
process.waitForProcessOutput( System.out, System.err )
Here's something a little simpler if you're just trying to grab the output of a simple command. You'll need to use threads like jitter does if you want to process in parallel or if your command takes stdin or generates stderr.
Use a buffered copy (like this) if you're getting lots of output.
import java.io.*;
public class test {
static void copy(InputStream in, OutputStream out) throws IOException {
while (true) {
int c = in.read();
if (c == -1) break;
out.write((char)c);
}
}
public static void main(String[] args) throws IOException, InterruptedException {
String cmd = "echo foo";
Process p = Runtime.getRuntime().exec(cmd);
copy(p.getInputStream(), System.out);
p.waitFor();
}
}
The following Groovy code will execute foo.bat and send the output to stdout:
println "foo.bat".execute().text
Asynchronous way to achieve it.
void inputStreamToOutputStream(final InputStream inputStream, final OutputStream out) {
Thread t = new Thread(new Runnable() {
public void run() {
try {
int d;
while ((d = inputStream.read()) != -1) {
out.write(d);
}
} catch (IOException ex) {
//TODO make a callback on exception.
}
}
});
t.setDaemon(true);
t.start();
}
{
Process p = ...;
inputStreamToOutputStream(p.getErrorStream(), System.out);
inputStreamToOutputStream(p.getInputStream(), System.out);
}
VerboseProcess from jcabi-log can help you:
String output = new VerboseProcess(new ProcessBuilder("foo.bat")).stdout();