Is there any sane way to run a system command from Java that ignores STDOUT and STDERR? For example, when I try the following:
Process p = Runtime.getRuntime().exec("some_executable_path param1 param2 >NUL 2>&1");
Java tries to parse the command, and ends up escaping the arguments (e.g., prevents the output from being redirected). If I don't redirect STDOUT/STDERR, the buffers fill up and prevent the system call from exiting. The following does what I want, but is extremely cumbersome and creates expensive resources just to throw the output of the system call away:
ProcessBuilder pb = new ProcessBuilder("some_executable_path", "param1", "param2");
pb.redirectErrorStream(true);
final Process p = pb.start();
final Thread redirectToNull = new Thread(() -> {
final InputStream stdout = process.getInputStream();
try {
while (stdout.read() != -1);
} catch (final Exception e) {
// Don't care
}
}, "Output Consumer Thread");
redirectToNull.setDaemon(true);
redirectToNull.start();
I realize the Java design team is known to be masochistic, but this is ridiculous. I would prefer to deliver a batch or Perl script that wraps the system call with my application rather than use the above code. There has to be an easier way to accomplish this.
So the question is, is there any sane way to run a system command from within Java and ignore the output printed to STDOUT/STDERR?
It's not that Java 'prevents' redirection, it just doesn't affirmatively do it, and neither does your program. When you give CMD a command like program arg1 arg2 >out 2>err <in, it is CMD that sets up those redirections and then invokes program with arg1 arg2 only, not >out etc. On Unix the shells do the same -- there is a choice of several shells, but all of them handle redirection like this. Similarly pipes are set up by CMD or shells, not by either or all of the programs run in those pipes.
Thus on Windows the way to do this is either run CMD and have it do the redirections:
Process p = new ProcessBuilder ("cmd", "/c", "program arg1 arg2 >NUL 2>&1").start();
// this uses CMD's default parsing for args, so they must not contain space
// unless you insert 'data' quotes, or things that look like a substitutable %var%
or (assuming Java7+) tell ProcessBuilder to do the redirections:
pb.redirectOutput (new File("NUL")).redirectErrorStream(true)
Related
When I try to run Runtime.exec(String), certain commands work, while other commands are executed but fail or do different things than in my terminal. Here is a self-contained test case that demonstrates the effect:
public class ExecTest {
static void exec(String cmd) throws Exception {
Process p = Runtime.getRuntime().exec(cmd);
int i;
while( (i=p.getInputStream().read()) != -1) {
System.out.write(i);
}
while( (i=p.getErrorStream().read()) != -1) {
System.err.write(i);
}
}
public static void main(String[] args) throws Exception {
System.out.print("Runtime.exec: ");
String cmd = new java.util.Scanner(System.in).nextLine();
exec(cmd);
}
}
The example works great if I replace the command with echo hello world, but for other commands -- especially those involving filenames with spaces like here -- I get errors even though the command is clearly being executed:
myshell$ javac ExecTest.java && java ExecTest
Runtime.exec: ls -l 'My File.txt'
ls: cannot access 'My: No such file or directory
ls: cannot access File.txt': No such file or directory
meanwhile, copy-pasting to my shell:
myshell$ ls -l 'My File.txt'
-rw-r--r-- 1 me me 4 Aug 2 11:44 My File.txt
Why is there a difference? When does it work and when does it fail? How do I make it work for all commands?
Why do some commands fail?
This happens because the command passed to Runtime.exec(String) is not executed in a shell. The shell performs a lot of common support services for programs, and when the shell is not around to do them, the command will fail.
When do commands fail?
A command will fail whenever it depends on a shell features. The shell does a lot of common, useful things we don't normally think about:
The shell splits correctly on quotes and spaces
This makes sure the filename in "My File.txt" remains a single argument.
Runtime.exec(String) naively splits on spaces and would pass this as two separate filenames. This obviously fails.
The shell expands globs/wildcards
When you run ls *.doc, the shell rewrites it into ls letter.doc notes.doc.
Runtime.exec(String) doesn't, it just passes them as arguments.
ls has no idea what * is, so the command fails.
The shell manages pipes and redirections.
When you run ls mydir > output.txt, the shell opens "output.txt" for command output and removes it from the command line, giving ls mydir.
Runtime.exec(String) doesn't. It just passes them as arguments.
ls has no idea what > means, so the command fails.
The shell expands variables and commands
When you run ls "$HOME" or ls "$(pwd)", the shell rewrites it into ls /home/myuser.
Runtime.exec(String) doesn't, it just passes them as arguments.
ls has no idea what $ means, so the command fails.
What can you do instead?
There are two ways to execute arbitrarily complex commands:
Simple and sloppy: delegate to a shell.
You can just use Runtime.exec(String[]) (note the array parameter) and pass your command directly to a shell that can do all the heavy lifting:
// Simple, sloppy fix. May have security and robustness implications
String myFile = "some filename.txt";
String myCommand = "cp -R '" + myFile + "' $HOME 2> errorlog";
Runtime.getRuntime().exec(new String[] { "bash", "-c", myCommand });
Secure and robust: take on the responsibilities of the shell.
This is not a fix that can be mechanically applied, but requires an understanding the Unix execution model, what shells do, and how you can do the same. However, you can get a solid, secure and robust solution by taking the shell out of the picture. This is facilitated by ProcessBuilder.
The command from the previous example that requires someone to handle 1. quotes, 2. variables, and 3. redirections, can be written as:
String myFile = "some filename.txt";
ProcessBuilder builder = new ProcessBuilder(
"cp", "-R", myFile, // We handle word splitting
System.getenv("HOME")); // We handle variables
builder.redirectError( // We set up redirections
ProcessBuilder.Redirect.to(new File("errorlog")));
builder.start();
When I try to run Runtime.exec(String), certain commands work, while other commands are executed but fail or do different things than in my terminal. Here is a self-contained test case that demonstrates the effect:
public class ExecTest {
static void exec(String cmd) throws Exception {
Process p = Runtime.getRuntime().exec(cmd);
int i;
while( (i=p.getInputStream().read()) != -1) {
System.out.write(i);
}
while( (i=p.getErrorStream().read()) != -1) {
System.err.write(i);
}
}
public static void main(String[] args) throws Exception {
System.out.print("Runtime.exec: ");
String cmd = new java.util.Scanner(System.in).nextLine();
exec(cmd);
}
}
The example works great if I replace the command with echo hello world, but for other commands -- especially those involving filenames with spaces like here -- I get errors even though the command is clearly being executed:
myshell$ javac ExecTest.java && java ExecTest
Runtime.exec: ls -l 'My File.txt'
ls: cannot access 'My: No such file or directory
ls: cannot access File.txt': No such file or directory
meanwhile, copy-pasting to my shell:
myshell$ ls -l 'My File.txt'
-rw-r--r-- 1 me me 4 Aug 2 11:44 My File.txt
Why is there a difference? When does it work and when does it fail? How do I make it work for all commands?
Why do some commands fail?
This happens because the command passed to Runtime.exec(String) is not executed in a shell. The shell performs a lot of common support services for programs, and when the shell is not around to do them, the command will fail.
When do commands fail?
A command will fail whenever it depends on a shell features. The shell does a lot of common, useful things we don't normally think about:
The shell splits correctly on quotes and spaces
This makes sure the filename in "My File.txt" remains a single argument.
Runtime.exec(String) naively splits on spaces and would pass this as two separate filenames. This obviously fails.
The shell expands globs/wildcards
When you run ls *.doc, the shell rewrites it into ls letter.doc notes.doc.
Runtime.exec(String) doesn't, it just passes them as arguments.
ls has no idea what * is, so the command fails.
The shell manages pipes and redirections.
When you run ls mydir > output.txt, the shell opens "output.txt" for command output and removes it from the command line, giving ls mydir.
Runtime.exec(String) doesn't. It just passes them as arguments.
ls has no idea what > means, so the command fails.
The shell expands variables and commands
When you run ls "$HOME" or ls "$(pwd)", the shell rewrites it into ls /home/myuser.
Runtime.exec(String) doesn't, it just passes them as arguments.
ls has no idea what $ means, so the command fails.
What can you do instead?
There are two ways to execute arbitrarily complex commands:
Simple and sloppy: delegate to a shell.
You can just use Runtime.exec(String[]) (note the array parameter) and pass your command directly to a shell that can do all the heavy lifting:
// Simple, sloppy fix. May have security and robustness implications
String myFile = "some filename.txt";
String myCommand = "cp -R '" + myFile + "' $HOME 2> errorlog";
Runtime.getRuntime().exec(new String[] { "bash", "-c", myCommand });
Secure and robust: take on the responsibilities of the shell.
This is not a fix that can be mechanically applied, but requires an understanding the Unix execution model, what shells do, and how you can do the same. However, you can get a solid, secure and robust solution by taking the shell out of the picture. This is facilitated by ProcessBuilder.
The command from the previous example that requires someone to handle 1. quotes, 2. variables, and 3. redirections, can be written as:
String myFile = "some filename.txt";
ProcessBuilder builder = new ProcessBuilder(
"cp", "-R", myFile, // We handle word splitting
System.getenv("HOME")); // We handle variables
builder.redirectError( // We set up redirections
ProcessBuilder.Redirect.to(new File("errorlog")));
builder.start();
I want to start a processes such that the JVM can die but the spawned processes continues to run even if it is writing to STDOUT.
I first tried using a ProcessBuilder with the output set to Files and passing in:
cmd /c myCmd.exe arg0 arg1
However even after closing all Input/Output streams, if I call Process#.waitFor, it does not return until myCmd.exe has finished. It seems it is still attached to the JVM in some way (even though the JVM can probably die at this point and not affect the child proc).
I then tried the start command, it seems that is not on the path (I couldn't find the bin in c:\windows) so I ran it under cmd the arguments (separated by space) passed to ProcessBuilder became:
cmd /c start /b myCmd.exe arg0 arg2 >log 2>&1
That results in:
✓ Process#.waitFor returning before myCmd.exe finished.
⚠ It seemed that I needed to use a different log file from the one passed to the ProcessBuilder
✘ I then found the escaping become weird if the command run was echo and the argument was ^^^^\foo it would write to the log file ^\foo, I also noticed if I gave it "^^^^\foo" it would return the same thing ie "^^^^\foo".
So:
Is calling cmd.exe /c start /b the correct thing to do?
Am I doing something wrong with the escaping (which is really what I give to process builder), should I perhaps be doing something different because of cmd.exe calling start, perhaps I need to actually escape in some way? Perhaps I don't understand windows processes do they even have proper support for taking an array of arguments?
Am I going about this the wrong way should I be trying to call a native library from C? If so what would it be I don't mind if I have to call a C program to get my process running in the background.
I think the solution to create a detached in background process which the JVM holds no references to which also supports the possibility to pass any arguments in a sane way is it to use the CreateProcess API. For this I used:
<dependency>
<groupId>net.java.dev.jna</groupId>
<artifactId>platform</artifactId>
<version>3.5.0</version>
</dependency>
(It is an older version but it happened to be already in use).
The Jave code to get it working is:
/**
*
* #param command a pre escaped command line e.g. c:\perl.exe c:\my.pl arg
* #param env A non null environment.
* #param stdoutFile
* #param stderrFile
* #param workingDir
*/
public void execute(String command, Map<String, String> env,
File stdoutFile, File stderrFile, String workingDir) {
WinBase.SECURITY_ATTRIBUTES sa = new WinBase.SECURITY_ATTRIBUTES();
sa.bInheritHandle = true; // I think the child processes gets handles I make with
// with this sa.
sa.lpSecurityDescriptor = null; // Use default access token from current proc.
HANDLE stdout = makeFileHandle(sa, stdoutFile);
HANDLE stderr = null;
if(stderrFile != null &&
!stderrFile.getAbsolutePath().equals(stdoutFile.getAbsolutePath())) {
stderr = makeFileHandle(sa, stderrFile);
}
try {
WinBase.STARTUPINFO si = new WinBase.STARTUPINFO();
// Assume si.cb is set by the JVM.
si.dwFlags |= WinBase.STARTF_USESTDHANDLES;
si.hStdInput = null; // No stdin for the child.
si.hStdOutput = stdout;
si.hStdError = Optional.ofNullable(stderr).orElse(stdout);
DWORD dword = new DWORD();
dword.setValue(WinBase.CREATE_UNICODE_ENVIRONMENT | // Probably makes sense.
WinBase.CREATE_NO_WINDOW | // Well we don't want a window so this makes sense.
WinBase.DETACHED_PROCESS); // I think this would let the JVM die without stopping the child
// Newer versions of platform don't use a reference.
WinBase.PROCESS_INFORMATION.ByReference processInfoByRef = new WinBase.PROCESS_INFORMATION.ByReference();
boolean result = Kernel32.INSTANCE
.CreateProcess(null, // use next argument to get the task to run
command,
null, // Don't let the child inherit a handle to itself, because I saw someone else do it.
null, // Don't let the child inherit a handle to its own main thread, because I saw someone else do it.
true, // Must be true to pass any handle to the spawned thread including STDOUT and STDERR
dword,
asPointer(createEnvironmentBlock(env)), // I hope that the new processes copies this memory
workingDir,
si,
processInfoByRef);
// Put code in try block.
try {
// Did it start?
if(!result) {
throw new RuntimeException("Could not start command: " + command);
}
} finally {
// Apparently both parent and child need to close the handles.
Kernel32.INSTANCE.CloseHandle(processInfoByRef.hProcess);
Kernel32.INSTANCE.CloseHandle(processInfoByRef.hThread);
}
} finally {
// We need to close this
// https://stackoverflow.com/questions/6581103/do-i-have-to-close-inherited-handle-later-owned-by-child-process
Kernel32.INSTANCE.CloseHandle(stdout);
if(stderr != null) {
Kernel32.INSTANCE.CloseHandle(stderr);
}
}
}
private HANDLE makeFileHandle(WinBase.SECURITY_ATTRIBUTES sa, File file) {
return Kernel32.INSTANCE
.CreateFile(file.getAbsolutePath(),
Kernel32.FILE_APPEND_DATA, // IDK I saw this in an example.
Kernel32.FILE_SHARE_WRITE | Kernel32.FILE_SHARE_READ,
sa,
Kernel32.OPEN_ALWAYS,
Kernel32.FILE_ATTRIBUTE_NORMAL,
null);
}
public static byte[] createEnvironmentBlock(Map<String, String> env) {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
// This charset seems to work.
Charset charset = StandardCharsets.UTF_16LE;
try {
for(Entry<String, String> entry : env.entrySet()) {
bos.write(entry.getKey().getBytes(charset));
bos.write("=".getBytes(charset));
bos.write(entry.getValue().getBytes(charset));
bos.write(0);
bos.write(0);
}
} catch (Exception e) {
throw new RuntimeException(e);
}
bos.write(0);
bos.write(0);
return bos.toByteArray();
}
public static Pointer asPointer(byte[] data) {
Pointer pointer = new Memory(data.length);
pointer.write(0, data, 0, data.length);
return pointer;
}
✓ The process started seems to keep running even if the JVM is stopped.
✓ I didn't need to have to deal with STDOUT/STDERR from ProcessBuilder and later another one from needing to redirect the command actually run.
✓ If you can correctly escape your command (code to do that is not in the answer as it is not mine to share) you can pass things like " which I could not work out how to do when using ProcessBuilder with cmd /c start /b command. It seemed the JVM was doing some escaping making it perhaps impossible to construct the needed string to get the correct command.
✓ I could see the file handles held by the JVM to stdout/stderr are released before the process finishes.
✓ I could create 13k tasks without the JVM throwing an OOM with 62MB of memory given to the JVM (looks like the JVM is not holding resources like some people will end up with just doing ProcessBuilder and cmd /c.
✓ an intermediate cmd.exe is not created
I use the following to launch a Java application from another Java app.
ProcessBuilder pb = new ProcessBuilder(javaPath + javaCommand, maxMemStr,
minMemStr, stackSizeStr, jarCommand, jarfile, jarArg);
try {
Process p = pb.start();
} catch (IOException ex) {
Logger.getLogger(launch.class.getName()).log(Level.SEVERE, null, ex);
}
where javaCommand is either java or javaw (javaPath is empty most of the time unless a user points to an alternate path). The problem is, after the app launches, even when I verify the process list to contain java, it doesn't show the console.
Is it because PrcoessBuilder doesn't invoke the command shell? Is there a way to show the console programatically?
Thanks in advance.
This is because the "command console" itself is a process that attaches to the std-in/-out/-err streams of another process and displays them on the screen. When you launch Java all by itself, no other processes will be handling those streams, hence the lack of a command console. To get the results you want, you will need to launch a new instance of the command console and subsequently have it run your custom java command.
There may be a better way to do this... but I think the solution to this is going to be platform-dependent. In Windows, you could do something like:
ProcessBuilder pb = new ProcessBuilder("start", "\"JAwesomeSauce\"", "cmd.exe",
"/k", javaPath + javaCommand, maxMemStr, minMemStr, stackSizeStr, jarCommand,
jarfile, jarArg);
try {
Process p = pb.start();
} catch (IOException ex) {
Logger.getLogger(launch.class.getName()).log(Level.SEVERE, null, ex);
}
I assume you could do something similar in Linux/Mac if that's the O/S you're using.
You may want to run the command like this:
cmd /K java ...
or
cmd /C java ...
As far as I remember the Processbuilder opens a pipe to a specific process.
Your command window is a process itself with all you see. If you enter commands the cmd/bash usually creates new processes and attaches to them.
I've been trying to use Java's ProcessBuilder to launch an application in Linux that should run "long-term". The way this program runs is to launch a command (in this case, I am launching a media playback application), allow it to run, and check to ensure that it hasn't crashed. For instance, check to see if the PID is still active, and then relaunch the process, if it has died.
The problem I'm getting right now is that the PID remains alive in the system, but the GUI for the application hangs. I tried shifting the ProcessBuilder(cmd).start() into a separate thread, but that doesn't seem to be solving anything, as I hoped it would have.
Basically the result is that, to the user, the program APPEARS to have crashed, but killing the Java process that drives the ProcessBuilder.start() Process actually allows the created Process to resume its normal behavior. This means that something in the Java application is interfering with the spawned Process, but I have absolutely no idea what, at this point. (Hence why I tried separating it into another thread, which didn't seem to resolve anything)
If anyone has any input/thoughts, please let me know, as I can't for the life of me think of how to solve this problem.
Edit: I have no concern over the I/O stream created from the Process, and have thus taken no steps to deal with that--could this cause a hang in the Process itself?
If the process writes to stderr or stdout, and you're not reading it - it will just "hang" , blocking when writing to stdout/err. Either redirect stdout/err to /dev/null using a shell or merge stdout/err with redirectErrorStream(true) and spawn another thread that reads from stdout of the process
You want the trick?
Don't start your process from ProcessBuilder.start(). Don't try to mess with stream redirection/consumption from Java (especially if you give no s**t about it ; )
Use ProcessBuilder.start() to start a little shell script that gobbles all the input/output streams.
Something like that:
#!/bin/bash
nohup $1 >/dev/null 2>error.log &
That is: if you don't care about stdout and still want to log stderr (do you?) to a file (error.log here).
If you don't even care about stderr, just redirect it to stdout:
#!/bin/bash
nohup $1 >/dev/null 2>1 &
And you call that tiny script from Java, giving it as an argument the name of the process you want to run.
If a process running on Linux that is redirecting both stdout and stderr to /dev/null still produce anything then you've got a broken, non-compliant, Linux install ;)
In other word: the above Just Works [TM] and get rid of the problematic "you need to consume the streams in this and that order bla bla bla Java-specific non-sense".
The thread running the process may block if it does not handle the output. This can be done by spawning a new thread that reads the output of the process.
final ProcessBuilder builder = new ProcessBuilder("script")
.redirectErrorStream(true)
.directory(workDirectory);
final Process process = builder.start();
final StringWriter writer = new StringWriter();
new Thread(new Runnable() {
public void run() {
IOUtils.copy(process.getInputStream(), writer);
}
}).start();
final int exitValue = process.waitFor();
final String processOutput = writer.toString();
Just stumbled on this after I had a similar issue. Agreeing with nos, you need to handle the output. I had something like this:
ProcessBuilder myProc2 = new ProcessBuilder(command);
final Process process = myProc2.start();
and it was working great. The spawned process even did output some output but not much. When I started to output a lot more, it appeared my process wasn't even getting launched anymore. I updated to this:
ProcessBuilder myProc2 = new ProcessBuilder(command);
myProc2.redirectErrorStream(true);
final Process process = myProc2.start();
InputStream myIS = process.getInputStream();
String tempOut = convertStreamToStr(myIS);
and it started working again. (Refer to this link for convertStreamToStr() code)
Edit: I have no concern over the I/O stream created from the Process, and have thus taken no steps to deal with that--could this cause a hang in the Process itself?
If you don't read the output streams created by the process then it is possible that the application will block once the application's buffers are full. I've never seen this happen on Linux (although I'm not saying that it doesn't) but I have seen this exact problem on Windows. I think this is likely related.
JDK7 will have builtin support for subprocess I/O redirection:
http://download.oracle.com/javase/7/docs/api/java/lang/ProcessBuilder.html
In the meantime, if you really want to discard stdout/stderr, it seems best (on Linux) to invoke ProcessBuilder on a command that looks like:
["/bin/bash", "-c", "exec YOUR_COMMAND_HERE >/dev/null 2>&1"]
Another solution is to start the process with Redirect.PIPE and close the InputStream like this:
ProcessBuilder builder = new ProcessBuilder(cmd);
builder.redirectOutput(Redirect.PIPE);
builder.redirectErrorStream(true); // redirect the SysErr to SysOut
Process proc = builder.start();
proc.getInputStream().close(); // this will close the pipe and the output will "flow"
proc.waitFor(); //wait
I tested this in Windows and Linux, and works!
In case you need to capture stdout and stderr and monitor the process then using Apache Commons Exec helped me a lot.
I believe the problem is the buffering pipe from Linux itself.
Try to use stdbuf with your executable
new ProcessBuilder().command("/usr/bin/stdbuf","-o0","*executable*","*arguments*");**
The -o0 says not to buffer the output.
The same goes to -i0 and -e0 if you want to unbuffer the input and error pipe.
you need to read the output before waiting to finish the cycle. You will not be notified If the output doesn't fill the buffer. If it does, it will wait until you read the output.
Suppose you have some errors or responses regarding your command which you are not reading. This would cause the application to stop and waitFor to wait forever. A simple way around is to re-direct the errors to the regular output.
I was spent 2 days on this issue.
public static void exeCuteCommand(String command) {
try {
boolean isWindows = System.getProperty("os.name").toLowerCase().startsWith("windows");
ProcessBuilder builder = new ProcessBuilder();
if (isWindows) {
builder.command("cmd.exe", "/c", command);
} else {
builder.command("sh", "-c", command);
}
Process process = builder.start();
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
while ((line = reader.readLine()) != null)
System.out.println("Cmd Response: " + line);
process.waitFor();
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
}