We Keep Coding

Using Java's Runtime.getRuntime().exec I get error with some commands, success with others -- how can this be determined? [duplicate]

When I try to run Runtime.exec(String), certain commands work, while other commands are executed but fail or do different things than in my terminal. Here is a self-contained test case that demonstrates the effect:
public class ExecTest {
static void exec(String cmd) throws Exception {
Process p = Runtime.getRuntime().exec(cmd);
int i;
while( (i=p.getInputStream().read()) != -1) {
System.out.write(i);
}
while( (i=p.getErrorStream().read()) != -1) {
System.err.write(i);
}
}
public static void main(String[] args) throws Exception {
System.out.print("Runtime.exec: ");
String cmd = new java.util.Scanner(System.in).nextLine();
exec(cmd);
}
}
The example works great if I replace the command with echo hello world, but for other commands -- especially those involving filenames with spaces like here -- I get errors even though the command is clearly being executed:
myshell$ javac ExecTest.java && java ExecTest
Runtime.exec: ls -l 'My File.txt'
ls: cannot access 'My: No such file or directory
ls: cannot access File.txt': No such file or directory
meanwhile, copy-pasting to my shell:
myshell$ ls -l 'My File.txt'
-rw-r--r-- 1 me me 4 Aug 2 11:44 My File.txt
Why is there a difference? When does it work and when does it fail? How do I make it work for all commands?

Why do some commands fail?
This happens because the command passed to Runtime.exec(String) is not executed in a shell. The shell performs a lot of common support services for programs, and when the shell is not around to do them, the command will fail.
When do commands fail?
A command will fail whenever it depends on a shell features. The shell does a lot of common, useful things we don't normally think about:
The shell splits correctly on quotes and spaces
This makes sure the filename in "My File.txt" remains a single argument.
Runtime.exec(String) naively splits on spaces and would pass this as two separate filenames. This obviously fails.
The shell expands globs/wildcards
When you run ls *.doc, the shell rewrites it into ls letter.doc notes.doc.
Runtime.exec(String) doesn't, it just passes them as arguments.
ls has no idea what * is, so the command fails.
The shell manages pipes and redirections.
When you run ls mydir > output.txt, the shell opens "output.txt" for command output and removes it from the command line, giving ls mydir.
Runtime.exec(String) doesn't. It just passes them as arguments.
ls has no idea what > means, so the command fails.
The shell expands variables and commands
When you run ls "$HOME" or ls "$(pwd)", the shell rewrites it into ls /home/myuser.
Runtime.exec(String) doesn't, it just passes them as arguments.
ls has no idea what $ means, so the command fails.
What can you do instead?
There are two ways to execute arbitrarily complex commands:
Simple and sloppy: delegate to a shell.
You can just use Runtime.exec(String[]) (note the array parameter) and pass your command directly to a shell that can do all the heavy lifting:
// Simple, sloppy fix. May have security and robustness implications
String myFile = "some filename.txt";
String myCommand = "cp -R '" + myFile + "' $HOME 2> errorlog";
Runtime.getRuntime().exec(new String[] { "bash", "-c", myCommand });
Secure and robust: take on the responsibilities of the shell.
This is not a fix that can be mechanically applied, but requires an understanding the Unix execution model, what shells do, and how you can do the same. However, you can get a solid, secure and robust solution by taking the shell out of the picture. This is facilitated by ProcessBuilder.
The command from the previous example that requires someone to handle 1. quotes, 2. variables, and 3. redirections, can be written as:
String myFile = "some filename.txt";
ProcessBuilder builder = new ProcessBuilder(
"cp", "-R", myFile, // We handle word splitting
System.getenv("HOME")); // We handle variables
builder.redirectError( // We set up redirections
ProcessBuilder.Redirect.to(new File("errorlog")));
builder.start();

How to pass file location(which is owned by different user) as argument to bash script from java code [duplicate]

When I try to run Runtime.exec(String), certain commands work, while other commands are executed but fail or do different things than in my terminal. Here is a self-contained test case that demonstrates the effect:
public class ExecTest {
static void exec(String cmd) throws Exception {
Process p = Runtime.getRuntime().exec(cmd);
int i;
while( (i=p.getInputStream().read()) != -1) {
System.out.write(i);
}
while( (i=p.getErrorStream().read()) != -1) {
System.err.write(i);
}
}
public static void main(String[] args) throws Exception {
System.out.print("Runtime.exec: ");
String cmd = new java.util.Scanner(System.in).nextLine();
exec(cmd);
}
}
The example works great if I replace the command with echo hello world, but for other commands -- especially those involving filenames with spaces like here -- I get errors even though the command is clearly being executed:
myshell$ javac ExecTest.java && java ExecTest
Runtime.exec: ls -l 'My File.txt'
ls: cannot access 'My: No such file or directory
ls: cannot access File.txt': No such file or directory
meanwhile, copy-pasting to my shell:
myshell$ ls -l 'My File.txt'
-rw-r--r-- 1 me me 4 Aug 2 11:44 My File.txt
Why is there a difference? When does it work and when does it fail? How do I make it work for all commands?

Why do some commands fail?
This happens because the command passed to Runtime.exec(String) is not executed in a shell. The shell performs a lot of common support services for programs, and when the shell is not around to do them, the command will fail.
When do commands fail?
A command will fail whenever it depends on a shell features. The shell does a lot of common, useful things we don't normally think about:
The shell splits correctly on quotes and spaces
This makes sure the filename in "My File.txt" remains a single argument.
Runtime.exec(String) naively splits on spaces and would pass this as two separate filenames. This obviously fails.
The shell expands globs/wildcards
When you run ls *.doc, the shell rewrites it into ls letter.doc notes.doc.
Runtime.exec(String) doesn't, it just passes them as arguments.
ls has no idea what * is, so the command fails.
The shell manages pipes and redirections.
When you run ls mydir > output.txt, the shell opens "output.txt" for command output and removes it from the command line, giving ls mydir.
Runtime.exec(String) doesn't. It just passes them as arguments.
ls has no idea what > means, so the command fails.
The shell expands variables and commands
When you run ls "$HOME" or ls "$(pwd)", the shell rewrites it into ls /home/myuser.
Runtime.exec(String) doesn't, it just passes them as arguments.
ls has no idea what $ means, so the command fails.
What can you do instead?
There are two ways to execute arbitrarily complex commands:
Simple and sloppy: delegate to a shell.
You can just use Runtime.exec(String[]) (note the array parameter) and pass your command directly to a shell that can do all the heavy lifting:
// Simple, sloppy fix. May have security and robustness implications
String myFile = "some filename.txt";
String myCommand = "cp -R '" + myFile + "' $HOME 2> errorlog";
Runtime.getRuntime().exec(new String[] { "bash", "-c", myCommand });
Secure and robust: take on the responsibilities of the shell.
This is not a fix that can be mechanically applied, but requires an understanding the Unix execution model, what shells do, and how you can do the same. However, you can get a solid, secure and robust solution by taking the shell out of the picture. This is facilitated by ProcessBuilder.
The command from the previous example that requires someone to handle 1. quotes, 2. variables, and 3. redirections, can be written as:
String myFile = "some filename.txt";
ProcessBuilder builder = new ProcessBuilder(
"cp", "-R", myFile, // We handle word splitting
System.getenv("HOME")); // We handle variables
builder.redirectError( // We set up redirections
ProcessBuilder.Redirect.to(new File("errorlog")));
builder.start();

Execute bash script and pass arguments with spaces from java

I know there are a lot of post about executing commands from Java but I just can't get this to work. Here is what I'm trying to do, I have a bash script, it receives 2 arguments which might or might not have spaces, then from Java I'm executing the script and passing the arguments like this(I'm surrounding the arguments with quotes and escaping them with backslashes):
String cmd = "/opt/myScript \"/opt/myPath1\" \"/opt/myPath2 with spaces\"";
Runtime rt = Runtime.getRuntime();
rt.exec(cmd);
I also tried to use the ProcessBuilder class like this:
String myScript = "/opt/myScript";
String myArg1= "/opt/myPath1";
String myArg2 = "/opt/myPath2 with spaces";
ProcessBuilder pb = new ProcessBuilder(myScript , myArg1, myArg2);
pb.start;
Arguments with no spaces are received successfully but I still have problems with the second one.
I thought the ProcessBuilder class would handle the spaces but seems like I'm missing something.
I'm not sure if it has something to do, but just in case here is my script:
#!/bin/bash
PATH=$PATH:$1
gnome-terminal --working-directory $2
$1 and $2 are the arguments sent from Java.

Get the same trouble, finally solved with:
Runtime.getRuntime().exec(new String[]{"bash", "-c", <command with spaces>});

Runtime.exec() is an overloaded method. There are several possible ways how to call it. The call exec(String command) executes the specified string command but the argument are separated by spaces here. The method exec(String[] cmdarray) executes the specified command and arguments. There are other exec() variants but the best for you is
String cmd[] = new String[] {"/opt/myScript", "/opt/myPath1", "/opt/myPath2 with spaces" };
Runtime rt = Runtime.getRuntime();
rt.exec(cmd);
It is possible to use ProcessBuilder can be used as well for argument passing. I think the only error is missing parenthesis after pb.start.
And last but not least the script has a major bug. It does not contain quutes arround $2. It should be
#!/bin/bash
PATH="$PATH:$1"
gnome-terminal --working-directory "$2"

Passing pre-escaped command-line arguments to ProcessBuilder

I bumped into this problem today when setting up a local set of communicating programs. Basically one of my applications is sending some data to another, and part of this data is a string containing a command to execute (like you would from the command-line). Let's say, for example:
g++ foo.cc bar.cc -o foobar
is the command sent by my first application. The second application, which receives the command (amongst other things), needs to execute this command after doing some other processing.
Now, at first I thought this would be trivial using a ProcessBuilder:
String exampleCommand = "g++ foo.cc bar.cc -o foobar";
ProcessBuilder builder = new ProcessBuilder(exampleCommand);
builder.start().waitFor();
However this is where the problem occurs.
CreateProcess error=2, The system cannot find the file specified
Okay, no worries I guess I can't just dump the whole thing into the builder. The first part of the command is usually a trivial string so I thought I could probably get away with a split around the first ' ' to separate the program name and arguments.
String exampleCommand = "g++ foo.cc bar.cc -o foobar";
String[] parts = exampleCommand.split(" ", 2);
ProcessBuilder builder = new ProcessBuilder(parts[0], parts[1]);
builder.start().waitFor();
And this brought me a little closer, the g++ file could now be found correctly, however after examining the stderr of g++ I found that the following error had occurred:
g++.exe: error: foo.cc bar.cc -o foobar: No such file or directory
At this point I realised that the ProcessBuilder class must be escaping all arguments passed to it in preparation for the command-line (hence the reason it usually takes arguments as an array of individual arguments rather than just a predefined argument string).
My question is, "Is there any way to pass a raw string of arguments to a ProcessBuilder and say THERE, execute EXACTLY this?"
Because the command comes from another application and is in no way static I can't just break the arguments down into an array beforehand and pass them to the ProcessBuilder constructor properly. The arguments are not so trivial that simply splitting the string around a ' ' will work properly either; arguments might contain spaces escaped with double quotes. For example:
g++ "..\my documents\foo.cpp" bar.cpp -o foobar
Could be a command coming from the application and splitting that string around ' ' and passing it to the ProcessBuilder will result in corrupt arguments.
If there is no proper way to do this can someone please point me to a standalone command line argument parser (in Java) that can turn a command-line string into a valid String[]?

Okay I feel rather foolish now but I achieved my desired result by simply reverting back to the good old Runtime.getRuntime().exec(...). I'll leave the question up in case anyone is as silly as me and find it useful.
String exampleCommand = "g++ foo.cc bar.cc -o foobar";
Runtime sys = Runtime.getRuntime();
sys.exec(exampleCommand);
Easy.

A comment to the Runtime.getRuntime().exec(...) solution:
The Runtime.getRuntime().exec(...) is not good anymore. In java executed on OSX El Capitan, 'Runtime.getRuntime().exec(...)' contains an error that sometimes closes the opened process when the java program exits. It works fine on previous OSX versions. However, ProcessBuilder works on all OSX versions.
(Haven't posted enough to have a enough rep points to make this as a normal comment.)

As a command it runs fine from cmd prompt but not from java code

The code given below actually tries running a command. This command when run from command prompt, produces the necessary output. But when i try to run the application from java code, it keeps on running and doesn't produce any output file.
String arg[]={"C:\\app1.exe", "C:\\app2.exe", "c:\\app3.exe"};
String pwd[]={"123","-x","-sf"};
String outputfile="c:\\output.xml"
String command=arg[0]+pwd[0]+arg[1]+pwd[1]+arg[2]+pwd[2]+output;
Process pr=rt.exec(command);

String command=arg[0]+pwd[0]+arg[1]+pwd[1]+arg[2]+pwd[3]+output;
At least you are missing the whitespace between the arguments!

You should not concatenate all arguments to one string. Instead, pass them as separate arguments to
Process exec(java.lang.String[])

I think you made mistake in generating command.
It would be
C:\\app1.exe123C:\\app2.exe-xc:\\app3.exe-sfc:\\output.xml
Make sure the space
And use this exec(String[]

My guess is that you haven't tried this in a debugger or printed what it is trying to run.
My guess is that when you make this compile, you don't have a command called.
C:\app1.exe123C:\app2.exe-xc:\app3.exe-sfc:\output.xml
You cannot have more than one : in the path.

You concatenate all commands and args, but you never insert spaces between the commands and args.
So your command looks like this: "C:\app1.exe123C:\app2.exe-xc:\app3.exe-sfc:\output.xml"
And also pwd[3] doesn't exist. You have an array with 3 elements, so the highest element would be pwd[2]. You should get and ArrayIndexOutOfBoundsException here (or is it just a copy-paste-mistake)?

Well there's a couple of things wrong with the code:
A space is needed between the commands and the arguments and pwd[3] is out of bounds. I ran this code and it works.
String arg[]={"C:\\app1.exe", "C:\\app2.exe", "c:\\app3.exe"};
String pwd[]={" 123"," -x"," -sf"};
String outputfile="c:\\output.xml";
String command=arg[0]+pwd[0]+arg[1]+pwd[1]+arg[2]+pwd[2]+outputfile;
try {
Process pr=Runtime.getRuntime().exec(command);
} catch (IOException e) {
e.printStackTrace();
}

Try:
String[] command = new String[] { arg[0], pwd[0], arg[1], pwd[1],
arg[2], pwd[2], output };
This is assuming the command you wish to run is
C:\app1.exe 123 C:\app2.exe -x C:\app3.exe -sf c:\output.xml
If you really want to run three separate commands, you will have to run exec() more than once.
See the javadoc at http://download.oracle.com/javase/1.4.2/docs/api/java/lang/Runtime.html#exec(java.lang.String[]) for details.
EDIT: As another answerer has pointed out, there is no pwd[3]!

If your apps "app1", "app2" ... are run from the command prompt you need open that before.
by launching cmd.exe first of all.
And then as others suggested add space between app and arguments.
Try by pasting this in the Run/Search input field in windows:
cmd.exe /K C:\app1.exe 123 C:\app2.exe
-x c:\app3.exe -sf c:\output.xml
cmd.exe /K keeps the propmt open after executing commands