We Keep Coding

Python 2.7 subprocess call method not working to run java command

I am trying to use a python script to manipulate the input files for my java program. The way I am doing it is I am generating the file name and passing it to subprocess.call() method to execute. Here is my program:
def execJava(self):
self.thisCmd="pause"
call(self.javaCmd,shell=True)
call(self.pauseCmd,shell=True)
Where,
self.javaCmd = 'java -ea -esa -Xfuture -Xss64m -classpath "C:\FVSDK_9_1_1\lib\x86_64\msc_12.0-sse2_crtdll\*" -Djava.library.path="C:\FVSDK_9_1_1\lib\x86_64\msc_12.0-sse2_crtdll;C:\FVSDK_9_1_1\lib\x86_64\share" com.cognitec.jfrsdk.examples.MatchFIRAgainstGallery C:\FVSDK_9_1_1\etc\frsdk.cfg 0 .\tmp\frsdk-scratch\probe_1.fir .\tmp\test\*'
Yes, it's a long complex java instruction but when I run it in the command prompt it works fine. Only when I pass it as a string it doesn't run and returns:
Exception in thread "main" java.lang.Error
After some exploration into the problem, I have discovered that it is due to \x, \t in the instruction, so it is executing
.\tmp\test\*
as
mp est\*
as it replaces \t with tab space while executing. I have looked up quite a bit and didn't find any solution. Any help is much appreciated.

Use forward slashes "/" instead of back slashes "\" for your paths.

Passing pre-escaped command-line arguments to ProcessBuilder

I bumped into this problem today when setting up a local set of communicating programs. Basically one of my applications is sending some data to another, and part of this data is a string containing a command to execute (like you would from the command-line). Let's say, for example:
g++ foo.cc bar.cc -o foobar
is the command sent by my first application. The second application, which receives the command (amongst other things), needs to execute this command after doing some other processing.
Now, at first I thought this would be trivial using a ProcessBuilder:
String exampleCommand = "g++ foo.cc bar.cc -o foobar";
ProcessBuilder builder = new ProcessBuilder(exampleCommand);
builder.start().waitFor();
However this is where the problem occurs.
CreateProcess error=2, The system cannot find the file specified
Okay, no worries I guess I can't just dump the whole thing into the builder. The first part of the command is usually a trivial string so I thought I could probably get away with a split around the first ' ' to separate the program name and arguments.
String exampleCommand = "g++ foo.cc bar.cc -o foobar";
String[] parts = exampleCommand.split(" ", 2);
ProcessBuilder builder = new ProcessBuilder(parts[0], parts[1]);
builder.start().waitFor();
And this brought me a little closer, the g++ file could now be found correctly, however after examining the stderr of g++ I found that the following error had occurred:
g++.exe: error: foo.cc bar.cc -o foobar: No such file or directory
At this point I realised that the ProcessBuilder class must be escaping all arguments passed to it in preparation for the command-line (hence the reason it usually takes arguments as an array of individual arguments rather than just a predefined argument string).
My question is, "Is there any way to pass a raw string of arguments to a ProcessBuilder and say THERE, execute EXACTLY this?"
Because the command comes from another application and is in no way static I can't just break the arguments down into an array beforehand and pass them to the ProcessBuilder constructor properly. The arguments are not so trivial that simply splitting the string around a ' ' will work properly either; arguments might contain spaces escaped with double quotes. For example:
g++ "..\my documents\foo.cpp" bar.cpp -o foobar
Could be a command coming from the application and splitting that string around ' ' and passing it to the ProcessBuilder will result in corrupt arguments.
If there is no proper way to do this can someone please point me to a standalone command line argument parser (in Java) that can turn a command-line string into a valid String[]?

Okay I feel rather foolish now but I achieved my desired result by simply reverting back to the good old Runtime.getRuntime().exec(...). I'll leave the question up in case anyone is as silly as me and find it useful.
String exampleCommand = "g++ foo.cc bar.cc -o foobar";
Runtime sys = Runtime.getRuntime();
sys.exec(exampleCommand);
Easy.

A comment to the Runtime.getRuntime().exec(...) solution:
The Runtime.getRuntime().exec(...) is not good anymore. In java executed on OSX El Capitan, 'Runtime.getRuntime().exec(...)' contains an error that sometimes closes the opened process when the java program exits. It works fine on previous OSX versions. However, ProcessBuilder works on all OSX versions.
(Haven't posted enough to have a enough rep points to make this as a normal comment.)

How to create a process which can execute concurrently with respect to java process [duplicate]

I am working on a program written in Java which, for some actions, launches external programs using user-configured command lines. Currently it uses Runtime.exec() and does not retain the Process reference (the launched programs are either a text editor or archive utility, so no need for the system in/out/err streams).
There is a minor problem with this though, in that when the Java program exits, it doesn't really quit until all the launched programs are exited.
I would greatly prefer it if the launched programs were completely independent of the JVM which launched them.
The target operating system is multiple, with Windows, Linux and Mac being the minimum, but any GUI system with a JVM is really what is desired (hence the user configurability of the actual command lines).
Does anyone know how to make the launched program execute completely independently of the JVM?
Edit in response to a comment
The launch code is as follows. The code may launch an editor positioned at a specific line and column, or it may launch an archive viewer. Quoted values in the configured command line are treated as ECMA-262 encoded, and are decoded and the quotes stripped to form the desired exec parameter.
The launch occurs on the EDT.
static Throwable launch(String cmd, File fil, int lin, int col) throws Throwable {
String frs[][]={
{ "$FILE$" ,fil.getAbsolutePath().replace('\\','/') },
{ "$LINE$" ,(lin>0 ? Integer.toString(lin) : "") },
{ "$COLUMN$",(col>0 ? Integer.toString(col) : "") },
};
String[] arr; // array of parsed tokens (exec(cmd) does not handle quoted values)
cmd=TextUtil.replace(cmd,frs,true,"$$","$");
arr=(String[])ArrayUtil.removeNulls(TextUtil.stringComponents(cmd,' ',-1,true,true,true));
for(int xa=0; xa<arr.length; xa++) {
if(TextUtil.isQuoted(arr[xa],true)) {
arr[xa]=TextDecode.ecma262(TextUtil.stripQuotes(arr[xa]));
}
}
log.println("Launching: "+cmd);
Runtime.getRuntime().exec(arr);
return null;
}
This appears to be happening only when the program is launched from my IDE. I am closing this question since the problem exists only in my development environment; it is not a problem in production. From the test program in one of the answers, and further testing I have conducted I am satisfied that it is not a problem that will be seen by any user of the program on any platform.

There is a parent child relation between your processes and you have to break that.
For Windows you can try:
Runtime.getRuntime().exec("cmd /c start editor.exe");
For Linux the process seem to run detached anyway, no nohup necessary.
I tried it with gvim, midori and acroread.
import java.io.IOException;
public class Exec {
public static void main(String[] args) {
try {
Runtime.getRuntime().exec("/usr/bin/acroread");
} catch (IOException e) {
e.printStackTrace();
}
System.out.println("Finished");
}
}
I think it is not possible to to it with Runtime.exec in a platform independent way.
for POSIX-Compatible system:
Runtime.getRuntime().exec(new String[]{"/bin/sh", "-c", "your command"}).waitFor();

I have some observations that may help other people facing similar issue.
When you use Runtime.getRuntime().exec() and then you ignore the java.lang.Process handle you get back (like in the code from original poster), there is a chance that the launched process may hang.
I have faced this issue in Windows environment and traced the problem to the stdout and stderr streams. If the launched application is writing to these streams, and the buffer for these stream fills up then the launched application may appear to hang when it tries to write to the streams. The solutions are:
Capture the Process handle and empty out the streams continually - but if you want to terminate the java application right after launching the process then this is not a feasible solution
Execute the process call as cmd /c <<process>> (this is only for Windows environment).
Suffix the process command and redirect the stdout and stderr streams to nul using 'command > nul 2>&1'

It may help if you post a test section of minimal code needed to reproduce the problem. I tested the following code on Windows and a Linux system.
public class Main {
/**
* #param args the command line arguments
*/
public static void main(String[] args) throws Exception {
Runtime.getRuntime().exec(args[0]);
}
}
And tested with the following on Linux:
java -jar JustForTesting.jar /home/monceaux/Desktop/__TMP/test.sh
where test.sh looks like:
#!/bin/bash
ping -i 20 localhost
as well as this on Linux:
java -jar JustForTesting.jar gedit
And tested this on Windows:
java -jar JustForTesting.jar notepad.exe
All of these launched their intended programs, but the Java application had no problems exiting. I have the following versions of Sun's JVM as reported by java -version :
Windows: 1.6.0_13-b03
Linux: 1.6.0_10-b33
I have not had a chance to test on my Mac yet. Perhaps there is some interaction occuring with other code in your project that may not be clear. You may want to try this test app and see what the results are.

You want to launch the program in the background, and separate it from the parent. I'd consider nohup(1).

I suspect this would require a actual process fork. Basically, the C equivalent of what you want is:
pid_t id = fork();
if(id == 0)
system(command_line);
The problem is you can't do a fork() in pure Java. What I would do is:
Thread t = new Thread(new Runnable()
{
public void run()
{
try
{
Runtime.getRuntime().exec(command);
}
catch(IOException e)
{
// Handle error.
e.printStackTrace();
}
}
});
t.start();
That way the JVM still won't exit, but no GUI and only a limited memory footprint will remain.

I tried everything mentioned here but without success. Main parent Java process can't quit until the quit of subthread even with cmd /c start and redirecting streams tu nul.
Only one reliable solution for me is this:
try {
Runtime.getRuntime().exec("psexec -i cmd /c start cmd.cmd");
}
catch (Exception e) {
// handle it
}
I know that this is not clear, but this small utility from SysInternals is very helpful and proven. Here is the link.

One way I can think of is to use Runtime.addShutdownHook to register a thread that kills off all the processes (you'd need to retain the process objects somewhere of course).
The shutdown hook is only called when the JVM exits so it should work fine.
A little bit of a hack but effective.

error while running CRF++ from a java project

I want to call CRF++ toolkit from a java program. I type the following:
Process process = runtime.exec("/home/toshiba/Bureau/CRF++-0.54/.libs/lt-crf_learn /home/toshiba/Bureau/CRF++-0.54/example/atb/template /home/toshiba/Bureau/CRF++-0.54/example/atb/tr_java.data");
process.waitFor();
But, I have the the following error:
CRF++: Yet Another CRF Tool Kit
Copyright (C) 2005-2009 Taku Kudo, All rights reserved.
Usage: /home/toshiba/Bureau/CRF++-0.54/.libs/lt-crf_learn [options] files
-f, --freq=INT use features that occuer no less than INT(default 1)
-m, --maxiter=INT set INT for max iterations in LBFGS routine(default 10k)
-c, --cost=FLOAT set FLOAT for cost parameter(default 1.0)
-e, --eta=FLOAT set FLOAT for termination criterion(default 0.0001)
-C, --convert convert text model to binary model
-t, --textmodel build also text model file for debugging
-a, --algorithm=(CRF|MIRA) select training algorithm
-p, --thread=INT number of threads(default 1)
-H, --shrinking-size=INT set INT for number of iterations variable needs to be optimal before considered for shrinking. (default 20)
-v, --version show the version and exit
-h, --help show this help and exit
I 'm wondering if any one could help me?

I don't think that's a bug in CRF++, since you are able to run it from command line. So the actual question is how to pass arguments properly when starting a process using Runtime.exec(). I would suggest trying the following:
String[] cmd = {"/home/toshiba/Bureau/CRF++-0.54/.libs/lt-crf_learn",
"/home/toshiba/Bureau/CRF++-0.54/example/atb/template",
"/home/toshiba/Bureau/CRF++-0.54/example/atb/tr_java.data"};
Process p = Runtime.getRuntime().exec(cmd);
This may help since Runtime.exec() sometimes splits the command line into arguments in a rather strange fashion.
Another potential problem is mentioned here: Java Runtime.exec()

There's a simple solution for this. Just write your command into a temporary file and execute that file as Runtime.getRuntime.exec("sh <temp-filename>"). Later you can delete this file. I will explain reason behind this if this solution works for you.

As a command it runs fine from cmd prompt but not from java code

The code given below actually tries running a command. This command when run from command prompt, produces the necessary output. But when i try to run the application from java code, it keeps on running and doesn't produce any output file.
String arg[]={"C:\\app1.exe", "C:\\app2.exe", "c:\\app3.exe"};
String pwd[]={"123","-x","-sf"};
String outputfile="c:\\output.xml"
String command=arg[0]+pwd[0]+arg[1]+pwd[1]+arg[2]+pwd[2]+output;
Process pr=rt.exec(command);

String command=arg[0]+pwd[0]+arg[1]+pwd[1]+arg[2]+pwd[3]+output;
At least you are missing the whitespace between the arguments!

You should not concatenate all arguments to one string. Instead, pass them as separate arguments to
Process exec(java.lang.String[])

I think you made mistake in generating command.
It would be
C:\\app1.exe123C:\\app2.exe-xc:\\app3.exe-sfc:\\output.xml
Make sure the space
And use this exec(String[]

My guess is that you haven't tried this in a debugger or printed what it is trying to run.
My guess is that when you make this compile, you don't have a command called.
C:\app1.exe123C:\app2.exe-xc:\app3.exe-sfc:\output.xml
You cannot have more than one : in the path.

You concatenate all commands and args, but you never insert spaces between the commands and args.
So your command looks like this: "C:\app1.exe123C:\app2.exe-xc:\app3.exe-sfc:\output.xml"
And also pwd[3] doesn't exist. You have an array with 3 elements, so the highest element would be pwd[2]. You should get and ArrayIndexOutOfBoundsException here (or is it just a copy-paste-mistake)?

Well there's a couple of things wrong with the code:
A space is needed between the commands and the arguments and pwd[3] is out of bounds. I ran this code and it works.
String arg[]={"C:\\app1.exe", "C:\\app2.exe", "c:\\app3.exe"};
String pwd[]={" 123"," -x"," -sf"};
String outputfile="c:\\output.xml";
String command=arg[0]+pwd[0]+arg[1]+pwd[1]+arg[2]+pwd[2]+outputfile;
try {
Process pr=Runtime.getRuntime().exec(command);
} catch (IOException e) {
e.printStackTrace();
}

Try:
String[] command = new String[] { arg[0], pwd[0], arg[1], pwd[1],
arg[2], pwd[2], output };
This is assuming the command you wish to run is
C:\app1.exe 123 C:\app2.exe -x C:\app3.exe -sf c:\output.xml
If you really want to run three separate commands, you will have to run exec() more than once.
See the javadoc at http://download.oracle.com/javase/1.4.2/docs/api/java/lang/Runtime.html#exec(java.lang.String[]) for details.
EDIT: As another answerer has pointed out, there is no pwd[3]!

If your apps "app1", "app2" ... are run from the command prompt you need open that before.
by launching cmd.exe first of all.
And then as others suggested add space between app and arguments.
Try by pasting this in the Run/Search input field in windows:
cmd.exe /K C:\app1.exe 123 C:\app2.exe
-x c:\app3.exe -sf c:\output.xml
cmd.exe /K keeps the propmt open after executing commands