We Keep Coding

Python 2.7 subprocess call method not working to run java command

I am trying to use a python script to manipulate the input files for my java program. The way I am doing it is I am generating the file name and passing it to subprocess.call() method to execute. Here is my program:
def execJava(self):
self.thisCmd="pause"
call(self.javaCmd,shell=True)
call(self.pauseCmd,shell=True)
Where,
self.javaCmd = 'java -ea -esa -Xfuture -Xss64m -classpath "C:\FVSDK_9_1_1\lib\x86_64\msc_12.0-sse2_crtdll\*" -Djava.library.path="C:\FVSDK_9_1_1\lib\x86_64\msc_12.0-sse2_crtdll;C:\FVSDK_9_1_1\lib\x86_64\share" com.cognitec.jfrsdk.examples.MatchFIRAgainstGallery C:\FVSDK_9_1_1\etc\frsdk.cfg 0 .\tmp\frsdk-scratch\probe_1.fir .\tmp\test\*'
Yes, it's a long complex java instruction but when I run it in the command prompt it works fine. Only when I pass it as a string it doesn't run and returns:
Exception in thread "main" java.lang.Error
After some exploration into the problem, I have discovered that it is due to \x, \t in the instruction, so it is executing
.\tmp\test\*
as
mp est\*
as it replaces \t with tab space while executing. I have looked up quite a bit and didn't find any solution. Any help is much appreciated.

Use forward slashes "/" instead of back slashes "\" for your paths.

Why System.getProperties() returns null for -D defined property?

I'm running a jarball using
java -classpath myBatch.jar some.package.MyMainClass \
-bloodyArgument "I got the argument!" -Dbloody.prop="I got the prop!"
Then in my main I have:
Properties argsProps = BatchUtils.argsToProperties(args);
System.out.println(argsProps.getProperty("bloodyArgument"));
System.out.println(System.getProperty("bloody.prop"));
And I get output:
I got the argument!
null
I can see the command line bloodyArgument (I added it to see if "something" gets passed to the program), but I'd expect the -D argument to set the system property. Why is the "bloody.prop" null?
PS: BatchUtils.argsToProperties() does what you'd expected it to do: parse -argName "value" from command line into argName=value property pair.

Everything after the class argument of java command gets passed to your main in String[] args and doesn't make it to the JVM.
The solution was to rearrange the properties like this:
java -classpath myBatch.jar -Dbloody.prop="I got the prop!" \
some.package.MyMainClass -bloodyArgument "I got the argument!"
I didn't find this anywhere explicitly stated when waddling the web with http://duckduckgo.com or googling this queation of "null -D defined property". I figured it a lot later when printing all system properties and the whole arguments array, so am posting for others.

Passing pre-escaped command-line arguments to ProcessBuilder

I bumped into this problem today when setting up a local set of communicating programs. Basically one of my applications is sending some data to another, and part of this data is a string containing a command to execute (like you would from the command-line). Let's say, for example:
g++ foo.cc bar.cc -o foobar
is the command sent by my first application. The second application, which receives the command (amongst other things), needs to execute this command after doing some other processing.
Now, at first I thought this would be trivial using a ProcessBuilder:
String exampleCommand = "g++ foo.cc bar.cc -o foobar";
ProcessBuilder builder = new ProcessBuilder(exampleCommand);
builder.start().waitFor();
However this is where the problem occurs.
CreateProcess error=2, The system cannot find the file specified
Okay, no worries I guess I can't just dump the whole thing into the builder. The first part of the command is usually a trivial string so I thought I could probably get away with a split around the first ' ' to separate the program name and arguments.
String exampleCommand = "g++ foo.cc bar.cc -o foobar";
String[] parts = exampleCommand.split(" ", 2);
ProcessBuilder builder = new ProcessBuilder(parts[0], parts[1]);
builder.start().waitFor();
And this brought me a little closer, the g++ file could now be found correctly, however after examining the stderr of g++ I found that the following error had occurred:
g++.exe: error: foo.cc bar.cc -o foobar: No such file or directory
At this point I realised that the ProcessBuilder class must be escaping all arguments passed to it in preparation for the command-line (hence the reason it usually takes arguments as an array of individual arguments rather than just a predefined argument string).
My question is, "Is there any way to pass a raw string of arguments to a ProcessBuilder and say THERE, execute EXACTLY this?"
Because the command comes from another application and is in no way static I can't just break the arguments down into an array beforehand and pass them to the ProcessBuilder constructor properly. The arguments are not so trivial that simply splitting the string around a ' ' will work properly either; arguments might contain spaces escaped with double quotes. For example:
g++ "..\my documents\foo.cpp" bar.cpp -o foobar
Could be a command coming from the application and splitting that string around ' ' and passing it to the ProcessBuilder will result in corrupt arguments.
If there is no proper way to do this can someone please point me to a standalone command line argument parser (in Java) that can turn a command-line string into a valid String[]?

Okay I feel rather foolish now but I achieved my desired result by simply reverting back to the good old Runtime.getRuntime().exec(...). I'll leave the question up in case anyone is as silly as me and find it useful.
String exampleCommand = "g++ foo.cc bar.cc -o foobar";
Runtime sys = Runtime.getRuntime();
sys.exec(exampleCommand);
Easy.

A comment to the Runtime.getRuntime().exec(...) solution:
The Runtime.getRuntime().exec(...) is not good anymore. In java executed on OSX El Capitan, 'Runtime.getRuntime().exec(...)' contains an error that sometimes closes the opened process when the java program exits. It works fine on previous OSX versions. However, ProcessBuilder works on all OSX versions.
(Haven't posted enough to have a enough rep points to make this as a normal comment.)

Unable to properly enter arguments for java file

I apologize for the simplicity of this problem.
I'm trying to use a program called ChromHMM to analyze some biological data. I try and run the program according to the instructions but can't seem to enter the arguments correctly.
Here is an example:
E:\ComputationalAnalysis\ChromHMM>java -mx1600M -jar ChromHMM.jar BinarizeBed
chromosomelengthfile=\CHROMSIZES\hg19.txt inputbeddir=\Donor1 cellmarkfileta
ble=\Donor1\cellmarkfile.txt outputbinarydir=\firstoutput
It returns just this:
usage BinarizeBed [-b binsize][-c controldir][-center][-colfields chromosome,sta
rt,end[,strand]][-e offsetend][-f foldthresh][-n shift][-o outputcontroldir][-p
poissonthresh][-peaks][-s offsetstart][-strictthresh][-t outputsignaldir][-u pse
udocountcontrol][-w flankwidthcontrol] chromosomelengthfile inputbeddir cellmark
filetable outputbinarydir
In the manual it says the 4 arguments at the end are the only ones required to run. Does anyone know what I'm doing wrong? I'm trying to do this from the windows command prompt

try removing the name of the argument and the equals;just pass the value. so in your example you would have the following.
E:\ComputationalAnalysis\ChromHMM>java -mx1600M -jar ChromHMM.jar BinarizeBed
\CHROMSIZES\hg19.txt \Donor1 cellmarkfileta \Donor1\cellmarkfile.txt \firstoutput

subprocess call in python to invoke java jar files with JAVA_OPTS

Example code:
import subprocess
subprocess.call(['java', '-jar', 'temp.jar'])
How to specify the JAVA_OPTS in the above command? I am getting a 'java.lang.OutOfMemoryError: unable to create new native thread' when I use the above command and I think specifying JAVA_OPTS in the command would solve the problem.
I did specify the JAVA_OPTS in .bashrc file and it had no effect.

You can do this, but finding how to do it in the documentation is kind of a wild goose chase.
The subprocess.call() documentation says,
subprocess.call(args, *, stdin=None, stdout=None, stderr=None, shell=False)
The arguments shown above are merely the most common ones, described below in Frequently Used Arguments (hence the slightly odd notation in the abbreviated signature).
Then the Frequently Used Arguments section, says, at the very end after describing a bunch of other arguments:
These options, along with all of the other options, are described in more detail in the Popen constructor documentation.
Well then! The Popen documentation gives the full signature:
class subprocess.Popen(args, bufsize=0, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=False, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0)
env is the one you want! However, if you just pass env={'JAVA_OPTS': 'foo'}, then that will override all environment variables, including stuff like CLASSPATH, which could break other things. So you probably want to use code like this to add a JAVA_OPTS environment variable for the new process execution, without setting it in the current process:
#!/usr/bin/env python2.7
import os
import subprocess
# Make a copy of the environment
env = dict(os.environ)
env['JAVA_OPTS'] = 'foo'
subprocess.call(['java', '-jar', 'temp.jar'], env=env)

There is no need to use JAVA_OPTS - just pass in some more arguments to call(). For example:
import subprocess
subprocess.call(['java', '-jar', 'temp.jar', '-Xmx1024m', '-Xms256m'])