We Keep Coding

Apache Commons Exec Problems Handling Regex Argument?

I'm trying to use Apache Commons Exec to run a git command which uses a regex.
When I form my CommandLine and print it out it looks like this:
[git, --no-pager, grep, --line-number, --untracked, --extended-regexp, "^\s*public void\s+(testFindByAdAccount).*", --, *Test.java]
However when I execute this, git returns no results, resulting in an exit code 1.
When I run this command manually though, it returns plenty of results and succeeds. Changing the --extended-regexp argument to just a string like testFindByAdAccount does yield results when run via Exec, so I think Apache Commons is doing something to the regexp argument making it invalid. Any ideas what is going on?
EDIT: Adding a reproducible example
Clone https://github.com/ragurney/min-example
Run gradlew shadowJar to produce jar file for project
Run the app with java -jar app/build/libs/app-all.jar
Note the output which shows the command printed fails with an exit code 1 (because there are no results returned by the git command)
$ java -jar app/build/libs/app-all.jar
HELLOOOOOO
WD::: null
[git, --no-pager, grep, --line-number, --untracked, --extended-regexp, "^\s*public void\s+(testAppHasAGreeting)\(\).*", --, *Test.java]
WD::: /Users/rgurney/Src/personal/min-example
Exception in thread "main" java.lang.RuntimeException: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
at min.example.App.lambda$runCommand$1(App.java:74)
at io.vavr.control.Try.getOrElseThrow(Try.java:748)
Running the command manually does produce expected results:
$ git --no-pager grep --line-number --untracked --extended-regexp "^\s*public void\s+(testAppHasAGreeting)\(\).*" -- "*Test.java"
app/src/test/java/min/example/AppTest.java:11: public void testAppHasAGreeting() {

I got a clue as to what's going on here when the sample you provided worked just fine on my Windows laptop but failed on my Linux desktop.
Once I made sure the git version wasn't the culprit (tested several versions between 2.17 and 2.39 on both machines), I figured the difference must be in the way different shells handle quoting. Specifically, the only argument here that has any potential quoting issues is the regex ("^\s*public void\s+(testFindByAdAccount).*"), which is added to the command line by commandLine.addArgument(regex);.
addArgument may look innocuous, but under the hood, it allows the CommandLine to handle the quoting itself (i.e., addArgument(String argument) calls addArgument(String argument, true). Since you've handled the quoting yourself, you should not allow the CommandLine to handle the quoting, and should explicitly call it with the second argument false. i.e.:
public static List<String> grep(String regex, String filePattern, String wd) {
CommandLine commandLine = CommandLine.parse("git");
commandLine.addArgument("--no-pager");
commandLine.addArgument("grep");
commandLine.addArgument("--line-number");
commandLine.addArgument("--untracked");
commandLine.addArgument("--extended-regexp");
commandLine.addArgument(regex, false);
// Here -----------------------^
commandLine.addArgument("--");
commandLine.addArgument(filePattern);
System.out.println(commandLine);
return List.of(runCommand(commandLine, wd).split("\n"));
}
This takes the quote-handling logic away and ensures the same code runs smoothly both on Windows and Linux (at least those I've tested).

Python 2.7 subprocess call method not working to run java command

I am trying to use a python script to manipulate the input files for my java program. The way I am doing it is I am generating the file name and passing it to subprocess.call() method to execute. Here is my program:
def execJava(self):
self.thisCmd="pause"
call(self.javaCmd,shell=True)
call(self.pauseCmd,shell=True)
Where,
self.javaCmd = 'java -ea -esa -Xfuture -Xss64m -classpath "C:\FVSDK_9_1_1\lib\x86_64\msc_12.0-sse2_crtdll\*" -Djava.library.path="C:\FVSDK_9_1_1\lib\x86_64\msc_12.0-sse2_crtdll;C:\FVSDK_9_1_1\lib\x86_64\share" com.cognitec.jfrsdk.examples.MatchFIRAgainstGallery C:\FVSDK_9_1_1\etc\frsdk.cfg 0 .\tmp\frsdk-scratch\probe_1.fir .\tmp\test\*'
Yes, it's a long complex java instruction but when I run it in the command prompt it works fine. Only when I pass it as a string it doesn't run and returns:
Exception in thread "main" java.lang.Error
After some exploration into the problem, I have discovered that it is due to \x, \t in the instruction, so it is executing
.\tmp\test\*
as
mp est\*
as it replaces \t with tab space while executing. I have looked up quite a bit and didn't find any solution. Any help is much appreciated.

Use forward slashes "/" instead of back slashes "\" for your paths.

Unable to properly enter arguments for java file

I apologize for the simplicity of this problem.
I'm trying to use a program called ChromHMM to analyze some biological data. I try and run the program according to the instructions but can't seem to enter the arguments correctly.
Here is an example:
E:\ComputationalAnalysis\ChromHMM>java -mx1600M -jar ChromHMM.jar BinarizeBed
chromosomelengthfile=\CHROMSIZES\hg19.txt inputbeddir=\Donor1 cellmarkfileta
ble=\Donor1\cellmarkfile.txt outputbinarydir=\firstoutput
It returns just this:
usage BinarizeBed [-b binsize][-c controldir][-center][-colfields chromosome,sta
rt,end[,strand]][-e offsetend][-f foldthresh][-n shift][-o outputcontroldir][-p
poissonthresh][-peaks][-s offsetstart][-strictthresh][-t outputsignaldir][-u pse
udocountcontrol][-w flankwidthcontrol] chromosomelengthfile inputbeddir cellmark
filetable outputbinarydir
In the manual it says the 4 arguments at the end are the only ones required to run. Does anyone know what I'm doing wrong? I'm trying to do this from the windows command prompt

try removing the name of the argument and the equals;just pass the value. so in your example you would have the following.
E:\ComputationalAnalysis\ChromHMM>java -mx1600M -jar ChromHMM.jar BinarizeBed
\CHROMSIZES\hg19.txt \Donor1 cellmarkfileta \Donor1\cellmarkfile.txt \firstoutput

Python: How can I execute a jar file through a python script

I have been looking for an answer for how to execute a java jar file through python and after looking at:
Execute .jar from Python
How can I get my python (version 2.5) script to run a jar file inside a folder instead of from command line?
How to run Python egg files directly without installing them?
I tried to do the following (both my jar and python file are in the same directory):
import os
if __name__ == "__main__":
os.system("java -jar Blender.jar")
and
import subprocess
subprocess.call(['(path)Blender.jar'])
Neither have worked. So, I was thinking that I should use Jython instead, but I think there must a be an easier way to execute jar files through python.
Do you have any idea what I may do wrong? Or, is there any other site that I study more about my problem?

I would use subprocess this way:
import subprocess
subprocess.call(['java', '-jar', 'Blender.jar'])
But, if you have a properly configured /proc/sys/fs/binfmt_misc/jar you should be able to run the jar directly, as you wrote.
So, which is exactly the error you are getting?
Please post somewhere all the output you are getting from the failed execution.

This always works for me:
from subprocess import *
def jarWrapper(*args):
process = Popen(['java', '-jar']+list(args), stdout=PIPE, stderr=PIPE)
ret = []
while process.poll() is None:
line = process.stdout.readline()
if line != '' and line.endswith('\n'):
ret.append(line[:-1])
stdout, stderr = process.communicate()
ret += stdout.split('\n')
if stderr != '':
ret += stderr.split('\n')
ret.remove('')
return ret
args = ['myJarFile.jar', 'arg1', 'arg2', 'argN'] # Any number of args to be passed to the jar file
result = jarWrapper(*args)
print result

I used the following way to execute tika jar to extract the content of a word document. It worked and I got the output also. The command I'm trying to run is "java -jar tika-app-1.24.1.jar -t 42250_EN_Upload.docx"
from subprocess import PIPE, Popen
process = Popen(['java', '-jar', 'tika-app-1.24.1.jar', '-t', '42250_EN_Upload.docx'], stdout=PIPE, stderr=PIPE)
result = process.communicate()
print(result[0].decode('utf-8'))
Here I got result as tuple, hence "result[0]". Also the string was in binary format (b-string). To convert it into normal string we need to decode with 'utf-8'.

With args: concrete example using Closure Compiler (https://developers.google.com/closure/) from python
import os
import re
src = test.js
os.execlp("java", 'blablabla', "-jar", './closure_compiler.jar', '--js', src, '--js_output_file', '{}'.format(re.sub('.js$', '.comp.js', src)))
(also see here When using os.execlp, why `python` needs `python` as argv[0])

How about using os.system() like:
os.system('java -jar blabla...')
os.system(command)
Execute the command (a string) in a subshell. This is implemented by calling the Standard C function system(), and has the same limitations. Changes to sys.stdin, etc. are not reflected in the environment of the executed command.

As a command it runs fine from cmd prompt but not from java code

The code given below actually tries running a command. This command when run from command prompt, produces the necessary output. But when i try to run the application from java code, it keeps on running and doesn't produce any output file.
String arg[]={"C:\\app1.exe", "C:\\app2.exe", "c:\\app3.exe"};
String pwd[]={"123","-x","-sf"};
String outputfile="c:\\output.xml"
String command=arg[0]+pwd[0]+arg[1]+pwd[1]+arg[2]+pwd[2]+output;
Process pr=rt.exec(command);

String command=arg[0]+pwd[0]+arg[1]+pwd[1]+arg[2]+pwd[3]+output;
At least you are missing the whitespace between the arguments!

You should not concatenate all arguments to one string. Instead, pass them as separate arguments to
Process exec(java.lang.String[])

I think you made mistake in generating command.
It would be
C:\\app1.exe123C:\\app2.exe-xc:\\app3.exe-sfc:\\output.xml
Make sure the space
And use this exec(String[]

My guess is that you haven't tried this in a debugger or printed what it is trying to run.
My guess is that when you make this compile, you don't have a command called.
C:\app1.exe123C:\app2.exe-xc:\app3.exe-sfc:\output.xml
You cannot have more than one : in the path.

You concatenate all commands and args, but you never insert spaces between the commands and args.
So your command looks like this: "C:\app1.exe123C:\app2.exe-xc:\app3.exe-sfc:\output.xml"
And also pwd[3] doesn't exist. You have an array with 3 elements, so the highest element would be pwd[2]. You should get and ArrayIndexOutOfBoundsException here (or is it just a copy-paste-mistake)?

Well there's a couple of things wrong with the code:
A space is needed between the commands and the arguments and pwd[3] is out of bounds. I ran this code and it works.
String arg[]={"C:\\app1.exe", "C:\\app2.exe", "c:\\app3.exe"};
String pwd[]={" 123"," -x"," -sf"};
String outputfile="c:\\output.xml";
String command=arg[0]+pwd[0]+arg[1]+pwd[1]+arg[2]+pwd[2]+outputfile;
try {
Process pr=Runtime.getRuntime().exec(command);
} catch (IOException e) {
e.printStackTrace();
}

Try:
String[] command = new String[] { arg[0], pwd[0], arg[1], pwd[1],
arg[2], pwd[2], output };
This is assuming the command you wish to run is
C:\app1.exe 123 C:\app2.exe -x C:\app3.exe -sf c:\output.xml
If you really want to run three separate commands, you will have to run exec() more than once.
See the javadoc at http://download.oracle.com/javase/1.4.2/docs/api/java/lang/Runtime.html#exec(java.lang.String[]) for details.
EDIT: As another answerer has pointed out, there is no pwd[3]!

If your apps "app1", "app2" ... are run from the command prompt you need open that before.
by launching cmd.exe first of all.
And then as others suggested add space between app and arguments.
Try by pasting this in the Run/Search input field in windows:
cmd.exe /K C:\app1.exe 123 C:\app2.exe
-x c:\app3.exe -sf c:\output.xml
cmd.exe /K keeps the propmt open after executing commands