We Keep Coding

Apache Commons Exec Problems Handling Regex Argument?

I'm trying to use Apache Commons Exec to run a git command which uses a regex.
When I form my CommandLine and print it out it looks like this:
[git, --no-pager, grep, --line-number, --untracked, --extended-regexp, "^\s*public void\s+(testFindByAdAccount).*", --, *Test.java]
However when I execute this, git returns no results, resulting in an exit code 1.
When I run this command manually though, it returns plenty of results and succeeds. Changing the --extended-regexp argument to just a string like testFindByAdAccount does yield results when run via Exec, so I think Apache Commons is doing something to the regexp argument making it invalid. Any ideas what is going on?
EDIT: Adding a reproducible example
Clone https://github.com/ragurney/min-example
Run gradlew shadowJar to produce jar file for project
Run the app with java -jar app/build/libs/app-all.jar
Note the output which shows the command printed fails with an exit code 1 (because there are no results returned by the git command)
$ java -jar app/build/libs/app-all.jar
HELLOOOOOO
WD::: null
[git, --no-pager, grep, --line-number, --untracked, --extended-regexp, "^\s*public void\s+(testAppHasAGreeting)\(\).*", --, *Test.java]
WD::: /Users/rgurney/Src/personal/min-example
Exception in thread "main" java.lang.RuntimeException: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
at min.example.App.lambda$runCommand$1(App.java:74)
at io.vavr.control.Try.getOrElseThrow(Try.java:748)
Running the command manually does produce expected results:
$ git --no-pager grep --line-number --untracked --extended-regexp "^\s*public void\s+(testAppHasAGreeting)\(\).*" -- "*Test.java"
app/src/test/java/min/example/AppTest.java:11: public void testAppHasAGreeting() {

I got a clue as to what's going on here when the sample you provided worked just fine on my Windows laptop but failed on my Linux desktop.
Once I made sure the git version wasn't the culprit (tested several versions between 2.17 and 2.39 on both machines), I figured the difference must be in the way different shells handle quoting. Specifically, the only argument here that has any potential quoting issues is the regex ("^\s*public void\s+(testFindByAdAccount).*"), which is added to the command line by commandLine.addArgument(regex);.
addArgument may look innocuous, but under the hood, it allows the CommandLine to handle the quoting itself (i.e., addArgument(String argument) calls addArgument(String argument, true). Since you've handled the quoting yourself, you should not allow the CommandLine to handle the quoting, and should explicitly call it with the second argument false. i.e.:
public static List<String> grep(String regex, String filePattern, String wd) {
CommandLine commandLine = CommandLine.parse("git");
commandLine.addArgument("--no-pager");
commandLine.addArgument("grep");
commandLine.addArgument("--line-number");
commandLine.addArgument("--untracked");
commandLine.addArgument("--extended-regexp");
commandLine.addArgument(regex, false);
// Here -----------------------^
commandLine.addArgument("--");
commandLine.addArgument(filePattern);
System.out.println(commandLine);
return List.of(runCommand(commandLine, wd).split("\n"));
}
This takes the quote-handling logic away and ensures the same code runs smoothly both on Windows and Linux (at least those I've tested).

How to provide the -Dlogback.configurationFile=./logback.xml option in nodejs spawn method

I have angular2-node.js application. I am executing a jar file through the node server.
Jar execution is happening fine but it's using the logback.xml present in the jar file.
Node js code:
app.get('/report/:parameter1/:parameter2', function(req, res) {
var fileName = path.join(__dirname, '..', 'javaFile', 'xyz.jar');
spawn('/usr/bin/java', ['-jar ', fileName, parameter1 , parameter2, '&'],{
stdio : ['ignore', out, err],
detached : true }).unref();
data = '{response: Success}';
res.status(200).json(data);
res.end();
});
I want to refer the different logback.xml file for jar execution while running the jar from UI. So, i tried the below code:
spawn('/usr/bin/java', ['-jar -Dlogback.configurationFile=./logback.xml', fileName, cacheName , cacheType, '&'],{
stdio : ['ignore', out, err],
detached : true }).unref();
But, it also didn't work and throw the below error:
Unrecognized option: -jar -Dlogback.configurationFile=./logback.xml
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
I am new to node js. I searched the web but couldn't get an answer.
Is there any way to provide the logback.xml file dynamically in node.js code something like we do in shell script like below:
nohup java -jar -Dlogback.configurationFile=./logback.xml xyz.jar
Can anyone provide any solution for this.

The args arguments is <string[]>, so you should split the multiple args into multiple elements of the array, like you've done for the other arguments. You can check the signature of the method here.
Try,
spawn('/usr/bin/java', ['-jar', '-Dlogback.configurationFile=./logback.xml'], ....

Python 2.7 subprocess call method not working to run java command

I am trying to use a python script to manipulate the input files for my java program. The way I am doing it is I am generating the file name and passing it to subprocess.call() method to execute. Here is my program:
def execJava(self):
self.thisCmd="pause"
call(self.javaCmd,shell=True)
call(self.pauseCmd,shell=True)
Where,
self.javaCmd = 'java -ea -esa -Xfuture -Xss64m -classpath "C:\FVSDK_9_1_1\lib\x86_64\msc_12.0-sse2_crtdll\*" -Djava.library.path="C:\FVSDK_9_1_1\lib\x86_64\msc_12.0-sse2_crtdll;C:\FVSDK_9_1_1\lib\x86_64\share" com.cognitec.jfrsdk.examples.MatchFIRAgainstGallery C:\FVSDK_9_1_1\etc\frsdk.cfg 0 .\tmp\frsdk-scratch\probe_1.fir .\tmp\test\*'
Yes, it's a long complex java instruction but when I run it in the command prompt it works fine. Only when I pass it as a string it doesn't run and returns:
Exception in thread "main" java.lang.Error
After some exploration into the problem, I have discovered that it is due to \x, \t in the instruction, so it is executing
.\tmp\test\*
as
mp est\*
as it replaces \t with tab space while executing. I have looked up quite a bit and didn't find any solution. Any help is much appreciated.

Use forward slashes "/" instead of back slashes "\" for your paths.

Python: How can I execute a jar file through a python script

I have been looking for an answer for how to execute a java jar file through python and after looking at:
Execute .jar from Python
How can I get my python (version 2.5) script to run a jar file inside a folder instead of from command line?
How to run Python egg files directly without installing them?
I tried to do the following (both my jar and python file are in the same directory):
import os
if __name__ == "__main__":
os.system("java -jar Blender.jar")
and
import subprocess
subprocess.call(['(path)Blender.jar'])
Neither have worked. So, I was thinking that I should use Jython instead, but I think there must a be an easier way to execute jar files through python.
Do you have any idea what I may do wrong? Or, is there any other site that I study more about my problem?

I would use subprocess this way:
import subprocess
subprocess.call(['java', '-jar', 'Blender.jar'])
But, if you have a properly configured /proc/sys/fs/binfmt_misc/jar you should be able to run the jar directly, as you wrote.
So, which is exactly the error you are getting?
Please post somewhere all the output you are getting from the failed execution.

This always works for me:
from subprocess import *
def jarWrapper(*args):
process = Popen(['java', '-jar']+list(args), stdout=PIPE, stderr=PIPE)
ret = []
while process.poll() is None:
line = process.stdout.readline()
if line != '' and line.endswith('\n'):
ret.append(line[:-1])
stdout, stderr = process.communicate()
ret += stdout.split('\n')
if stderr != '':
ret += stderr.split('\n')
ret.remove('')
return ret
args = ['myJarFile.jar', 'arg1', 'arg2', 'argN'] # Any number of args to be passed to the jar file
result = jarWrapper(*args)
print result

I used the following way to execute tika jar to extract the content of a word document. It worked and I got the output also. The command I'm trying to run is "java -jar tika-app-1.24.1.jar -t 42250_EN_Upload.docx"
from subprocess import PIPE, Popen
process = Popen(['java', '-jar', 'tika-app-1.24.1.jar', '-t', '42250_EN_Upload.docx'], stdout=PIPE, stderr=PIPE)
result = process.communicate()
print(result[0].decode('utf-8'))
Here I got result as tuple, hence "result[0]". Also the string was in binary format (b-string). To convert it into normal string we need to decode with 'utf-8'.

With args: concrete example using Closure Compiler (https://developers.google.com/closure/) from python
import os
import re
src = test.js
os.execlp("java", 'blablabla', "-jar", './closure_compiler.jar', '--js', src, '--js_output_file', '{}'.format(re.sub('.js$', '.comp.js', src)))
(also see here When using os.execlp, why `python` needs `python` as argv[0])

How about using os.system() like:
os.system('java -jar blabla...')
os.system(command)
Execute the command (a string) in a subshell. This is implemented by calling the Standard C function system(), and has the same limitations. Changes to sys.stdin, etc. are not reflected in the environment of the executed command.

Compiling a class using Java code using process

I have this piece of code that compiles a class called tspClassName, when I compile using this code:
Process compileProc = null;
try {
compileProc = Runtime.getRuntime().exec("javac -classpath ."
+ File.separator + "src" + File.separator
+ File.separator + "generated." + tspClassName + ".java -d ." + File.separator + "bin");
// catch exception
if (compileProc.exitValue() != 0)
{
System.out.println("Compile exit status: "
+ compileProc.exitValue());
System.err.println("Compile error:" +
compileProc.getErrorStream());
it outputs this:
"Compile exit status: 2
Compile error:java.io.FileInputStream#17182c1"
The class tspClassName.java compiles without errors otherwise, so I am guessing it has to do with the path,and in my eclipse project, the tspClassName.java resides in package homework4.generated inside src, is there something wrong with the path that I use in the code?
thanks

Your Java code runs a command that looks something like this:
javac -classpath ./src//generated.ClassName.java -d ./bin
I don't think that's what you want. I think you need to change your Java code so it maybe generates something like:
javac -classpath . src/generated/ClassName.java -d ./bin
^
Note the space after the classpath (".").

You can use the javax.tools.JavaCompiler or JCI that wrap this functionality.

I recommend doing something like this:
String command = String.format(
"javac -classpath . src%1$sgenerated%1$s%2$s.java -d .%1$sbin",
File.separator,
tspClassName
);
LOG("Executing " + command);
//... exec(command) etc
... where LOG is whatever your logging framework uses to log the command to be executed. This will help debugging immensely, since it was pointed out that the command you built is ill-constructed.
Alternately you can also build the string using replace
String command =
"javac -classpath . src/generated/ClassName.java -d ./bin"
.replace("/", File.separator)
.replace("ClassName", tspClassName);
This is perhaps more readable.
On draining Process streams
OP's comment suggests that waitFor() never returns. This is likely caused by compilation errors/warnings in javac process.
From the API:
Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.
You need to continuously drain the Process.getOutputStream() et.al.
See also
Java Puzzlers, Puzzle 82: Beer Blast
Related questions
Draining standard error in Java

I think the proper way to do this kind of work is programatically using the javax.tools API, not an external process:
JavaCompiler compiler = ToolProvider.getSystemJavaCompiler()
Reference:
ToolProvider.getSystemJavaCompiler()

The problem could be with the file location instead of using single value parameter for exec()
try 3 parameter method which has the command, environment and location as parameters which helps us to move to the specified location and execute the command
check 6 and 8 methods for reference
http://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.html
Process compile = Runtime.getRuntime().exec("javac "+fname,null,new File(dir));

firstly you should use apache exec library if you want to use processes like that. Apache exec library makes things very easy.
Secondly you should print your std output and std error streams of your process which you are executing. Without them its no way to know whats being executed and what's it doing.
Thirdly, try to print the full cmd line which the process is executing. Copy that cmd line and try to run it manually. Most of the time you would find your issues this way.
And finally if your aim is just to compile a class / generate or modify a class file at runtime give this a good read and try. It has examples too. You could also try code generation / class manipulation libraries like BCEL, JavaAssist etc.
Best of luck.