We Keep Coding

Apache Commons Exec Problems Handling Regex Argument?

I'm trying to use Apache Commons Exec to run a git command which uses a regex.
When I form my CommandLine and print it out it looks like this:
[git, --no-pager, grep, --line-number, --untracked, --extended-regexp, "^\s*public void\s+(testFindByAdAccount).*", --, *Test.java]
However when I execute this, git returns no results, resulting in an exit code 1.
When I run this command manually though, it returns plenty of results and succeeds. Changing the --extended-regexp argument to just a string like testFindByAdAccount does yield results when run via Exec, so I think Apache Commons is doing something to the regexp argument making it invalid. Any ideas what is going on?
EDIT: Adding a reproducible example
Clone https://github.com/ragurney/min-example
Run gradlew shadowJar to produce jar file for project
Run the app with java -jar app/build/libs/app-all.jar
Note the output which shows the command printed fails with an exit code 1 (because there are no results returned by the git command)
$ java -jar app/build/libs/app-all.jar
HELLOOOOOO
WD::: null
[git, --no-pager, grep, --line-number, --untracked, --extended-regexp, "^\s*public void\s+(testAppHasAGreeting)\(\).*", --, *Test.java]
WD::: /Users/rgurney/Src/personal/min-example
Exception in thread "main" java.lang.RuntimeException: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
at min.example.App.lambda$runCommand$1(App.java:74)
at io.vavr.control.Try.getOrElseThrow(Try.java:748)
Running the command manually does produce expected results:
$ git --no-pager grep --line-number --untracked --extended-regexp "^\s*public void\s+(testAppHasAGreeting)\(\).*" -- "*Test.java"
app/src/test/java/min/example/AppTest.java:11: public void testAppHasAGreeting() {

I got a clue as to what's going on here when the sample you provided worked just fine on my Windows laptop but failed on my Linux desktop.
Once I made sure the git version wasn't the culprit (tested several versions between 2.17 and 2.39 on both machines), I figured the difference must be in the way different shells handle quoting. Specifically, the only argument here that has any potential quoting issues is the regex ("^\s*public void\s+(testFindByAdAccount).*"), which is added to the command line by commandLine.addArgument(regex);.
addArgument may look innocuous, but under the hood, it allows the CommandLine to handle the quoting itself (i.e., addArgument(String argument) calls addArgument(String argument, true). Since you've handled the quoting yourself, you should not allow the CommandLine to handle the quoting, and should explicitly call it with the second argument false. i.e.:
public static List<String> grep(String regex, String filePattern, String wd) {
CommandLine commandLine = CommandLine.parse("git");
commandLine.addArgument("--no-pager");
commandLine.addArgument("grep");
commandLine.addArgument("--line-number");
commandLine.addArgument("--untracked");
commandLine.addArgument("--extended-regexp");
commandLine.addArgument(regex, false);
// Here -----------------------^
commandLine.addArgument("--");
commandLine.addArgument(filePattern);
System.out.println(commandLine);
return List.of(runCommand(commandLine, wd).split("\n"));
}
This takes the quote-handling logic away and ensures the same code runs smoothly both on Windows and Linux (at least those I've tested).

Manage Java process with go

I have JVM and all dependencies for my Java program ready. With Java I would run like:
javac HelloWorld.java
java HelloWorld
Now I want to, in Linux environment, control this Java program processes using Go's cmd package. In Go, when you run command you are given the PID. With this PID, I want to terminate the Java program whenever j want and restart using the same cmd package. Would this work correctly as long as I have JVM installed? I want to do:
cmd := exec.Command("bash", "-c", " "java HelloWorld")
cmd.Start()
syscall.Kill(cmd.Process.Pid)
Thanks!

In short, yes.
As a test, with added interrupt handling so your own Go process doesn't terminate this will work:
package main
import (
"os/exec"
"syscall"
"os"
"os/signal"
"fmt"
)
func main() {
cmd := exec.Command("bash", "-c", "java HelloWorld")
err := cmd.Start()
fmt.Printf("Starting java proccess with pid %d\n", cmd.Process.Pid)
if err != nil {
// do something about it
}
c := make(chan os.Signal, 1)
done := make(chan bool, 1)
signal.Notify(c, os.Interrupt)
signal.Notify(c, syscall.SIGTERM)
go func() {
<-c
fmt.Printf("Sending interrupt to pid: %d\n", cmd.Process.Pid)
syscall.Kill(cmd.Process.Pid, syscall.SIGHUP)
done <- true
}()
<-done
}
Companion Java class:
public class HelloWorld {
public static void main(String[] args) throws Exception {
System.out.println("Hello World from Go! But you cant see me :)");
while (true) {
System.out.println("you cant see this because I am outing to the STDOUT of a subshell!");
Thread.sleep(5000);
}
}
}
But it is full of gotchas. As long as your Go process exits normally, it will send the signal you specify (sighup would be natural choice, if I'd venture a guess) to the java pid. But you need to ensure that you wont let a zombie in case your own Go process crash or in case your java application hangs on after failing to shut down cleanly when you tell it to. Saving that pid to a /tmp/ file and doing all sorts of things with it in case of a restart could be interesting, but you know your needs.
Edit: controlling a JVM process from another program might get finicky quick. You should evaluate if you really want to do that. If you are in Linux, I'd take a look at the SysV init/systemd/upstart/start-stop-daemon system your distro uses if your companion java program acts as a daemon.

Invoking shell script contains SSH from java

I am trying to invoke a shell script containing SSH command from a java program. However it is failing with a error code 1.
My java code is as:
public class CallScript {
private static String filePath="";
private static String args1="";
public static void main(String[] args) throws Exception{
if(args!=null && args.length > 0){
filePath = args[0];
if(args.length > 1){
args1=args[1];
}
}else{
throw new Exception("File Path should be first Argument");
}
System.out.println(args.length);
invokeScript(filePath,args1);
}
private static void invokeScript(String filePath, String arg1) throws Exception{
System.out.println("Inside invoke Script " + arg1);
Process p = Runtime.getRuntime().exec(filePath);
p.waitFor();
int exitVal = p.exitValue();
System.out.println("The Exit Value " + exitVal);
}
}
I compiled the program and placed the executable jar in my Unix environment.
shell script which is invoked from java
ssh -l >test.log
I used the following command to run my java program :
java -jar invokeScript.jar /tmp/upog/test.sh
output
Inside invoke Script
The Exit Value 1.
If I have some other command in the shell script like ls -al > test.log, the code is working with success and I am getting the return value 0.
Also if I invoke the shell script containing ssh command directly in Unix box, it is working fine.(the box have password-less connectivity)
But it is failing, if i call from java...
Any advice....

invokeScript.jar works fine for me here. The issue looks to be with /tmp/upog/test.sh. If I try to run the command I get:
$ ssh -l >test.log
ssh: option requires an argument -- l
[...]
Did you mean ssh -l someuser example.com >test.log?
When I test it with an error-free shell script, running ssh works:
$ cat >/tmp/upog/test.sh
#!/bin/bash
ssh example.com ls /tmp/upog >test.log
$ chmod +x /tmp/upog/test.sh
$ java -jar invokeScript.jar /tmp/upog/test.sh
1
Inside invoke Script
The Exit Value 0
$ cat test.log
bar
baz
foo
(Using example.com as the replacement text for my actual server)

there are two possible return values:
return value of 0 means success
return value of >=1 means an error. One should be seeking for an error code in this case. that is why the error codes are different.
There are two general approaches to use 1 as a flag to show you get an error and the error code itself is in the other place. The other way is to return 0 as success and in case of error is to return an error code itself which is >0. In nix-like systems major command return 0 or 1.
ls -al > test.log
there are two command here, the both are executed with success 0 + 0 - return value 0.
ssh -l >test.log
Here, the first command "ssh -l" executed with an error because you have not provided any arguments. No matter how ">test.log" will go you got 1 already.
your app works ok it seems - it return an exit value.
in case you need more info about the error of nested app.
you may execute
ssh -l >test.log 2>&1 here test.log will have errors.
or you may use something like
BufferedReader err = new BufferedReader(
new InputStreamReader(p.getErrorStream())
); //
in your java app to handle error there.
Note: java code might be incorrect since I use it rarely
Upd: edited command syntax
wrong 2.&1 -> correct 2>&1

How to create a process which can execute concurrently with respect to java process [duplicate]

I am working on a program written in Java which, for some actions, launches external programs using user-configured command lines. Currently it uses Runtime.exec() and does not retain the Process reference (the launched programs are either a text editor or archive utility, so no need for the system in/out/err streams).
There is a minor problem with this though, in that when the Java program exits, it doesn't really quit until all the launched programs are exited.
I would greatly prefer it if the launched programs were completely independent of the JVM which launched them.
The target operating system is multiple, with Windows, Linux and Mac being the minimum, but any GUI system with a JVM is really what is desired (hence the user configurability of the actual command lines).
Does anyone know how to make the launched program execute completely independently of the JVM?
Edit in response to a comment
The launch code is as follows. The code may launch an editor positioned at a specific line and column, or it may launch an archive viewer. Quoted values in the configured command line are treated as ECMA-262 encoded, and are decoded and the quotes stripped to form the desired exec parameter.
The launch occurs on the EDT.
static Throwable launch(String cmd, File fil, int lin, int col) throws Throwable {
String frs[][]={
{ "$FILE$" ,fil.getAbsolutePath().replace('\\','/') },
{ "$LINE$" ,(lin>0 ? Integer.toString(lin) : "") },
{ "$COLUMN$",(col>0 ? Integer.toString(col) : "") },
};
String[] arr; // array of parsed tokens (exec(cmd) does not handle quoted values)
cmd=TextUtil.replace(cmd,frs,true,"$$","$");
arr=(String[])ArrayUtil.removeNulls(TextUtil.stringComponents(cmd,' ',-1,true,true,true));
for(int xa=0; xa<arr.length; xa++) {
if(TextUtil.isQuoted(arr[xa],true)) {
arr[xa]=TextDecode.ecma262(TextUtil.stripQuotes(arr[xa]));
}
}
log.println("Launching: "+cmd);
Runtime.getRuntime().exec(arr);
return null;
}
This appears to be happening only when the program is launched from my IDE. I am closing this question since the problem exists only in my development environment; it is not a problem in production. From the test program in one of the answers, and further testing I have conducted I am satisfied that it is not a problem that will be seen by any user of the program on any platform.

There is a parent child relation between your processes and you have to break that.
For Windows you can try:
Runtime.getRuntime().exec("cmd /c start editor.exe");
For Linux the process seem to run detached anyway, no nohup necessary.
I tried it with gvim, midori and acroread.
import java.io.IOException;
public class Exec {
public static void main(String[] args) {
try {
Runtime.getRuntime().exec("/usr/bin/acroread");
} catch (IOException e) {
e.printStackTrace();
}
System.out.println("Finished");
}
}
I think it is not possible to to it with Runtime.exec in a platform independent way.
for POSIX-Compatible system:
Runtime.getRuntime().exec(new String[]{"/bin/sh", "-c", "your command"}).waitFor();

I have some observations that may help other people facing similar issue.
When you use Runtime.getRuntime().exec() and then you ignore the java.lang.Process handle you get back (like in the code from original poster), there is a chance that the launched process may hang.
I have faced this issue in Windows environment and traced the problem to the stdout and stderr streams. If the launched application is writing to these streams, and the buffer for these stream fills up then the launched application may appear to hang when it tries to write to the streams. The solutions are:
Capture the Process handle and empty out the streams continually - but if you want to terminate the java application right after launching the process then this is not a feasible solution
Execute the process call as cmd /c <<process>> (this is only for Windows environment).
Suffix the process command and redirect the stdout and stderr streams to nul using 'command > nul 2>&1'

It may help if you post a test section of minimal code needed to reproduce the problem. I tested the following code on Windows and a Linux system.
public class Main {
/**
* #param args the command line arguments
*/
public static void main(String[] args) throws Exception {
Runtime.getRuntime().exec(args[0]);
}
}
And tested with the following on Linux:
java -jar JustForTesting.jar /home/monceaux/Desktop/__TMP/test.sh
where test.sh looks like:
#!/bin/bash
ping -i 20 localhost
as well as this on Linux:
java -jar JustForTesting.jar gedit
And tested this on Windows:
java -jar JustForTesting.jar notepad.exe
All of these launched their intended programs, but the Java application had no problems exiting. I have the following versions of Sun's JVM as reported by java -version :
Windows: 1.6.0_13-b03
Linux: 1.6.0_10-b33
I have not had a chance to test on my Mac yet. Perhaps there is some interaction occuring with other code in your project that may not be clear. You may want to try this test app and see what the results are.

You want to launch the program in the background, and separate it from the parent. I'd consider nohup(1).

I suspect this would require a actual process fork. Basically, the C equivalent of what you want is:
pid_t id = fork();
if(id == 0)
system(command_line);
The problem is you can't do a fork() in pure Java. What I would do is:
Thread t = new Thread(new Runnable()
{
public void run()
{
try
{
Runtime.getRuntime().exec(command);
}
catch(IOException e)
{
// Handle error.
e.printStackTrace();
}
}
});
t.start();
That way the JVM still won't exit, but no GUI and only a limited memory footprint will remain.

I tried everything mentioned here but without success. Main parent Java process can't quit until the quit of subthread even with cmd /c start and redirecting streams tu nul.
Only one reliable solution for me is this:
try {
Runtime.getRuntime().exec("psexec -i cmd /c start cmd.cmd");
}
catch (Exception e) {
// handle it
}
I know that this is not clear, but this small utility from SysInternals is very helpful and proven. Here is the link.

One way I can think of is to use Runtime.addShutdownHook to register a thread that kills off all the processes (you'd need to retain the process objects somewhere of course).
The shutdown hook is only called when the JVM exits so it should work fine.
A little bit of a hack but effective.

Running a program from within Java code

What is the simplest way to call a program from with a piece of Java code? (The program I want to run is aiSee and it can be run from command line or from Windows GUI; and I am on Vista but the code will also be run on Linux systems).

Take a look at Process and Runtime classes. Keep in mind that what you are trying to accomplish is probably not platform independent.
Here is a little piece of code that might be helpful:
public class YourClass
{
public static void main(String args[])
throws Exception
{
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec("name_of_your_application.exe");
int exitVal = proc.exitValue();
System.out.println("Process exitValue: " + exitVal);
}
}
One question in S.O. discussing similiar issues. Another one. And another one.

You can get a runtime instance using Runtime.getRuntime() and call the runtime's exec method, with the command to execute the program as an argument.
For example:
Runtime runTime = Runtime.getRuntime ();
Process proc = rt.exec("iSee.exe");
You can also capture the output of the program by using getting the InputStream from the process.

The difficulty you will run into is how to get the application to know the path. You may want to use an xml or config file, but if you use this link, it should explain how to run a file:
http://www.javacoffeebreak.com/faq/faq0030.html

You may also want to consider passing in some kind of argument to your program to facilitate finding the specific program you want to run.
This could be with command line arguments, properties files or system properties.