We Keep Coding

How to wait for multi-threaded shell script execution to finish called inside my web service?

I have a java restful service method which executes a myscript.sh using processBuilder. My script takes one input (example - myscript.sh /path/to-a/folder).
Inside the script something like this
-> execute a command which is multithreaded i.e parallel processing
-> echo "my message"
Now when call my script from a linux command line it executes fine. First all the threads running finishes and then some text output from threaded command execution shown on terminal and then echo my message is shown.
But when I call the same script from java using processBuilder, the last echo message comes immidiately and execution ends.
Following the way I call my script from java
ProcessBuilder processBuilder = new ProcessBuilder("/bin/bash","/path/to/myscript.sh","/path/to/folder/data");
Process proc = processBuilder.start();
StringBuffer output = new StringBuffer();
BufferedReader reader = new BufferedReader(new InputStreamReader(proc.getInputStream()));
String line = "";
while((line = reader.readLine()) != null){
output.append(line + "\n");
}
System.out.println("### " + output);
I don't know whats happening, how to debug also.
Can someone enlighten me on how to get the same behaviour from shell script when run from terminal or from java processBuilder?

Use ProcessBuilder.redirectErrorStream(boolean redirectErrorStream) with argument true to merge the errors into output. Alternatively, you could also use the shell command syntax cmd 2>&1 to merge the error with output.
These are some of the cases why you may be immediately getting the output of the last echo statement (instead of the script taking time to run and return proper results):
Missing environment variables
The launched bash needs to source .bashrc or some such recource file
The launched bash may not be running in right directory (you can set this in ProcessBuilder)
The launched bash may not be finding some script/executable in its PATH
The launched bash may not be finding proper libraries in the path for any of the executables
Once you merge error, you would be able to debug and see the errors for yourself.

In your context, separate processes may be spawned in two ways:
1) Bash
/path/to/executables/executable &
This will spawn a new executable executable and you need to wait for it to finish. Here's an answer that will help you.
2) Java
Process exec = Runtime.getRuntime().exec(command);
status = exec.waitFor();
Essentially, you need to wait for the process to end before you start reading its std/err streams.
If I understand the problem correctly, adding just this line to your code should suffice: status = exec.waitFor() (Before you obtain the streams)
Here's the JavaDoc for Process.waitFor() :
Causes the current thread to wait, if necessary, until the process represented by this Process object has terminated. This method returns immediately if the subprocess has already terminated. If the subprocess has not yet terminated, the calling thread will be blocked until the subprocess exits.
Returns:
the exit value of the subprocess represented by this Process object. By convention, the value 0 indicates normal termination.
Throws:
InterruptedException - if the current thread is interrupted by another thread while it is waiting, then the wait is ended and an InterruptedException is thrown

Cannot run Java Runtime.exec() on Opensuse

I am using Runtime.getRuntime().exec function to launch independent GUI Java application for subroutine task.
The code used is in simple manner:
Runtime.getRuntime().exec("java -jar /home/user/jar.jar");
Executing the code doesn't cause any process launch nor error occured! ProcessBuilder has same effect.
Checked to work correctly on Windows.
As seems, on some platforms it is ignored on system level outside Java, as JRE does not return any kind of error.
EDT: I edited the code to read stderr and stdout by parallel thread to preserve main app execution:
Process p = Runtime.getRuntime().exec(runCmd);
new DaemonFailPrint(p).start();
Thread code is:
public class DaemonFailPrint extends Thread {
private Process process;
public DaemonFailPrint(Process process) {
this.process = process;
}
#Override
public void run() {
try {
process.waitFor();
String out = "";
while (process.getInputStream().available() > 0) {
out += (char) process.getInputStream().read();
}
out += System.lineSeparator();
while (process.getInputStream().available() > 0) {
out += (char) process.getErrorStream().read();
}
JOptionPane.showMessageDialog(null, out);
} catch (InterruptedException | IOException ex) {
JOptionPane.showMessageDialog(null, ex);
}
}
}
The result is: I got empty message box straight after subprocess is "launched".
The mean is Process object seems to be created and finished in same time, but no error out exists.

The Runtime::exec() does not wait for the process to exit, so if you want to detect errors in the executed program itself, you'd need to use something like:
Process process = Runtime.getRuntime().exec("java -jar /home/user/jar.jar");
int rc = process.waitFor();
if (rc != 0)
{
System.err.println("The process failed with error: " + rc);
}
It might be, that the jar is not found or cannot be executed etc., those errors you normally see on the console, but if you have no console, the only clue might be the return code.
You might also want to check here how to capture the output console:
Capturing stdout when calling Runtime.exec
Seems that you can use process.getInputStream() to connect to the output stream of the process. So you can simply copy it to the console to see what happened.

So far so good! I found the answer recently by myself, still don't have a reason why it works this way, but I suppose it's all about internal difference of handling new processes in VM's on different platforms.
I had to edit the code this way, and now it works:
String[] runcmd = {"java","-jar","/home/user/jar.jar"};
Runtime.getRuntime().exec(runcmd);
Now it seems to work perfect. As I see it fails to process the file and execute then command with parameters given as same string while no error thrown on Java code level, it's possibly lost in VM internals.

Java need process to return when finished

my problem would take 2 questions, but I'll keep it short. So I need to launch a bat file. Right now I do it like this:
public static void check() throws InterruptedException{
try {
Runtime.getRuntime().exec("cmd /c start build.bat");
Thread.sleep(3000);
} catch (IOException e) {
e.printStackTrace();
}
}
The bat file launches the java compiler to compile another java file and direct the error messages into a txt file. This is what the bat file looks like:
#echo off
javac -Xstdout error.txt MainApp.java
exit
Now the problem is, that I have to include a 3 second sleep, in order to be sure, that the error.txt has been created and filled with errors. This is very unsatisfying. I'd either need a return value from the bat file, so I the rest of the program waits, until it's done or a way to launch the java compiler out of the program and direct the error messages into a txt file.
Thanks everybody.

You can use Process#waitFor:
Causes the current thread to wait, if necessary, until the process
represented by this Process object has terminated
Process p = Runtime.getRuntime().exec("cmd /c start build.bat");
p.waitFor();

How to create a process which can execute concurrently with respect to java process [duplicate]

I am working on a program written in Java which, for some actions, launches external programs using user-configured command lines. Currently it uses Runtime.exec() and does not retain the Process reference (the launched programs are either a text editor or archive utility, so no need for the system in/out/err streams).
There is a minor problem with this though, in that when the Java program exits, it doesn't really quit until all the launched programs are exited.
I would greatly prefer it if the launched programs were completely independent of the JVM which launched them.
The target operating system is multiple, with Windows, Linux and Mac being the minimum, but any GUI system with a JVM is really what is desired (hence the user configurability of the actual command lines).
Does anyone know how to make the launched program execute completely independently of the JVM?
Edit in response to a comment
The launch code is as follows. The code may launch an editor positioned at a specific line and column, or it may launch an archive viewer. Quoted values in the configured command line are treated as ECMA-262 encoded, and are decoded and the quotes stripped to form the desired exec parameter.
The launch occurs on the EDT.
static Throwable launch(String cmd, File fil, int lin, int col) throws Throwable {
String frs[][]={
{ "$FILE$" ,fil.getAbsolutePath().replace('\\','/') },
{ "$LINE$" ,(lin>0 ? Integer.toString(lin) : "") },
{ "$COLUMN$",(col>0 ? Integer.toString(col) : "") },
};
String[] arr; // array of parsed tokens (exec(cmd) does not handle quoted values)
cmd=TextUtil.replace(cmd,frs,true,"$$","$");
arr=(String[])ArrayUtil.removeNulls(TextUtil.stringComponents(cmd,' ',-1,true,true,true));
for(int xa=0; xa<arr.length; xa++) {
if(TextUtil.isQuoted(arr[xa],true)) {
arr[xa]=TextDecode.ecma262(TextUtil.stripQuotes(arr[xa]));
}
}
log.println("Launching: "+cmd);
Runtime.getRuntime().exec(arr);
return null;
}
This appears to be happening only when the program is launched from my IDE. I am closing this question since the problem exists only in my development environment; it is not a problem in production. From the test program in one of the answers, and further testing I have conducted I am satisfied that it is not a problem that will be seen by any user of the program on any platform.

There is a parent child relation between your processes and you have to break that.
For Windows you can try:
Runtime.getRuntime().exec("cmd /c start editor.exe");
For Linux the process seem to run detached anyway, no nohup necessary.
I tried it with gvim, midori and acroread.
import java.io.IOException;
public class Exec {
public static void main(String[] args) {
try {
Runtime.getRuntime().exec("/usr/bin/acroread");
} catch (IOException e) {
e.printStackTrace();
}
System.out.println("Finished");
}
}
I think it is not possible to to it with Runtime.exec in a platform independent way.
for POSIX-Compatible system:
Runtime.getRuntime().exec(new String[]{"/bin/sh", "-c", "your command"}).waitFor();

I have some observations that may help other people facing similar issue.
When you use Runtime.getRuntime().exec() and then you ignore the java.lang.Process handle you get back (like in the code from original poster), there is a chance that the launched process may hang.
I have faced this issue in Windows environment and traced the problem to the stdout and stderr streams. If the launched application is writing to these streams, and the buffer for these stream fills up then the launched application may appear to hang when it tries to write to the streams. The solutions are:
Capture the Process handle and empty out the streams continually - but if you want to terminate the java application right after launching the process then this is not a feasible solution
Execute the process call as cmd /c <<process>> (this is only for Windows environment).
Suffix the process command and redirect the stdout and stderr streams to nul using 'command > nul 2>&1'

It may help if you post a test section of minimal code needed to reproduce the problem. I tested the following code on Windows and a Linux system.
public class Main {
/**
* #param args the command line arguments
*/
public static void main(String[] args) throws Exception {
Runtime.getRuntime().exec(args[0]);
}
}
And tested with the following on Linux:
java -jar JustForTesting.jar /home/monceaux/Desktop/__TMP/test.sh
where test.sh looks like:
#!/bin/bash
ping -i 20 localhost
as well as this on Linux:
java -jar JustForTesting.jar gedit
And tested this on Windows:
java -jar JustForTesting.jar notepad.exe
All of these launched their intended programs, but the Java application had no problems exiting. I have the following versions of Sun's JVM as reported by java -version :
Windows: 1.6.0_13-b03
Linux: 1.6.0_10-b33
I have not had a chance to test on my Mac yet. Perhaps there is some interaction occuring with other code in your project that may not be clear. You may want to try this test app and see what the results are.

You want to launch the program in the background, and separate it from the parent. I'd consider nohup(1).

I suspect this would require a actual process fork. Basically, the C equivalent of what you want is:
pid_t id = fork();
if(id == 0)
system(command_line);
The problem is you can't do a fork() in pure Java. What I would do is:
Thread t = new Thread(new Runnable()
{
public void run()
{
try
{
Runtime.getRuntime().exec(command);
}
catch(IOException e)
{
// Handle error.
e.printStackTrace();
}
}
});
t.start();
That way the JVM still won't exit, but no GUI and only a limited memory footprint will remain.

I tried everything mentioned here but without success. Main parent Java process can't quit until the quit of subthread even with cmd /c start and redirecting streams tu nul.
Only one reliable solution for me is this:
try {
Runtime.getRuntime().exec("psexec -i cmd /c start cmd.cmd");
}
catch (Exception e) {
// handle it
}
I know that this is not clear, but this small utility from SysInternals is very helpful and proven. Here is the link.

One way I can think of is to use Runtime.addShutdownHook to register a thread that kills off all the processes (you'd need to retain the process objects somewhere of course).
The shutdown hook is only called when the JVM exits so it should work fine.
A little bit of a hack but effective.

Capturing a java NoClassDefFoundError thrown by program called via shell script

I have a java program which is a compiler and executor for Jasper Reports and is called via shell script on our reports server.
I just fixed a bug caused by a missing dependancy, but it took me a while to figure out what was going wrong. Normally, problems caused by the compile process are captured, recorded to a log file and emailed to the relevant person. As this was a NoClassDefFoundError, it basically exited the program and failed in a silent manner from the users perspective.
Is there any way for me to capture errors like this one so that they can also be emailed away? I have the authority to modify the executing shell script.

Typically errors are not caught by application code and are thrown to JVM level where they are printed to STDERR. So, your way to track this error is to redirect STDERR to file:
java -cp YourMain 1>stdout.log 2>stderr.log
You can also put both STDOUT and STDERR together:
java -cp YourMain 1>&2 2>wholelog.log
There is a lot of reference about stream redirection in web. You can take a look there if my examples do not satisfy you. And it a depends on your OS.

Just can catch the error i.e.
try {
numericDefinition = new net.sf.cb2xml.def.BasicNumericDefinition(
binName, binarySizes, SynchronizeAt, usePositive, floatSynchronize, doubleSynchronize
);
} catch (NoClassDefFoundError e) {
System.out.println("Class Not Found: " + e.getMessage());
}
You do need to be very careful of your coding though, it is easy to get NoClassDefFoundError thrown at class initialisation time and not get into to the try .. catch block.
The NoClassDefFoundError will be thrown the first time a class is refereneced which could be when could when a class uses a class which uses a class which uses a class ... which uses a class that references a class that does not exist.
The following may fail with NoClassDefFoundError at class initialization because of the import.
import net.sf.cb2xml.def.BasicNumericDefinition; // could cause the NoClassDefFoundError
...........
try {
numericDefinition = new BasicNumericDefinition(
binName, binarySizes, SynchronizeAt, usePositive, floatSynchronize, doubleSynchronize
);
} catch (NoClassDefFoundError e) {
System.out.println("Class Not Found: " + e.getMessage());
}