We Keep Coding

Java jar execution stuck when launched in a process by another java program

This is a very unusual problem I've come across and I'm hoping someone might have some insight on it. I'm on macOS Mojave (10.14.6), using Amazon's JRE Corretto-11.0.9.12.1 (build 11.0.9.1+12-LTS)
I have a program I've written that is something of a scripting engine that we use to do our routine bulk data processing. It takes an xml "script" file which contains the processing directions to execute and a list of input files as arguments. I'll refer to this as the "engine" from this point on. This engine is the backbone of a large portion of our automation. It shows a progress bar while processing to let users know that it is working.
There are 2 programs that use this engine:
One is a thin UI written in Swing, which we use to manually process data; it generates an xml file from the user input and passes it along with the input files and launches the engine in a separate process; the UI itself doesn't process any data.
The other watches a folder on our file server and processes incoming data from our clients daily when a folder is created inside of it so we can rip the data into our database. I'll call this the "importer".
Recently, a problem has come up where the engine becomes stuck while processing. Older versions of the engine did not have this problem, and I'm trying to figure out what exactly changed that caused this to happen, but while I've been trying to do this, our UI and importer programs have been using and older version of the engine. There are new features that we need to use in the new version of the engine, but we can't use it until this problem is solved.
The programs that uses the engine launch it in a process then waits for the result before continuing:
// example command generated from external program
String commandString = "java -jar engine.jar script.xml input_file1.txt input_file2.txt input_file3.txt";
String[] command = {"bash", "-c", commandString};
// I can grab the command from here for debugging
System.out.println(Arrays.toString(command));
ProcessBuilder pb = new ProcessBuilder(command);
// wait for the process to complete before continuing
Process p = pb.start();
p.waitFor();
int result = p.exitValue();
try (BufferedReader e = new BufferedReader(new InputStreamReader(proc.getErrorStream()));
BufferedReader i = new BufferedReader(new InputStreamReader(proc.getInputStream()))) {
String line;
while ((line = e.readLine()) != null) {
System.out.println(line);
}
while ((line = i.readLine()) != null) {
System.out.println(line);
}
}
p.destroy();
// do other stuff
When launched in this way, there is a specific operation that causes the engine to hang. But if I take the command and launch it directly from the command line, the engine runs just fine! This is making it difficult to pin down where exactly the problem is; is it in the engine, or in the other programs? I've spent a couple of days looking for answers and come up with nothing. It's even more frustrating that this problem has appeared seemingly out of nowhere when it was working perfectly before, using the exact code above, for a quite a long time.
The operation where the engine hangs sorts files into folders based on their file names. When I watch my activity monitor while it runs, it's not taxing my resources at all, and disk space isn't an issue. It isn't a file permission issue, as the engine is creating files and folders all the time and in every step leading up to the step where it hangs. And as I said, if I run the command directly from the command line, it creates the folders and sorts the files without issue, to my extreme confusion.
The importer and UI run locally on a station, but the engine jar file lives on our file server, so that it is accessible to every station without individually downloading it everywhere each time there is an update. I thought at first that the issue might lie in the fact that it is being accessed over the network, but the problem occurs even when I use a local copy of the engine on my dev machine, so I have ruled that out. I've also ruled out that it's the JRE, even though we switched to it recently, since the older version of the engine still perform as expected.

There might of course be any reason why your 'engine' program may hang ;-) but certainly it will hang you don't read the its output, and in the right way:
The parent process needs to read the standard output and standard error streams of the child process, given that the child process does generate any substantial amount of output on any of these two channels. This must be done in two separate background threads. If the parent does not read the child's output, then the child process will block as soon as the (small) buffer between the processes is filled up.
The threads should be started as soon as the child process is started, and before the parent calls process.waitFor().
The simplest way to do this is the following:
Process process = processBuilder.start();
InputStream stdout = process.getInputStream();
InputStream stderr = process.getErrorStream();
Thread stdoutThread = new Thread() {
#Override
public void run() {
// read stdout here, e.g.
try {
int c;
while (-1 != (c = stdout.read())) {
// do whatever with c
}
} catch (IOException ioex) {
// ...
}
}
};
Thread stderrThread = new Thread() {
// ... same as for stdout
};
stdoutThread.start();
stderrThread.start();
}
Only after both threads have been started you may wait for the child process and join the threads:
int exitValue = process.waitFor();
stdoutThread.join();
stderrThread.join();
There might be more sophisticated ways to work with background threads using the Concurrency Framework introduced in Java 5, but this basic code gives the idea.

Create process with method called in Java [duplicate]

Is there a way to start a process in Java? in .Net this is done with for example:
System.Diagnostics.Process.Start("processname");
Is there an equivalent in Java so I can then let the user find the application and then it would work for any OS?

http://www.rgagnon.com/javadetails/java-0014.html
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.nio.file.Paths;
public class CmdExec {
public static void main(String args[]) {
try {
// enter code here
Process p = Runtime.getRuntime().exec(
Paths.get(System.getenv("windir"), "system32", "tree.com /A").toString()
);
// enter code here
try(BufferedReader input = new BufferedReader(new InputStreamReader(p.getInputStream()))) {
String line;
while ((line = input.readLine()) != null) {
System.out.println(line);
}
}
} catch (Exception err) {
err.printStackTrace();
}
}
}
You can get the local path using System properties or a similar approach.
http://download.oracle.com/javase/tutorial/essential/environment/sysprop.html

The Java Class Library represents external processes using the java.lang.Process class. Processes can be spawned using a java.lang.ProcessBuilder:
Process process = new ProcessBuilder("processname").start();
or the older interface exposed by the overloaded exec methods on the java.lang.Runtime class:
Process process = Runtime.getRuntime().exec("processname");
Both of these will code snippets will spawn a new process, which usually executes asynchronously and can be interacted with through the resulting Process object. If you need to check that the process has finished (or wait for it to finish), don't forget to check that the exit value (exit code) returned by process.exitValue() or process.waitFor() is as expected (0 for most programs), since no exception is thrown if the process exits abnormally.
Also note that additional code is often necessary to handle the process's I/O correctly, as described in the documentation for the Process class (emphasis added):
By default, the created subprocess does not have its own terminal or console. All its standard I/O (i.e. stdin, stdout, stderr) operations will be redirected to the parent process, where they can be accessed via the streams obtained using the methods getOutputStream(), getInputStream(), and getErrorStream(). The parent process uses these streams to feed input to and get output from the subprocess. Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, or even deadlock.
One way to make sure that I/O is correctly handled and that the exit value indicates success is to use a library like jproc that deals with the intricacies of capturing stdout and stderr, and offers a simple synchronous interface to run external processes:
ProcResult result = new ProcBuilder("processname").run();
jproc is available via maven central:
<dependency>
<groupId>org.buildobjects</groupId>
<artifactId>jproc</artifactId>
<version>2.5.1</version>
</dependency>

See Runtime.exec() and the Process class. In its simplest form:
Process myProcess = Runtime.getRuntime().exec(command);
...
Note that you also need to read the process' output (eg: myProcess.getInputStream()) -- or the process will hang on some systems. This can be highly confusing the first time, and should be included in any introduction to these APIs. See James P.'s response for an example.
You might also want to look into the new ProcessBuilder class, which makes it easier to change environment variables and to invoke subprocesses :
Process myProcess = new ProcessBuilder(command, arg).start();
...

Launch a process with java and then terminate it

Normally I use this code to run a bash script and get it's output
ProcessBuilder pb = new ProcessBuilder("/home/myscript");
Process p = pb.start();
BufferedReader stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));
String inputRead;
p.waitFor();
while((inputRead=stdInput.readLine()) != null){
Helper.log(inputRead);
}
This works fine but this time the bash script I am using didn't terminate. It's always active and when it detect something it print it. I want to start the script, wait for some time and than check if it detected something.
I tried this code
Thread.sleep(5000);
p.destroy();
String inputRead;
while((inputRead=stdInput.readLine()) != null){
Helper.log(inputRead);
}
But I get as result
java.io.IOException: Stream closed
I also tried
p.waitFor(5, TimeUnit.SECONDS);
while((inputRead=stdInput.readLine()) != null){
Helper.log(inputRead);
}
But the process never ended and I still get the input in the log for ever.
I just want to run the script for a short period of time, in this example 5 seconds and than stop it and get the output generated by the script in this period.

Process.destroy might (and probably will, but this is implementation dependent) kill your process and this is why you're getting your java.io.IOException.
If you're using Java 8, I'd suggest using Process.waitFor with the timeout constructor. You'll be able to check whether your process has hanged or not. If it returns true, you will have captured the process output using normal means. If it returns false, your process might have hanged, so you can either terminate it, or wait some more and try to see if anything has changed.
An alternative would be to control your bash script using the timeout command, so you can be sure that your script will eventually terminate in a finite amount of time.

p.waitFor() does not wait for process to complete (Java)

This is a spinoff of this thread: Killing a JFrame/Java when process finishes
I know this question has been asked various times here on SO but none of the answers seem to solve my problem. The issue I am having is this: I have a java program that calls a batch file. I need the batch file to finish before the Java program continues. For this I use the waitFor() method, which should wait until the process finishes. However, it doesn't. I found this thread, which suggests that the process waitFor() is actually waiting for is simply cmd.exe, which returns before the actual process does.
Runtime.exec().waitFor() doesn't wait until process is done
That thread recommends using the /wait command, but when I do it makes no difference. My code is very in-depth with several classes that all rely on one another, but here is an attempted SSCCE:
public class BatchFileRun(){
public static void main(String[] args){
ArrayList<String> list = new ArrayList<String>();
list.add("cmd.exe");
list.add("/C");
list.add("Start");
list.add("/wait");
list.add(("C:\\.....\\scripts\\myBatchFile.bat"));
list.add("batchVariable");
String[] commands = new String[list.size()];
commands = list.toArray(commands);
Process p = Runtime.getRuntime().exec(commands);
p.waitFor();
closeWindow();
}
void closeWindow(){
WindowEvent close = new WindowEvent(this, WindowEvent.WINDOW_CLOSING);
Toolkit.getDefaultToolkit().getSystemEventQueue().postEvent(close);
System.exit(0);
}
}
The batch file simply operates on the variable passed into it as a parameter (batchVariable). All prior commands should be used to simply run the batch file. The problem is that the closeWindow() method is called before the process in the batch file is finished. Therefore, the Java process that appears in the Windows task manager remains running, even though it should close when closeWindow() is called (if I am using it correctly). I really need said process to close when the process running from the batch file is complete. I have tried different ways of running the batch file (using ProcessBuilder, for example), but no matter what I try I have the same problem.

You can modify the batch file so it only finishes when the process it launches finishes.

Waiting for a single Shell script in a series of scripts to run to completion before continuing (Java, MySQL, JUnit)

I'm working on a Java program that incorporates Process and Runtime to run several shell scripts for automated testing. All but one of these scripts runs fine, which causes an issue with script calls following it. Ex:
process = runtime.exec("A.sh");
process = runtime.exec("B.sh");
process = runtime.exec("C.sh");
A.sh runs fine and only takes a few seconds to run. B.sh, however, takes a couple minutes to run, and I think this is causing a problem with running C.sh, since they both interact with the same MySQL table and the overlap causes a Communications Link Failure.
Without overloading you with unnecessary information, my question is, how can I wait to ensure a run shell script has been run to completion/termination before moving on to the next exec() call?
What I've tried:
process.waitFor()
This doesn't work, I don't think it waits until the script is completely done
process.wait(long time_period)
This doesn't work since it causes the current thread to wait which results in the remaining shell script calls to get skipped and the next test case to begin prematurely.
The shell script I call that causes the problem is not a simple script, but I didn't write it myself and have little understanding of what it does behind the scenes. The only relevant information I have about it is that it directly connects to the MySQL database in question whereas my program uses java.sql.* to (I believe) remotely connect (although it is a local database on a remote machine).
Edit:
After following a suggestion, I've looked into the Apache Commons Exec and tried a new strategy, unsuccessfully.
ExecuteWatchdog watchdog = new ExecuteWatchdog(300000); //For five minutes
CommandLine cmdline = CommandLine.parse("./directory/shell.sh");
DefaultExecutor executor = setExitValue(0);
executor.setWatchdog(watchdog);
int exitVal = executor.execute(cmdLine);
//A line to log the exit val in another file
My log gives no implication that the shell script was actually run, as the time between a logged statement saying "shell.sh begins" and "test 2 starts" are essentially the same instant, which means the ~2 minute process that shell.sh runs never happens. Where did I go wrong?

I use Apache Commons Exec. Have synchronous and asynchronous execution support. Execution timeout can be set.
First paragraph from their tutorial page:
At this point we can safely assume that you would like to start some
subprocesses from within your Java application and you spent some time
here to do it properly. You look at Commons Exec and think "Wow -
calling Runtime.exec() is easy and the Apache folks are wasting their
and my time with tons of code". Well, we learned it the hard way (in
my case more than once) that using plain Runtime.exec() can be a
painful experience. Therefore you are invited to delve into
commons-exec and have a look at the hard lessons the easy way ...
Advanced usage example (some code is missing like BusinessException and "StreamUtil.closeQuietly", but it could be easily replaced):
ExecuteWatchdog watchdog = new ExecuteWatchdog(EXECUTION_TIMEOUT_IN_MS);
DefaultExecutor executor = new DefaultExecutor();
executor.setWatchdog(watchdog);
executor.setExitValue(0);
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
ByteArrayOutputStream errorStream = new ByteArrayOutputStream();
executor.setStreamHandler(new PumpStreamHandler(outputStream, errorStream));
try {
log.info(commandLine.toString());
int exitCode = executor.execute(commandLine, (Map<?, ?>) null);
if (exitCode != 0)
throw new BusinessException("Process exited with non-zero exit code.");
return outputStream.toString();
} catch (ExecuteException e) {
String errorStreamStr = null;
if (errorStream.size() != 0)
errorStreamStr = errorStream.toString();
StringBuilder errorMessageBuilder = new StringBuilder();
errorMessageBuilder.append("main.error").append(":\n").append(
e.getMessage()).append("\n\n");
if (errorStreamStr != null) {
errorMessageBuilder.append("additional.error").append(":\n").append(errorStreamStr).append("\n\n");
}
errorMessageBuilder.append("command.line").append(":\n").append(commandLine.toString());
if (log.isDebugEnabled())
log.debug(errorMessageBuilder.toString());
throw new BusinessException(errorMessageBuilder.toString());
} catch (IOException e) {
throw new IllegalStateException(e);
} finally {
StreamUtil.closeQuietly(outputStream, errorStream);
}