We Keep Coding

What could make the JVM fail catastrophically (or hang indefinitely) when executing ffmpeg?

I am trying to debug a Java microservice hosted in Kubernetes, in pods which are created dynamically by KEDA feeding off an SQS queue. Pods have dedicated resources of up to 3GB RAM, which to our knowledge is sufficient (although this assumption might need to be revisited).
The service at one point calls ffmpeg, and sometimes randomly this appears to cause catastrophic failure of the process. (Sometimes it succeed on the next attempt of exactly the same job.)
It is so catastrophic that no logs are generated, no exceptions thrown, and even the finally {} block is not even triggered.
This appears to occur in the remote k8 environment only and I have failed to replicate it locally.
What general reasons are there for a Java, executing another program with Execution.executeAndWait(), to fail so catastrophically?
How could I continue investigating the cause?
public void executeFfmpeg(String[] cmd) {
log.info("1"); // <-- this IS logged!
int retval = 0;
try {
retval = Execution.executeAndWait(cmd, stdout, stderr);
if ( retval != 0 ) {
log.error("FFMPEG broke. stderr: {}", errorFile); // <-- this log never occurs
throw new TranscodingFailedException("ffmpeg command failed: " + String.valueOf(retval) + " - " + String.join(" ", cmd)); // <-- never thrown
}
} catch (InterruptedException e) {
log.error("InterruptedException caught in the middle of the execution of ffmpeg. this will now proceed to crash...", e); // <-- this log never occurs
throw e;
} finally {
log.info("2"); // <-- THIS IS NOT CALLED!!! even in the finally block!?
stdout.close();
stderr.close();
}
}
/// Execution class:
public static int executeAndWait(String[] cmd, Writer stdout, Writer stderr) throws IOException, InterruptedException {
Process proc = null;
ExecutorService outExecutor = Executors.newSingleThreadExecutor();
ExecutorService errExecutor = Executors.newSingleThreadExecutor();
try {
proc = exec(cmd, stdout, stderr, outExecutor, errExecutor);
// block until sub-process exits
return proc.waitFor();
} finally {
shutdownProcess(proc);
shutdownAndAwaitTermination(outExecutor);
shutdownAndAwaitTermination(errExecutor);
}
}
private static void shutdownProcess(Process proc) {
try {
if (proc != null) {
proc.destroy();
}
} catch (Exception e) {
log.error(e.getMessage(), e);
}
}
private static void shutdownAndAwaitTermination(ExecutorService pool) {
try {
pool.shutdown(); // Disable new tasks from being submitted
try {
// Wait a while for existing tasks to terminate
if (!pool.awaitTermination(800, TimeUnit.MILLISECONDS)) {
pool.shutdownNow(); // Cancel currently executing tasks
}
} catch (InterruptedException ie) {
// (Re-)Cancel if current thread also interrupted
pool.shutdownNow();
}
} catch (Exception e) {
log.error(e.getMessage(), e);
}
}
private static Process exec(String[] cmd, Writer stdout, Writer stderr, ExecutorService outExecutor, ExecutorService errorExecutor) throws IOException {
// execute input command in a sub-process
Process proc = Runtime.getRuntime().exec(cmd);
StreamConsumer outConsumer = new StreamConsumer(proc.getInputStream(), stdout);
StreamConsumer errConsumer = new StreamConsumer(proc.getErrorStream(), stderr);
// execute data read/write in separate threads
outExecutor.submit(outConsumer);
errorExecutor.submit(errConsumer);
return proc;
}
An example of the ffmpeg command is:
/usr/local/bin/ffmpeg -i /tmp/transcode-a5ff7706-488a-4e24-9ef8-9657d1254a26626348807122071896/str_CAM_Z2VH_con_H0Zqr2flbT.webm -filter_complex [0:v]setpts=0.8363824*PTS,scale=w=1280:h=720:force_original_aspect_ratio=1,pad=1280:720:(ow-iw)/2:(oh-ih)/2[v0];[0:a]aresample=async=1000[0sync] -map [v0] -map [0sync] -vsync vfr -r 25 -c:v libx264 -c:a mp3 /tmp/transcode-a5ff7706-488a-4e24-9ef8-9657d1254a26626348807122071896/intermediate-str_CAM_Z2VH_con_H0Zqr2flbT.webm.mkv | stdout: /tmp/intermediate-stdout-12310474463787935763.log | stderr: /tmp/intermediate-stderr-12166558954928907997.log

Some suggestions for checking that the sub-process handling is correct:
Carefully check the logs to see if ffmpeg is waiting on input (eg overwrite file confirmation etc). You could eliminate that possibility by closing STDIN explicitly (also see -y flag to force overwrite, or -nostdin to remove use of stdin altogether):
proc.getOutputStream().close();
Your command looks like a shell command as it contains pipe so isn't valid for direct launch from Java, drop the pipe and following parts, or change command to launch inside a shell which would handle the pipe correctly:
/usr/local/bin/ffmpeg -i ... | stdout: xxx.log | stderr: yyy.log
=>
/usr/local/bin/ffmpeg -i ...
or
{"bash","-c","/usr/local/bin/ffmpeg -i ... | stdout: xxx.log | stderr: yyy.log"}
The STDOUT/ERR streams should be dealt with before calling proc.waitFor(), not afterwards and not after destroy(). You may be better off ensuring that these streams end inside exec():
// execute data read/write in separate threads
Future<?> so = outExecutor.submit(outConsumer);
Future<?> se = errorExecutor.submit(errConsumer);
// Await STDOUT/ERR termination:
try {
so.get();
se.get();
} catch (InterruptedException |ExecutionException e) {
throw new RuntimeException("failed", e);
}

process.waitFor(timeout, timeUnit) does not quit the process after specified time

I'm trying to execute a visual basic script code in my java application using process builder. As script provided by the user might not finish its execution in time, I want to provide means to limit this execution time. In the following code, you can see my logic but it doesn't really do what it supposed to do. How can I make this waitfor work in order to limit the execution time?
private void run(String scriptFilePath) throws ScriptPluginException {
BufferedReader input = null;
BufferedReader error = null;
try {
ProcessBuilder p = new ProcessBuilder("cscript.exe", "//U", "\"" + scriptFilePath + "\"");
String path = "";
if (scriptFilePath.indexOf("/") != -1) {
path = scriptFilePath.substring(0, scriptFilePath.lastIndexOf("/"));
}
path += "/" + "tempvbsoutput.txt";
p.redirectOutput(new File(path));
Process pp = p.start();
try {
pp.waitFor(executionTimeout, TimeUnit.MINUTES);
} catch (InterruptedException e) {
SystemLog.writeError(jobId, ScriptConsts.COMPONENT_ID, "VBScriptExecutor", "run", 80401104,
"VB Script executes fail.");
}
if (!pp.isAlive()) {
pp.getOutputStream().close();
}
// rest of the code flow
}

Process.waitFor(long, TimeUnit) waits until the process has terminated or the specified time elapsed (Javadoc). The return value indicates whether the process exited or not.
if (process.waitFor(1, TimeUnit.MINUTES)) {
System.out.println("process exited");
} else {
System.out.println("process is still running");
}
waitFor() does not kill the process after the time elapsed.
If you want to kill the subprocess, use either destroy() or destroyForcibly().

Detecting completion of process inside microservice

I have a Java microservice which shells out to execute a program and then monitors stderr until nothing is returned:
Process p = Runtime.getRuntime().exec("/bin/sh /var/task/bin/iTMSTransporter " + commandLine);
BufferedReader stdError = new BufferedReader(new InputStreamReader(p.getErrorStream()));
logger.log("StdErr:\n");
while ((s = stdError.readLine()) != null && p.isAlive()) {
System.out.println("stderr: " + s);
if (s.indexOf("DBG-X") == -1) {
result.stdErr += s + "\n";
}
}
this.logger.log("finished reading stderr\n");
This was effective at detecting the completion of the external program and it always worked. Now the external program has been updated and it seems to start outputting stderr but then just stops (there should be more to the stderr stream) and eventually it times out.
I then added the p.isAlive() as an attempt to capture the shelled program's completion. This seemed to have no impact. Now here's the frustrating part ... in an older version of my microservice I used NodeJS instead of Java to shell out to run this program. The NodeJS version still works by listening for the close event:
shell.stdout.on('data', data => {
stdout += data;
});
shell.stderr.on('data', data => {
stderr += data;
});
shell.on('error', error => {
reject(error);
});
shell.on('close', () => {
resolve({
stdout: stdout,
stderr: stderr
});
});
Is there something equivalent I can do with Java?
---- Addition -----
I tried something I'd seen online:
Runtime rt = Runtime.getRuntime();
rt.addShutdownHook(new Thread(new Runnable() {
public void run() {
System.out.println("\nGOT HERE\n");
}
}));
Process p = rt.exec("/bin/sh /var/task/bin/iTMSTransporter " + commandLine);
thinking I'd get it detect in a new thread and output "GOT HERE" but that never is sent to console.

If you just want to wait for the process to finish, waitFor should be appropriate.
Process p = Runtime.getRuntime().exec("/bin/sh /var/task/bin/iTMSTransporter " + commandLine);
try {
p.waitFor();
// now, the process has terminated
} catch (InterruptedException e) {
// something went wrong
e.printStackTrace();
}
If you do want to capture the out- and err-streams, that's actually somewhat complicated because you'd need to create new threads for that (not that that's a huge problem, but it's OTT if you just want to know that the process has finished).
Also maybe see process.waitFor() never returns.

Java subprocess terminated by itself

My application uses some daemon subprocesses for subtasks. The subprocesses are launched using ProcessBuilder and working fine on their own, but then starting them as subprocesses every associated Process.isAlive() method return FALSE. As following, no access to process is possible.
Further investigation shows the subprocesses are not started at all (don't exist in Task Manager) with no error generated at all.

Daemons typically start a separate process and exit almost immediately, which makes checks like isAlive() useless.
Often the program will have a command line switch that make the program stay in the foreground, not becoming a daemon - use that if possible. Otherwise you'll need some other way of monitoring the daemon execution, for example using the daemon's PID file.

Is the command really running? Often there are weird little issues when trying to run a program from inside Java.
For example, the PATH environment variable may not be set correctly so it fails to load a dependency.
Use this method to see if there is any console output and what the exit code is. This uses the old Runtime class instead of ProcessBuilder. It can probably be adapted to use ProcessBuilder.
public static void runExe(String[] command) throws IOException {
Runtime runtime = Runtime.getRuntime();
long start = System.currentTimeMillis();
Process proc = runtime.exec(command);
BufferedReader br = new BufferedReader(new InputStreamReader(proc.getInputStream()));
try {
while (true) {
// enter a loop where we read what the program has to say and wait for it to finish
// read all the program has to say
while (br.ready()) {
String line = br.readLine();
System.out.println("CMD: " + line);
}
try {
int exitCode = proc.exitValue();
System.out.println("exit code: " + exitCode);
// if we get here then the process finished executing
break;
} catch (IllegalThreadStateException ex) {
// ignore
}
// wait 200ms and try again
Thread.sleep(200);
}
} catch (InterruptedException ex) {
ex.printStackTrace();
}
long end = System.currentTimeMillis();
System.out.println("Command took: " + (end - start) + "ms");
}

Java runtime.exec() running multiple process using Threadpool

In my program I have a list of n test scripts , I need to iterate the list and run 3 test scripts in parallel. To achieve this task I created a Threadpool of size 3. My implementation is like below for the thread pool
ExecutorService executor = Executors.newFixedThreadPool(3);
for (int threadpoolCount = 0; threadpoolCount < classNames.size(); threadpoolCount++) {
Runnable worker = new ProcessRunnable(classNames.get(threadpoolCount));
executor.execute(worker);
}
executor.shutdown();
while (!executor.isTerminated()) {
}
System.out.println("Finished all threads");
below is my thread implementation where in i execute a batch file with maven commands in it
public void run() {
try {
System.out.println(testname);
System.out.println("Task ID : " + this.testname + " performed by " + Thread.currentThread().getName());
Process p = Runtime.getRuntime().exec("cmd /C Junit_runner.bat" + " " + testname);
p.waitFor();
Thread.sleep(5000);
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
Here is what I get in my console (I am not starting the command prompt and running it in background)
com.selenium.test.testname1
Task ID : com.selenium.test.testname1
performed by pool-1-thread-1
com.selenium.test.testname1
Task ID : com.selenium.test.testname2
performed by pool-1-thread-2
com.selenium.test.testname1
Task ID : com.selenium.test.testname3
performed by pool-1-thread-3
The execution pauses here and it didn't do anything , I am not sure what's happening behind. I also cross checked that the batch file its working fine.

The process takes long to execute and so your control is not returning back.
public abstract int waitFor() throws InterruptedException
Causes the current thread to wait, if necessary, until the process represented by this Process object has terminated. This method returns immediately if the subprocess has already terminated. If the subprocess has not yet terminated, the calling thread will be blocked until the subprocess exits.
As waitFor() is a blocking call all the 3 threads are stuck at this line.
NOTE: You don't need Thread.sleep(5000); as waitFor() is itself blocking in nature.
Try executing some other command and see if the control returns.
Also instead of:
while (!executor.isTerminated()) {
}
You can use ExecutorService#awaitTermination()

Read the p.getInputStream() and p.getErrorStream() and write it to your console instead of thread.sleep(), you will get an indication of what the threads are doing.