We Keep Coding

How to fix "Non-monotonous DTS in output stream 0:1" when trying to convert a m3u8 file into a mp4 using ffmpeg

I'm currently coding a desktop util that downloads Twich Clips if they meet some requirements. As the twitch api gives as a response a m3u8 file, I have to convert it into a mp4. To do with I'm executing a cmd command through my Java application using the ProcessBuilder API built into Java. The command works fine, but sometimes I get a buggy video that has broken frames and audio. Here's my code:
public static void convertFile(File input, File output) {
String command = "ffmpeg -protocol_whitelist file,http,https,tcp,tls,crypto -vsync 2 -i \"" + input.getAbsolutePath() + "\" -c copy " + "\"" + output.getAbsolutePath() + "\"";
System.out.println("______________________________________________________");
System.out.println("Converting file...");
System.out.println("Using as input " + input.getAbsolutePath());
System.out.println("Using as output " + output.getAbsolutePath());
System.out.println("Using as command: " + command);
System.out.println("______________________________________________________");
setHasDone(false);
ProcessBuilder pb = new ProcessBuilder();
pb.command(command.split(" "));
pb.redirectOutput(ProcessBuilder.Redirect.INHERIT);
pb.redirectError(ProcessBuilder.Redirect.INHERIT);
Process process;
try {
process = pb.start();
}catch (IOException e){
e.printStackTrace();
return;
}
while (process.isAlive()){
setHasDone(false);
}
setHasDone(true);
}
Example of broken video:
https://youtu.be/FfFvStNl-9o
When uploaded to youtube lot's of the video bugs are not present, not really know why, but at the end you can clearly see what I'm talking about. Am I using ffmepg wrong?
I posted this as none of the 1000 answers on similar posts helped me.

Java program that calls external program behaving asynchronously?

All,
I originally had a shell script that called SQLLoader (Oracles data upload tool).
The problem was that SQLLoader takes a plain text password as input so I decided to build a Java application to call SQLLoader internally passing a decrypted password into the command string.
e.g.
sqlldr user/pass#DBServer control=../sqlloader.ctl log=sqlloader.log data=mydata.csv
So with my java wrapper it became this in my shell script
java -jar sqlloader.jar sqlloader.ctl mydata.csv
However a new problem developed when SQLLoader complained there was no file to load. After some head scratching it was discovered that a subsequent command in my shell script seemed to be executing while my java application was still running. Therefore it was behaving asynchronously.
The next command was moving the input file sqlloader was using before it could get a chance to use it. So I put a sleep command in of 20 seconds to give my java application time to run.
java -jar sqlloader.jar sqlloader.ctl mydata.csv
echo $?
sleep 20
if [ $? -ne 0 ]
then
echo "SQLLoader failed during execution, please check the log : "
mv mydata.csv
else
echo "SQLLoader successfully processed file : "
mv mydata.csv
fi
Does anyone know why unix is behaving this way, does Java execute my SQLLoader as a different user/ thread?
This is my java code:
Runtime Rt;
Process Prc;
Prc = Rt.exec("sqlldr user/decryptedpass#DBServer control=../sqlloader.ctl log=sqlloader.log data=mydata.csv);
system.exit(0);
I checked the Runtime Class for anything about it being Asynchronous but couldnt find anything
http://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.html
Any theories or suggestions?
Thanks

Yes. If you look at Runtime.exec again it does specify that it will launch a new process in the specified environment (e.g. independently of the current "environment" or as you put it asynchronously). You should use ProcessBuilder to create a Process and then waitFor that Process to finish before calling System.exit - which certainly isn't mandatory. Something like this
public static void main(String[] args) {
// String command = "/usr/bin/sleep 5";
List<String> command = new ArrayList<String>();
command.add("c:/cygwin/bin/sleep");
command.add("5");
ProcessBuilder pb = new ProcessBuilder(command);
BufferedReader is = null;
try {
System.out.println("Starting command " + command);
Process p = pb.start();
int ret = p.waitFor();
is = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line;
while ((line = is.readLine()) != null) {
System.out.println(line);
}
if (ret == 0) {
System.out.println("Command has completed.");
System.exit(ret);
} else {
System.out.println("Command completed with return code " + ret);
System.exit(ret);
}
} catch (Exception e) {
System.out.println("Caught Exception " + e.getMessage()
+ " running command " + command);
e.printStackTrace();
} finally {
if (is != null) {
try {
is.close();
} catch (IOException e) {
}
}
}
System.out.println("COMMAND FAILED");
System.exit(1);
}

You need to wait for process completion, you should also read all output (stdout and stderr) from the process you are starting.
If you call exit() after exec(), Java will do just that - exit immediatedly.
Here is an article that explains Runtime.exec pitfalls: http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html?page=4 (also consider the other pages).

How to detect java process exit?

In a java program, I am generating an sh script for use on a centOS machine, which will use sox and lame to decode an MP3 audio file, then apply some gain to the file respectively. Im having some issues getting the Process.waitFor() method to do anything other than hang indefinitely. Here is the code:
try
{
// TODO code application logic here
String reviewPath = "/SomeDirectory/";
String fileName = "FileName";
String extension = ".mp3";
StringBuilder sb = new StringBuilder();
sb.append("#!/bin/bash\n");
sb.append("cd " + reviewPath + "\n");
sb.append("lame --decode " + fileName + extension + "\n");
File script = new File(reviewPath + fileName + ".sh");
script.createNewFile();
script.setExecutable(true);
FileWriter writer = new FileWriter(script);
writer.write(sb.toString());
writer.close();
Process p = Runtime.getRuntime().exec(script.getAbsolutePath());
String line;
BufferedReader bri = new BufferedReader
(new InputStreamReader(p.getInputStream()));
BufferedReader bre = new BufferedReader
(new InputStreamReader(p.getErrorStream()));
while ((line = bri.readLine()) != null) {
System.out.println(line);
}
bri.close();
while ((line = bre.readLine()) != null) {
System.out.println(line);
}
bre.close();
p.waitFor();
System.out.println("Done.");
}
catch (Exception e)
{
System.out.println(e.getMessage());
}
The odd part is that when I run the .sh file it generates by hand, it runs and exits nicely, but when I execute it from a process object in java, it never exits. The exitValue of the process is always "Process has not exited". Ive tried adding set -e to the script, and exit to the end of the script. Short of using the kill command (which I dont really think I can do here) Im at a loss as to what is going on here. Any suggestions?

Add something like while(p.getInputStream().read() != -1); after starting the process. The buffer will get filled and the process will stop waiting for something (in this case, your program) to read from it to free up space.

I figured it out! The problem here was indeed that the output streams needed to be flushed for the application to exit, but simply reading from the streams is not enough. I used Suresh Koya's suggestion and used the processBuilder api, and redirected the error stream on the process before starting it, and read from the streams. This fixed the issues I was having :D

Download an exe file and run

I've been trying to make it where I can download a .exe file from the web, read it, and write it to a file locally, and then execute.
URL url = new URL("http://www.ddlands.com/downloads/Calc.exe");
URLConnection c = url.openConnection();
BufferedReader br = new BufferedReader(new InputStreamReader(c.getInputStream()));
String line = "";
File file = new File("analbread"+".exe");
BufferedWriter bw = new BufferedWriter(new FileWriter(file));
while((line = br.readLine()) != null){
bw.write(line + "\n");
}
br.close();
bw.close();
Process r = Runtime.getRuntime().exec("analbread" + ".exe");
System.out.println(r.toString());
System.out.println("WORKS!");
Although I know that doesn't work due to using BufferedWriter, and i'm not sure if it runs the exe.

For the downloading part, you'll need to use binary read/write. See this for further details: Working unbuffered streams.
For the executing part, the problem is that the Runtime.exec()-method can't launch your executable file.
At least under Linux (I can't test it on Windows), you'll need the full path to the executable file (or use ./[file] when the file is in the same directory as your application) to be able to execute it.
Only giving the command works for executables which are part of your systems PATH-variable.

Have a look at ClickOnce: http://en.wikipedia.org/wiki/ClickOnce
We've used that succesfully.

Ive used the following with good results to run command line scripts. You can create a batch script that runs the executable or run it directly using the exec method - probably pass "cmd ". This opens a command prompt from which you can run anything.
public static void runScript(String batchFile, boolean waitForExit0, int retryTime)
{
try
{
String runString = "cmd /c start " + (waitForExit0?"/wait ":"") + "/MIN " + batchFile;
Process p = Runtime.getRuntime().exec(runString); // /c start /wait
while (true)
{
try
{
int exit = p.exitValue();
if (exit == 0)
{
System.out.println("completed: " + runString);
return;
}
}
catch(Exception e)
{
String s = "";
}
Thread.sleep(retryTime);
}
}
catch(Exception e)
{
String s = "";
}
}

wkhtmltopdf process hangs in Java

I am running wkhtmltopdf from Java. I create a process but this one seems to be hanging, as it repeats again and again the the console, also in the Task Manager.
This is how I run wkhtmltopdf:
String command = applicationLocation + "wkhtmltopdf.exe -O Landscape " + reqURL + "?" + reqQuery + " c:/PDF/" + folderName + "/" + id + "/" + folderName + ".pdf";
Process p = Runtime.getRuntime().exec(command);
How can I "destroy" the process, after the job has been done?
This did not work for me, the process never stopped and the code never entered the while loop either:
ProcessBuilder pb = new ProcessBuilder(application, htmlFilePath, pdfFilePath);
Process process = pb.start();
BufferedReader errStreamReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
System.out.println("read errstreamreader");
//not "process.getInputStream()"
String line = null;
line = errStreamReader.readLine();
while(line != null) {
System.out.println(line);
line = errStreamReader.readLine();
if(line.equals("Done")) {
process.destroy();
System.out.println("destroyed process");
}
}

That was a very ugly problem. My code caused the loop. I called this class which generates the PDF from a servlet with this code:
// create pdf
if(action != null && action.equals("pdf")) {
String reqURL = request.getRequestURL().toString();
String reqQuery = "id=" + bomModuleId+ "&action=pdf";
String folderName = "doonot";
GeneratePDF obj = new GeneratePDF();
obj.genrateCmd(reqURL, "xxx", "xxx", reqQuery, folderName, "10.07.2012");
}
It turned out, that wkhtmltopdf used exactely the same URL, so it made a request to that page, landed in this loop, and called wkhtmltopdf again. So I ended up in like 450 wkhtmltopdf processes, and everything crashed.
Solution: I removed "&action=pdf"!

As it turns out i had the same problem too: looping wkhtmltopdf creates a bunch of processes causing my machine,running Windows 7, to drastically slow down.
The problem isnt really with wkhtmltopdf as it is with Runtime exec. It creates more wkhtmltopdf processes without waiting for any of the previous to finish until the lack of the system's resources freezes your computer.
My Solution: add p.waitFor(). It will wait for the wkhtmltopdf process to finish and then continue the loop.
This solution can still be slow because now a large html file can slow down the completion of your loop. So, if you want a still faster solution I suggest using the --read-from-stdin option from wkhtmltopdf.
As for reading the output of wkhtmltopdf I suggest looking at this question and ALL of its answers.