We Keep Coding

Nothing happens until the external python command finish running in java

I have a logText(String text) function which basically shows some text in a text area. I need some text to be shown before running the external python command. But the text is shown after the execution of external python command.
Here is my code.
logText("Please Wait Until The Testing Is Finished");
logText("Starting Testing...");
String command = "python3 python/Predict.py";
try {
String line;
Process process = Runtime.getRuntime().exec(command);
process.waitFor();
BufferedReader error = new BufferedReader(new InputStreamReader(process.getErrorStream()));
error.close();
BufferedReader input = new BufferedReader(new InputStreamReader(process.getInputStream()));
input.close();
OutputStream outputStream = process.getOutputStream();
PrintStream printStream = new PrintStream(outputStream);
printStream.println();
printStream.flush();
printStream.close();
logText("Images Created At Output Directory");
logText("Testing Completed");
} catch (IOException ex) {
Logger.getLogger(MasterFrame.class.getName()).log(Level.SEVERE, null, ex);
} catch (InterruptedException ex) {
Logger.getLogger(MasterFrame.class.getName()).log(Level.SEVERE, null, ex);
}
My logText() function.
private void logText(String logText) {
if (logArea.getText().equals("")) {
logArea.setText(">>> " + logText);
} else {
logArea.append(System.lineSeparator() + ">>> " + logText);
}
}
How can I show the text in the text area before execution of python command?

what you are doing is a classic example of running long-running code on GUI thread and as a result, freezing the main thread. The solution is the same as always (no matter what language or framework): run the long-running work in a background thread such as a new Thread(...) or a SwingWorker.
for example:
logText("Please Wait Until The Testing Is Finished");
logText("Starting Testing...");
String command = "python3 python/Predict.py";
new Thread(new Runnable() {
public void run() {
try {
String line;
Process process = Runtime.getRuntime().exec(command);
process.waitFor();
BufferedReader error = new BufferedReader(new InputStreamReader(process.getErrorStream()));
error.close();
BufferedReader input = new BufferedReader(new InputStreamReader(process.getInputStream()));
input.close();
OutputStream outputStream = process.getOutputStream();
PrintStream printStream = new PrintStream(outputStream);
printStream.println();
printStream.flush();
printStream.close();
SwingUtilities.invokeLater(new Runnable() {
public void run() {
// Here, we can safely update the GUI
// because we'll be called from the
// event dispatch thread
logText("Images Created At Output Directory");
logText("Testing Completed");
}
});
} catch (IOException ex) {
Logger.getLogger(MasterFrame.class.getName()).log(Level.SEVERE, null, ex);
} catch (InterruptedException ex) {
Logger.getLogger(MasterFrame.class.getName()).log(Level.SEVERE, null, ex);
}
}
}).start();
p.s. as you can see in the comments the last two commands in the new thread needed to be called with SwingUtilities.invokeLater(...).

FFMpeg process created from Java on CentOS doesn't exit

I need to convert a lot of wave files simultaneously. About 300 files in parallel. And new files come constantly. I use ffmpeg process call from my Java 1.8 app, which is running on CentOS. I know that I have to read error and input streams for making created process from Java possible to exit.
My code after several expirements:
private void ffmpegconverter(String fileIn, String fileOut){
String[] comand = new String[]{"ffmpeg", "-v", "-8", "-i", fileIn, "-acodec", "pcm_s16le", fileOut};
Process process = null;
BufferedReader reader = null;
try {
ProcessBuilder pb = new ProcessBuilder(comand);
pb.redirectErrorStream(true);
process = pb.start();
//Reading from error and standard output console buffer of process. Or it could halts because of nobody
//reads its buffer
reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String s;
//noinspection StatementWithEmptyBody
while ((s = reader.readLine()) != null) {
log.info(Thread.currentThread().getName() + " with fileIn " + fileIn + " and fileOut " + fileOut + " writes " + s);
//Ignored as we just need to empty the output buffer from process
}
log.info(Thread.currentThread().getName() + " ffmpeg process will be waited for");
if (process.waitFor( 10, TimeUnit.SECONDS )) {
log.info(Thread.currentThread().getName() + " ffmpeg process exited normally");
} else {
log.info(Thread.currentThread().getName() + " ffmpeg process timed out and will be killed");
}
} catch (IOException | InterruptedException e) {
log.error(Thread.currentThread().getName() + "Error during ffmpeg process executing", e);
} finally {
if (process != null) {
if (reader != null) {
try {
reader.close();
} catch (IOException e) {
log.error("Error during closing the process streams reader", e);
}
}
try {
process.getOutputStream().close();
} catch (IOException e) {
log.error("Error during closing the process output stream", e);
}
process.destroyForcibly();
log.info(Thread.currentThread().getName() + " ffmpeg process " + process + " must be dead now");
}
}
}
If I run separate test with this code it goes normally. But in my app I have hundreds of RUNNING deamon threads "process reaper" which are waiting for ffmpeg process finish. In my real app ffpmeg is started from timer thread. Also I have another activity in separate threads, but I don't think that this is the problem. Max CPU consume is about 10%.
Here is that I usual see in thread dump:
"process reaper" #454 daemon prio=10 os_prio=0 tid=0x00007f641c007000 nid=0x5247 runnable [0x00007f63ec063000]
java.lang.Thread.State: RUNNABLE
at java.lang.UNIXProcess.waitForProcessExit(Native Method)
at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:289)
at java.lang.UNIXProcess$$Lambda$32/2113551491.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
What am I doing wrong?
UPD:
My app accepts a lot of connects with voice traffic. So I have about 300-500 another "good" threads in every moment. Could it be the reason? Deamon threads have low priority. But I don't beleive that they really can't do their jobs in one hour. Ususally it takes some tens of millis.
UPD2:
My synthetic test that runs fine. I tried with new threads option and without it just with straigt calling of run method.
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.InputStreamReader;
public class FFmpegConvert {
public static void main(String[] args) throws Exception {
FFmpegConvert f = new FFmpegConvert();
f.processDir(args[0], args[1], args.length > 2);
}
private void processDir(String dirPath, String dirOutPath, boolean isNewThread) {
File dir = new File(dirPath);
File dirOut = new File(dirOutPath);
if(!dirOut.exists()){
dirOut.mkdir();
}
for (int i = 0; i < 1000; i++) {
for (File f : dir.listFiles()) {
try {
System.out.println(f.getName());
FFmpegRunner fFmpegRunner = new FFmpegRunner(f.getAbsolutePath(), dirOut.getAbsolutePath() + "/" + System.currentTimeMillis() + f.getName());
if (isNewThread) {
new Thread(fFmpegRunner).start();
} else {
fFmpegRunner.run();
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
class FFmpegRunner implements Runnable {
private String fileIn;
private String fileOut;
FFmpegRunner(String fileIn, String fileOut) {
this.fileIn = fileIn;
this.fileOut = fileOut;
}
#Override
public void run() {
try {
ffmpegconverter(fileIn, fileOut);
} catch (Exception e) {
e.printStackTrace();
}
}
private void ffmpegconverter(String fileIn, String fileOut) throws Exception{
String[] comand = new String[]{"ffmpeg", "-i", fileIn, "-acodec", "pcm_s16le", fileOut};
Process process = null;
try {
ProcessBuilder pb = new ProcessBuilder(comand);
pb.redirectErrorStream(true);
process = pb.start();
//Reading from error and standard output console buffer of process. Or it could halts because of nobody
//reads its buffer
BufferedReader reader =
new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
//noinspection StatementWithEmptyBody
while ((line = reader.readLine()) != null) {
System.out.println(line);
//Ignored as we just need to empty the output buffer from process
}
process.waitFor();
} catch (IOException | InterruptedException e) {
throw e;
} finally {
if (process != null)
process.destroy();
}
}
}
}
UPD3:
Sorry, I forgot to notice that I see the work of all these process - they created new converted files but anyway don't exit.

Your application is I/O bound, not CPU bound. If all your files are in the same HDD, then opening simultaneously 300 files will definitely degrade the performance. (that is a likely reason on why you have hundreds of processes waiting).
If I understood correctly, you mentioned that processing 1 file takes some tens of millis? (and this is doing sequential reads - the fastest that your HDD will read a file)
in this case, processing 300 files sequentially should take no more than 30 seconds.
100 millis - process 1 file
1 second - process 10 files
30 second - process 300 files
EDIT
I did 2 simple changes to your sample code (I removed the first loop, then changed the codec) finally I put one song in "ogg" format in "/tmp/origin" directory. now the program works well).
see code below:
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.InputStreamReader;
public class FFMpegConvert {
public static void main(String[] args) throws Exception {
FFMpegConvert f = new FFMpegConvert();
f.processDir("/tmp/origin", "/tmp/destination", false);
}
private void processDir(String dirPath, String dirOutPath, boolean isNewThread) {
File dir = new File(dirPath);
File dirOut = new File(dirOutPath);
if (!dirOut.exists()) {
dirOut.mkdir();
}
for (File f : dir.listFiles()) {
try {
System.out.println(f.getName());
FFmpegRunner fFmpegRunner = new FFmpegRunner(f.getAbsolutePath(), dirOut.getAbsolutePath() + "/" + System.currentTimeMillis() + f.getName());
if (isNewThread) {
new Thread(fFmpegRunner).start();
} else {
fFmpegRunner.run();
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
class FFmpegRunner implements Runnable {
private String fileIn;
private String fileOut;
FFmpegRunner(String fileIn, String fileOut) {
this.fileIn = fileIn;
this.fileOut = fileOut;
}
#Override
public void run() {
try {
ffmpegconverter(fileIn, fileOut);
} catch (Exception e) {
e.printStackTrace();
}
}
private void ffmpegconverter(String fileIn, String fileOut) throws Exception {
String[] comand = new String[]{"ffmpeg", "-i", fileIn, "-acodec", "copy", fileOut};
Process process = null;
try {
ProcessBuilder pb = new ProcessBuilder(comand);
pb.redirectErrorStream(true);
process = pb.start();
//Reading from error and standard output console buffer of process. Or it could halts because of nobody
//reads its buffer
BufferedReader reader =
new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
//noinspection StatementWithEmptyBody
while ((line = reader.readLine()) != null) {
System.out.println(line);
//Ignored as we just need to empty the output buffer from process
}
process.waitFor();
} catch (IOException | InterruptedException e) {
throw e;
} finally {
if (process != null)
process.destroy();
}
}
}
}

Got it!
In some cases ffmpeg wants to ask me should it override already existed file. In my code I close outputstream of this child process. But as it appears this only closes outputstream for Java but not for the process.
So my solution is to make ffmpeg silent at all: no output from this process with "-v -8", no asking question with default "Yes" "-y".

Detecting terminal command errors in Java

In my Java application I am using the exec() command to call a terminal function:
p = Runtime.getRuntime().exec(command);
p.waitFor();
The call uses the zip and unzip calls. Originally I call:
zip -P password -r encrypted.zip folderIWantToZip
When I call the unzip function through java, I specify the password as the method parameter. If the correct password is specified then the call should unzip the encrypted folder:
unzip -P password encrypted.zip
I want a way to find out if the password entered is incorrect. For example, if password is correct, then the call will correctly unzip the zip file. But I notice that no exception is thrown for an incorrect password. How can I determine this?

You could read the process's ErrorStream and InputStream to determine the process output. Sample code given below
public static void main(String[] args) {
try {
String command = "zip -P password -r encrypted.zip folderIWantToZip";
Process p = Runtime.getRuntime().exec(command);
InputStream is = p.getInputStream();
int waitFor = p.waitFor();
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
String line;
while ((line = reader.readLine()) != null) {
System.out.println("line:" + line);
}
is = p.getErrorStream();
reader = new BufferedReader(new InputStreamReader(is));
while ((line = reader.readLine()) != null) {
System.out.println("ErrorStream:line: " + line);
}
System.out.println("waitFor:" + waitFor);
System.out.println("exitValue:" + p.exitValue());
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
You could use the exitcode to validate the process status as well but it is specific to to program. Normally zero means successfully terminated otherwise abnormal termination.

As per my comment, first thing I would do would be to capture the Process's InputStream and ErrorStream via getInputStream() and getErrorStream(), but especially the latter, the ErrorStream, and check to see what it outputs if the input is in error. Note that these would have to be done in their own thread, else you'll tie up your program. I usually use some type of StreamGobbler class for this. Also, don't ignore the int returned by p.waitFor().
e.g.,
ProcessBuilder pBuilder = new ProcessBuilder(COMMAND);
Process process = null;
try {
process = pBuilder.start();
new Thread(new StreamGobbler("Input", process.getInputStream())).start();
new Thread(new StreamGobbler("Error", process.getErrorStream())).start();
int exitValue = process.waitFor();
System.out.println("Exit Value: " + exitValue);
process.destroy();
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
if (process != null) {
process.destroy();
}
}
And:
class StreamGobbler implements Runnable {
private String name;
private Scanner scanner;
public StreamGobbler(String name, InputStream inputStream) {
this.name = name;
scanner = new Scanner(inputStream);
}
#Override
public void run() {
while (scanner.hasNextLine()) {
String line = scanner.nextLine();
System.out.println(name + ": " + line); // or better, log the line
}
scanner.close();
}
}

Process started from Java hangs

I am trying to execute a c++ code from java on a remote Windows machine. In order to deal with the remote part, I have created a Web service from where the actual command is run using Runtime.exec(). The c++ exe is not being called directly from the java code. I have a batch file that eventually calls the exe.
The problem is, both java and c++ processes hang. The java code on server side does handle the output stream and error stream. Also, the c++ code is logging everything in a file on Windows. The strange thing is that, when I remove the WS call and run the java code on server side as a standalone java program, it succeeds. Here is the java code:
public class RunCPlusPlusExecutable {
public int runExecutable() {
int exitValue = 0;
try {
Process p = null;
Runtime rt = Runtime.getRuntime();
System.out.println("About to execute" + this + rt);
p = rt.exec("c:/temp/execcplusplus.bat");
System.out.println("Process HashCode=" + p.hashCode());
StreamProcessor errorHandler = new StreamProcessor(p.getErrorStream(), "Error");
StreamProcessor outputHandler = new StreamProcessor(p.getInputStream(), "Output");
errorHandler.start();
outputHandler.start();
exitValue = p.waitFor();
System.out.println("Exit value : " + exitValue);
if (exitValue == 0)
System.out.println("SUCCESS");
else
System.out.println("FAILURE");
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (Exception e) {
}
return exitValue;
}
class StreamProcessor extends Thread {
private InputStream is = null;
private String type = null;
private InputStreamReader isr = null;
private BufferedReader br = null;
private FileWriter writer = null;
private BufferedWriter out = null;
StreamProcessor(InputStream is, String type) {
this.is = is;
this.type = type;
}
public void run() {
try {
isr = new InputStreamReader(is);
br = new BufferedReader(isr);
writer = new FileWriter("*******path to log file********");
out = new BufferedWriter(writer);
String line = null;
while ((line = br.readLine()) != null) {
Date date = new Date();
out.write("[" + type + "]: " + date + " : " + line);
out.newLine();
}
writer.flush();
} catch (IOException ioe) {
ioe.printStackTrace();
} finally {
try {
if (br != null)
br.close();
if (isr != null)
isr.close();
if (out != null)
out.close();
if (writer != null)
writer.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
}
Any idea what is causing the problem and how to debug it? Please note that I won't be able to debug the c++ code.
Thanks
Update 1:
Here are some more details...
The WS server is running from some admin user. And I have been running the standalone java program from some other user.
*It seems that the c++ executable is giving referenced memory error while executing from WS call. There are pop-ups citing the error with OK and Cancel buttons. *
Update 2:
The tomcat server where the WS is deployed is running as a Windows NT service. Can that be the cause of the error? If yes, how to resolve this?

StreamGobbler short output or very quick process

I use StreamGobbler to read output from external program. Before I used to read and parse big(20-40 strings) output from stderr of ffmpeg. Now I want to read and parse only one string from stdout(one string is all output of program and this program ends very quickly) of identify(ImageMagick). And sometimes it works, but sometimes I have error "Stream closed". I think that StreamGobbler has no time to work with stream(process ends before streamGobler do some work).
Below you can see class ExecThread and example of it's usage.
Sorry for my English...
String command = ffmpegExe.getAbsolutePath()+ " -i \""+ fileName +"\"";
ExecThread thread = new ExecThread(command);
thread.setPriority(ExecThread.MIN_PRIORITY);
thread.start();
thread.waitFor(DEFAULT_WAIT);
//reading ffmpeg stderr
BufferedReader ffmpegErr = thread.getErrBufferedReader();
String line;
try
{
while((line = ffmpegErr.readLine()) !=null)
//some code
private static class ExecThread extends Thread
{
public ExecThread(String command)
{
this.command = command;
}
#Override
public void run()
{
try
{
ExecCommand(command);
exitVal = process.waitFor();
}
catch (Exception e )
{
logger.error("Error while executing command: " + command + e.getMessage(),e);
System.err.println("Error");
System.exit(1);
}
}
public void waitFor(long millis)
{
try
{
this.join(millis);
process.destroy();
} catch (Exception e)
{
logger.error("Error while interupting command: " + command + e.getMessage(), e);
}
}
private void ExecCommand(String command) throws IOException
{
//run command
Runtime rt = Runtime.getRuntime();
process = rt.exec(new String[]{shellName,shellOption, command}, null, null);
//get stderr buffered reader
StreamGobbler errGobbler = new StreamGobbler(process.getErrorStream());
StreamGobbler outGobbler = new StreamGobbler(process.getInputStream());
ErrBufferedReader = new BufferedReader(new InputStreamReader(errGobbler));
OutBufferedReader = new BufferedReader(new InputStreamReader(outGobbler));
}
private BufferedReader ErrBufferedReader;
public BufferedReader getOutBufferedReader()
{
return OutBufferedReader;
}
private BufferedReader OutBufferedReader;
private Process process;
private String command;
private int exitVal=-1;
public BufferedReader getErrBufferedReader()
{
return ErrBufferedReader;
}
public Process getProcess()
{
return process;
}
public String getCommand()
{
return command;
}
public int getExitVal()
{
return exitVal;
}
}

I'd recommend two changes:
Use the Java Process instead of writing a Thread.
Close your input, output, and error streams and destroy the Process. You'll have file handle leaks if you don't.

This is not good decision but it works. Before it was String command = identifyExe.getAbsolutePath() + " \"" + file.getAbsolutePath() + "\"";
now it String command = "sleep 1 ; " + identifyExe.getAbsolutePath() + " \"" + file.getAbsolutePath() + "\"";
Sleep is very usefull function :-)
But, if you know better decision please tell...