I wrote a java class in order to perform multithreaded tasks, each task running an external process.
The process is in charge of converting ".chp" files into ".txt" files. It is written in C.
This process breaks at one point because it disappears when looking at a "top" in my terminal (probably due to a corrupted chp file). The problem is that the process in my java thread does not return. The "process.waitFor()" seems to go on forever (at least 'til the 12 hours I specified for the ExecutorService.
Am I doing something wrong (not catching an exception?)?
I tried setting a class variable of type String in MyThread and putting an error message in place of throwing a new RuntimeException, then print the String at the end of the main, but the thread code doesn't reach to this point. It still gets stuck at the waitFor().
Shouldn't the process terminate once the C program has failed?
The program prints on the terminal (cf: MyThread):
A
B
C
main:
String pathToBin = "/path/to/bin";
List<MyThread> threadList = new ArrayList<MyThread>();
for (File f : folderList) {
File[] chpFilesInFolder = f.listFiles(new FilenameFilter() {
#Override
public boolean accept(File dir, String name) {
if (name.endsWith(".chp")){
return true;
}else{
return false;
}
}
});
File chpFile = writeChpFiles(chpFilesInFolder);
String[] cmd = {pathToBin, "--arg1", chpFile, "--out-dir", outputFolder};
MyThread t = new MyThread(cmd, f, chpFilesInFolder);
threadList.add(t);
}
ExecutorService threadExecutor = Executors.newFixedThreadPool(4);
for(MyThread th : threadList){
threadExecutor.execute(th);
}
threadExecutor.shutdown();
try {
threadExecutor.awaitTermination(12, TimeUnit.HOURS);
} catch (InterruptedException e) {
e.printStackTrace();
}
MyThread:
class MyThread extends Thread{
private String[] cmd;
private File chpFolder;
private File[] chpFilesInFolder;
public MyThread(String[] cmd, File chpFolder, File[] chpFilesInFolder){
this.cmd = cmd;
this.chpFolder = chpFolder;
this.chpFilesInFolder = chpFilesInFolder;
}
#Override
public void run() {
Process process = null;
try{
System.err.println("A ");
ProcessBuilder procBuilder = new ProcessBuilder(cmd);
procBuilder.redirectErrorStream(true);
System.err.println("B");
process = procBuilder.start();
System.err.println("C");
process.waitFor();
System.err.println("D");
if(process.exitValue()!=0) System.err.println("ERROR !"+process.exitValue());
System.err.println("E");
}catch(IOException e){
e.printStackTrace();
}catch(InterruptedException e){
e.printStackTrace();
}catch(Throwable e){
e.printStackTrace();
}finally{
System.err.println("F");
if(process!=null) {try { process.destroy();} catch(Exception err) {err.printStackTrace();}}
}
File[] txtFilesInFolder = chpFolder.listFiles(new FilenameFilter() {
#Override
public boolean accept(File dir, String name) {
if (name.endsWith(".chp.txt")){
return true;
}else{
return false;
}
}
});
if (txtFilesInFolder.length==chpFilesInFolder.length){
for (File chp : chpFilesInFolder) {
chp.delete();
}
File logFile = new File(chpFolder, "apt-chp-to-txt.log");
if (logFile.exists()){
logFile.delete();
}
}else{
throw new RuntimeException("CHPs have not all been transformed to TXT in "+chpFolder.getAbsolutePath());
}
Is it possible that your C program is producing output on stdout? If so, you need to read Process.getOutputStream() before Process.waitFor() returns - see http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4254231
Alternatively, call your C program that a shell script that redirects stdout.
You can use the jstack command to confirm that the thread is indeed blocked at Process.waitFor().
You could have the main thread wait for a reasonable amount of time and then call some method on the MyThread class to kill the started process, thus causing the thread to finish.
as often I would suggest to use a more robust and professional point of view while using a messsaging solution to make your C program interact with your Java application, it will be easy and clean to avoid those non daemon threads waiting for ever because of the crash of your C application... now all brokers have a STOMP interface which is pretty cool for any kind of application to invoke (just use any Http library), broker configuration will enable to restart non finished jobs, to put some timeouts and so one..Even if JMS does not support request and response it's quite easy to implement such paradigm....
HTH
Jerome
If I understad correctly, your Java threads remain waiting after the C program crashes.
Make the spawned C process send heart beats. You can do this even by printing sth to console (or inserting in a table) and have the Java thread every so often wake up and check the heartbeat. If it's not there, assume the C process died and terminate the thread.
Launching external processes in Java can get a little bit tricky. I usually try to avoid them as you'll have to deal with different error codes and some terminal madness. I recommend you use specialized libraries such as commons-exec (http://commons.apache.org/exec/)
Related
The InputStream of my Process should attach and detach whenever the user wants to see it or not. The attaching works fine, but the detach fails. Default answer to interrupt the readLine() method is always to close the stream, but I cant in this case or the Process will finish or at least not available for future attachments. This is how the stream is read:
BufferedReader reader = new BufferedReader(new InputStreamReader(getProcess().getInputStream()));
String line;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
To detach I tried some stuff:
Close any of the streams, failed: close method is blocking and waits for the readLine()
Implement another stream to send null / abortion value with SequenceInputStream, failed: when one InputStream was waiting for input, the other was not even called
Use reflections to unlock the read() method inside any of the streams, failed: not sure why, but did not work. Should we go on with this try? Here is the sourcecode:
try {
Field modifiers = Field.class.getDeclaredField("modifiers");
modifiers.setAccessible(true);
Field fdecoder = stream.getClass().getDeclaredField("sd");
fdecoder.setAccessible(true);
modifiers.setInt(fdecoder, 1);
StreamDecoder decoder = (StreamDecoder) fdecoder.get(stream);
Field flock = decoder.getClass().getSuperclass().getDeclaredField("lock");
flock.setAccessible(true);
modifiers.setInt(flock, 1);
Object lock = (Object) flock.get(decoder);
synchronized (lock) {
lock.notifyAll();
}
} catch (NoSuchFieldException | IllegalAccessException e) {
Wrapper.handleException(Thread.currentThread(), e);
}
Not sure how I can fix this. Could you help me interrupting the readLine() method without closing the stream, simple and performant? Thanks.
Edit:
What do I mean by "performant"? My application has not much users, but a lot of processes. The answer by #EJP is not wrong - but unperformant in the case of my application. I cannot have hundreds of threads for hundreds of processes, but I can have as many processes as I have users watching. That's why I try to interrupt the process gracefully. Fewer threads, less running/blocked threads.
Here is the application described (https://imgur.com/VUcYUfi.png)
The Thread that sends the information to the user is the same that reads the input.
I didn't expect it to work, but futures are actually cancelable (but why?).
After #Tarun Lalwani mentioned the TimeLimiter of Googles Guava library, I inspected the code, tried it in my examples (worked!) and rewrote it a bit - make it not time-based, but method-call-based?!
Here is what I got from my research: A wrapper for the BufferedReader:
public class CancelableReader extends BufferedReader {
private final ExecutorService executor;
private Future future;
public CancelableReader(Reader in) {
super(in);
executor = Executors.newSingleThreadExecutor();
}
#Override
public String readLine() {
future = executor.submit(super::readLine);
try {
return (String) future.get();
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
} catch (CancellationException e) {
return null;
}
return null;
}
public void cancelRead() {
future.cancel(true);
}
}
This class allows you to use the BufferedReader#readLine() when you need it and cancel it when you want to continue / interrupt the Thread it is running in. Here is some example code of it in action:
public static void main(String[] args) {
System.out.println("START");
CancelableReader reader = new CancelableReader(new InputStreamReader(System.in));
String line;
new Thread(() -> {
try {
Thread.sleep(10000);
reader.cancelRead();
} catch (InterruptedException e) {
e.printStackTrace();
}
}).start();
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
System.out.println("END");
}
And the output of it:
START
> Hello World!
Hello World!
> What's up?
What's up?
END //Exactly after 5 seconds, when the cancel was called
> Hey, you still there?
//No output as expected
And the last thing I wanna say is why this and not closing InputStream or create one Thread per process?
In this case the InputStream is the stream of a Process, which means we cannot close it. One way would be to unblock readLine() and return null to finish the while-loop, but this is made with Reflection, which is not as beautiful as our solution now and didn't work for any reason. The application uses many processes but has a limited amount of users - thats why we decide for the amount of threads per user and not per process.
I hope you guys will find this Thread in the future and it is helpful for you. Would be awesome if you leave an upvote, so I can get back my rep of the bounty.
Dont forget to upvote the comments either! They helped me alot and brought me to the right solution:
Interrupt BufferedReader#readLine() without closing InputStream
You're going at this back to front.
You can't stop collecting the process's output, or you will stall the child process.
You want to stop displaying the output when the user doesn't want to see it. Look on it as a user interface issue only.
I am using Java's ProcessBuilder to start a subprocess, which is another Java program that has to run in a separate JVM.
I start two Threads to read from the stdout and stderr streams from the Process, so that there is no hang if the stream buffers are full.
The call to Process.waitFor returns but the streams aren't terminated.
The code I am using looks something like (command is a list of strings):
ProcessBuilder pb = new ProcessBuilder(command);
final Process p = pb.start();
final ByteArrayOutputStream outStream = new ByteArrayOutputStream();
final ByteArrayOutputStream errStream = new ByteArrayOutputStream();
Thread outputThread = new Thread() {
#Override
public void run() {
try {
IOUtils.copy(p.getInputStream(), outStream);
} catch (IOException e) {
e.printStackTrace();
}
};
};
outputThread.start();
Thread errorThread = new Thread() {
#Override
public void run() {
try {
IOUtils.copy(p.getErrorStream(), errStream);
} catch (IOException e) {
e.printStackTrace();
}
};
};
errorThread.start();
int returncode = p.waitFor();
outputThread.join();
errorThread.join();
If I run something else, such as "java -version" or "dir" or something, the code works fine.
I have access to the Java code that I am trying to run, but I have never heard that you should call close() on System.out.
Apache commons exec does all this for you. A lot more easier to do...
http://commons.apache.org/exec/
As I know from this website you must close all std-streams from Process object on your own. Regardless if used before or not. This seems highly related to G1 garbage collector (default since Java 7 afaik) which keeps pipes opened when not closed explicitly - even when the subprocess terminated.
I am not familiar with the internals here but the code I posted in my answer to my question works fine on a 24/7 system calling GhostScript and others a lot.
Other questions and answers on SO stating that you must close Process' std-streams explicit:
Process Builder waitFor() issue and Open file limitations
i ve written a simple program to run on a mac, the program opens an excel file and waits for the the user to close the file after which a simple output is given. when i run the program, excel opens, the proc.waitfor is ignored and it just skips to the output without waiting, any help
thanks
Thread myThread = new Thread() {
#Override
public void run() {
try {
String userDir = (System.getProperty("user.home"));
String fileName = userDir + "/Desktop/test/testfile.xlsx";
File theFile = new File(fileName);
Process proc = new ProcessBuilder("/usr/bin/open", fileName).start();
int waitFor = proc.waitFor();
} catch (InterruptedException ex) {
Logger.getLogger(MacTester.class.getName()).log(Level.SEVERE, null, ex);
} catch (IOException ex) {
Logger.getLogger(MacTester.class.getName()).log(Level.SEVERE, null, ex);
}
}
};
myThread.start();
System.out.println("excel is now closed");
This line:
System.out.println("excel is now closed");
Should be inside the run method. Your main thread, the thread that is starting your other thread continues with the execution after start has been invoked.
Another alternative is to place:
myThread.join();
on the line after:
myThread.start();
/usr/bin/open doesn't run modally, it returns the control after launching the appropriate application. You should use open -W. Consider using open -W -n which opens the file in a new instance of the application. Consult man open and try in terminal before testing your java code.
You're doing your process (appropriately) in a background thread, and so what effect will waitfor have on the completely separate calling thread? Answer: none. The solution I see has been given in the other answer -- +1 to him. :)
I am new to java. I am tasked to write java program to run the command lines. I tested the command line under the DOS prompt since i do not have have access to Linux box yet. it worked fine. See the PROGRAM below for full command line syntax. the job will take 6 input files and generate some output files. Next i tried to create a class to and using getruntime and process to process this job. Even it compiled without error but when i run it just show the cursor blinking... i thought i need to use Thread async technique. please provide some advices since i do not have enough time for the projects. I also would like to implement a call back or return values when the job is done. an example would be greatly appreciated. Thanks
import java.io.*;
public class RunJob {
// public static final String PROGRAM = "c:\\wrk\\java.exe Hello";
//one command line below
public static final String PROGRAM = "c:/java.exe -cp \"wrk/jmp.jar;wrk/colt.jar\" gov.lanl.yadas.reliability.UltimateMissileReliabilityModel 10000 \"wrk/\" x1.dat x2c.dat x3.dat x4.dat x5.dat x6.dat true";
// Set to true to end the loop
static boolean done = false;
public static void main(String argv[]) throws IOException {
BufferedReader is;
String line;
String returnMsg = "Start ";
final Process p = Runtime.getRuntime().exec(PROGRAM);
System.out.println("start");
Thread waiter = new Thread() {
public void run() {
try {
p.waitFor();
} catch (InterruptedException ex) {
System.out.println("InterruptedException");
return;
}
System.out.println("Program terminated!");
done = true;
}
};
waiter.start();
is = new BufferedReader(new InputStreamReader(p.getInputStream()));
while (!done && ((line = is.readLine()) != null))
{
System.out.println(line);
returnMsg = returnMsg + line;
}
System.out.println(returnMsg);
System.out.println("End");
return;
}// main
}
I assume that there is a good reason why you want to run a java program from another java program and not just from a shell script, or by invoking an API - but if not - please reconsider.
As to your problem - if your application produces a lot of output (the one you are running as a process) - your application will hang. The p.waitFor() will halt until the process ends. But if you don't read the information from the InputStream - it will overflow and hang!
Advice #1: put the p.waitFor() at the end.
Advice #2: read this article. If I remember correctly it is the one I read when I had a similar problem. You can also google for "StreamGobbler" - it is a common name for a separate thread that "gobbles" your streams.
Advice #3: Don't forget the ErrorStream - if your application will produce too many errors - that stream will cause the process to hang as well.
I want to launch a process from Java, read its output, and get its return code. But while it's executing, I want to be able to cancel it. I start out by launching the process:
ProcessBuilder pb = new ProcessBuilder(args);
pb.redirectErrorStream(true);
Process proc = pb.start();
If I call proc.waitFor(), I can't do anything until the process exits. So I'm assuming I need to something like this:
while (true) {
see if process has exited
capture some output from the process
decide if I want to cancel it, and if so, cancel it
sleep for a while
}
Is this right? Can someone give me an example of how to do this in Java?
Here's an example of what I think you want to do:
ProcessBuilder pb = new ProcessBuilder(args);
pb.redirectErrorStream(true);
Process proc = pb.start();
InputStream is = proc.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line;
int exit = -1;
while ((line = br.readLine()) != null) {
// Outputs your process execution
System.out.println(line);
try {
exit = proc.exitValue();
if (exit == 0) {
// Process finished
}
} catch (IllegalThreadStateException t) {
// The process has not yet finished.
// Should we stop it?
if (processMustStop())
// processMustStop can return true
// after time out, for example.
proc.destroy();
}
}
You can improve it :-) I don't have a real environment to test it now, but you can find some more information here.
I recommend checking out Apache Commons Exec to avoid recreating the wheel. It has some nice features like choosing between synchronous vs. asynchronous execution, as well as a standard solution to spawning a watchdog process that can help in timing out the execution in case it gets stuck.
A helper class like this would do the trick:
public class ProcessWatcher implements Runnable {
private Process p;
private volatile boolean finished = false;
public ProcessWatcher(Process p) {
this.p = p;
new Thread(this).start();
}
public boolean isFinished() {
return finished;
}
public void run() {
try {
p.waitFor();
} catch (Exception e) {}
finished = true;
}
}
You would then implement your loop exactly as you describe:
Process p = Runtime.getRuntime().exec("whatever command");
ProcessWatcher pw = new ProcessWatcher(p);
InputStream output = p.getInputStream();
while(!pw.isFinished()) {
processOutput(output);
if(shouldCancel()) p.destroy();
Thread.sleep(500);
}
Depending upon what conditions would make you want to destroy the process, you might want to do that in a separate thread. Otherwise, you may block while waiting for more program output to process and never really get the option to destroy it.
EDIT: McDowell is 100% right in his comment below, so I've made the finished variable volatile.
How about this (see how it works in jcabi-heroku-maven-plugin):
/**
* Wait for the process to stop, logging its output in parallel.
* #param process The process to wait for
* #return Stdout produced by the process
* #throws InterruptedException If interrupted in between
*/
private String waitFor(final Process process) throws InterruptedException {
final BufferedReader reader = new BufferedReader(
new InputStreamReader(process.getInputStream())
);
final CountDownLatch done = new CountDownLatch(1);
final StringBuffer stdout = new StringBuffer();
new Thread(
new VerboseRunnable(
new Callable<Void>() {
#Override
public Void call() throws Exception {
while (true) {
final String line = reader.readLine();
if (line == null) {
break;
}
System.out.println(">> " + line);
stdout.append(line);
}
done.countDown();
return null;
}
},
false
)
).start();
try {
process.waitFor();
} finally {
done.await();
IOUtils.closeQuietly(reader);
}
return stdout.toString();
}
ps. Now this implementation is available as com.jcabi.log.VerboseProcess class from jcabi-log artifact.
What would make you decide to kill the process -- an asynchronous event (such as input from the user), or a synchronous event (e.g., the process has done what you wanted it to do)? I'm guessing it's the former -- input from the user makes you decide to cancel the subprocess.
Also, how much output do you expect the subprocess to produce? If it's a lot, then the subprocess may block if you don't read from its output stream quickly enough.
Your situation may vary, but it seems that you're likely going to need at least two different threads -- one to decide whether to cancel the process, and one that handles the output from the subprocess.
Have a look here for a bit more detail: http://java.sun.com/developer/JDCTechTips/2005/tt0727.html#2