We Keep Coding

Running process from cmd.exe process inside thread executor service does not delete terminated processes

I have a program that creates a lot of information and I take that information and throw it into a thread by way of
final static ExecutorService service = Executors.newFixedThreadPool(10);
and pass it information via:
service.submit(new threadTry(str));
Where threadTry is the thread that takes the information, passes it inside command line arguments to generate a command, executes the command, then the cmd process runs the next process if criteria met.
Then the Executor service takes the list of submitted jobs, runs the 10 threads I've limited it to with the information given to them, creates a process which runs a command line window which can pop up another if a certain criteria is met. The command line window and the additional window that pops up do in fact terminate and the thread containing the command line window does terminate as well. The ExecutorService starts another thread with another bit of information. However, The cmd.exe and the other program are listed in windows 7 resource monitor as terminated. My program generates between 150-250 of these terminated cmd's/second (The handles to the javaw.exe process goes up and up with this). Only after I forcibly stop the main program, all the terminated processes are released.
My main problem is that I have to pass so much information that my program gets to about 56% done, then halts. I assume this is because the operating system cannot keep track of so many processes (even though they are terminated, but still listed). Halting is not caused by main program, it generates all information and terminates within 2 seconds (without doing anything with the info). With the processing of the information, my program takes about 2-3 hours to get to 56%.
public class threadTry extends Thread {
String str="";
public threadTry(String str){
this.str=str;
}
public void run(){
try{
String[] cmd={"...",str,"..."}; //input many arguments into cmd
ProcessBuilder probuilder= new ProcessBuilder( cmd );
Process process = probuilder.start();
if(!process.waitFor(5, TimeUnit.SECONDS)){//if we wait longer than 5 seconds,
//then print this message (this is the standard wait for the process that
//waits until termination)
System.out.println(str+": has exited with thread: "+this.getId());
}
int exitValue = process.exitValue();
if(exitValue!=3){
System.out.println("---------------------------------------------------");
System.out.println("\n\nExit Value is " + exitValue+" For: "+str);
System.out.println("---------------------------------------------------");
}
process.destroyForcibly();//does nothing to help clear the already
//terminated process
}catch(Exception e){
e.printStackTrace();
}
}
}

Why there is no system call being invoked for thread creation, when i am creating multiple threads in java?

The following code creates 100 new java threads and runs them.
class ThreadTest {
public static void main(String[] args) {
for (int i = 0; i < 100; i++) {
final int tNo = i;
new Thread(new Runnable() {
#Override
public void run() {
System.out.println("thread #" + tNo);
}
}).start();
}
}
}
When i run the above code and records the system calls executed by this using strace, i cannot find any system call(maybe clone()) which is creating a new thread.
But when i check the threads for the above process using ps -eLf command then it lists(>100) threads with different thread ids.
How does these threads got created without any system call? And if jvm were creating threads in userspace then these shouldn't be listed by ps -eLf.
The output of strace command
mprotect(0xf95000, 8876032, PROT_READ|PROT_EXEC) = 0
munmap(0xf7762000, 92395) = 0
mmap2(NULL, 331776, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0xfffffffff770f000
mprotect(0xf770f000, 4096, PROT_NONE) = 0
clone(child_stack=0xf775f494, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0xf775fbd8, tls=0xf775fbd8, child_tidptr=0xffdb53d0) = 31692
futex(0xf775fbd8, FUTEX_WAIT, 31692, NULLthread #1
thread #5
thread #4
thread #3
.....
) = 0
exit_group(0) = ?
I have removed the initial system calls required for launching jvm. The only clone system call being shown is the one for creating the main thread.

On my machine running Oracle Java 8u5, I see very similar output from running just strace java ThreadTest. However, when I enable the -f flag to follow child processes, I see dozens of clone() calls being made.
Your invocation of strace is telling it just to pay attention to the process launched directly (the JVM bootstrap thread), and it spawns only the main thread. Then that thread is the one calling clone() to create all your workers.

How does these threads got created without any system call?
There certainly is system call clone involved. Try strace with -f option, ref: man strace
-f Trace child processes as they are created by currently traced processes as a result of the fork(2) system call.
On non-Linux platforms the new process is attached to as soon as its pid is known (through the return value of
fork(2) in the parent process). This means that such children may run uncontrolled for a while (especially in the
case of a vfork(2)), until the parent is scheduled again to complete its (v)fork(2) call. On Linux the child is
traced from its first instruction with no delay. If the parent process decides to wait(2) for a child that is cur-
rently being traced, it is suspended until an appropriate child process either terminates or incurs a signal that
would cause it to terminate (as determined from the childâs current signal disposition).
And if jvm were creating threads in userspace then these shouldn't be
listed by ps -eLf.
The threads created by jvm are user level threads represented by LWP (kernel level abstraction) on linux.
see the lwp (native thread id) & nlwp columns in ps -eLF result.

Elegant way to check whether external long-term jar launched successfully

I am trying to execute external jar from java app.
What is the most elegant way to check if the process has been started successfully and running?
ExtApp.jar is long-term running process, so I can not use Process.waiFor() because it would block my app. I have come up with following code, with idea behind is that the exitValue() throws IllegalThreadStateException if the process has not been yet terminated.
boolean success = false;
try {
Process process = Runtime.getRuntime().exec("java -jar ExtApp.jar");
try {
if (process.exitValue() == 0)
success = true;
} catch (IllegalThreadStateException e) {
success = true;
}
} catch (Exception e) {}
System.out.println(success);
But it is kind of ugly solution. Any ideas for a better one?

There seems to be no elegant solution to the problem. E.g. I ran your code on my PC and got "success" though there is no ExtApp.jar on it. That is, from the point of view of Runtime.exec the process (java.exe) started successfully, no matter what happens afterwards.

The above seems very dubious. You're going to spawn off your process and then test it immediately. The process itself may not have determined whether it's running ok or not (e.g. when does it actually check that jar file eixsts/is loadable/is valid ?)
I think you're better off spawning the process via a new thread, calling/blocking in that thread via Process.waitFor() and then notifying the parent thread (via whatever means - state variable, wait()/notify(), a java.util.concurrent.Future etc.) once the process has exited and you've collected the exit status.
Apache Commons Exec is a useful library for doing this sort of work, including asynchronous spawning/notification of process exit. See the DefaultExecuteResultHandler for more info.

How do you gracefully exit a Process in Java?

I am trying to make a Java program which will run several other unrelated Java programs, specifically a Minecraft server.
Currently, I am trying to work out how to end a java.lang.Process gracefully.
This is the code for my spawner program:
http://dl.dropbox.com/u/26746878/SpawnerSource/Main.java.txt
And this is the code for the program which is spawned:
http://dl.dropbox.com/u/26746878/SpawnerSource/Tester.java.txt
What I do is run my spawner program. Then, after a few seconds, I terminate it with Ctrl-C.
What I want to see is my program output 'Shutting Down' followed by 'Ending'. I also want to see a file 'test.txt'.
What I actually see is only 'Shutting Down', with no 'Ending' nor 'test.txt'
I believe the problem is that Process.destroy() is forcefully ending the process without letting the shutdown hooks run.
Is there an alternative to Process.destroy() which will exit the process gracefully (ie: as if I had pressed Ctrl-C)?

You should never destroy a working process as it might get the whole OS into an unstable state (believe me, this caused us 2 hours downtime and cost 10000$ to my company :( )
What you should do instead is as #Kane mentioned, send a shutdown request to all your child processes and wait until they are all finished (every child process sends an RMI notification back to the main process right before gracefully exiting)
class ParentProcess{
Map<int, CountDownLatch> finishSignals = new ConcurrentHashMap<int, CountDownLatch>();
public void startProcess(){
// Start child process
// get its ID
// and create a count down latch for it
finishSignals.add(processId, new CountDownLatch(1));
}
public void shutDownProcess(processId){
// Send an RMI request to process ID to shutdown
}
// RMI request sent from child process before stopping
public void processFinishedNotification(processId){
finishSignals[processId].countDown()
}
public void waitForChildsToFinish(){
// This for loop will block until all child processes have sent a finish notification
for(CountDownLatch childFinishSignal : finishSignals){
childFinishSignal.await();
}
}
}

You may want to look into Remote Method Invocation, and have your spawner process ask the child processes to shut themselves down instead of having the spawner process kill the child processes itself.

Shutdown spawned java child process gracefully

I use ProcessBuilder to spawn a child process for executing writing some data to file system. And a problem occurs because the parent process may crash/ killed unexpectedly, the child process just hangs even I use e.g. jps checking if its parent process dies then exits. What is the right way for a spawned child process to detects if its parent process is dead and then exit?
Also, After searching on the internet, most solution use Runtime.addShutdownHook(), but this is not provided in ProcessBuilder. Does it have equivalent one?

You did a good search around the problem. So you may want to use the returned reference to the Process instance and the Runtime.addShutdownHook(Thread) method you mentioned. This is the last step that I believe you need to take:
List commands = new ArrayList();
commands.add("xeyes"); // launch this command
ProcessBuilder pb = new ProcessBuilder(commands);
final Process p = pb.start();
Runtime.getRuntime().addShutdownHook(new Thread() {
#Override
public void run() {
p.destroy();
}
});
Thread.sleep(2000); // sleep for some time

In a similar situation I implemented a heartbeat mechanism where the parent process had to regularly send heartbeats. If that did not happen the child process would shut itself down.