In my program I do have a list of n items.
I will be iterating the list and initiate a process like this:
Runtime.getRuntime.exec("cmd /C start abc.bat"+listitem() )
I need to maintain a count of 4 processes. Once any one of the process is completed, I need to start the next process , so the process count should be 4.
I am able to initiate 4 process simultaneously, but not sure on how to keep the count of 4. Basically I need some notification once a process is terminated, so I can start the next, any threading is possible.
Any help on how to implement this, can somebody share a snippet on this above requirement?
Use an ThreadPoolExecutor of size 4 and a Runnable implementation which starts the Process and then invokes Process.waitFor(). Since the thread pool will be restricted to 4 threads and all 4 threads will start a process and then wait for it, you'll be certain there won't be more than 4 child processes running.
Some sample code to help you along your way:
ExecutorService executor = Executors.newFixedThreadPool(4);
executor.execute(new Runnable() {
public void run() {
//use ProcessBuilder here to make the process
Process p = processBuilder.start();
p.waitFor();
}
});
public class ProcessRunner {
public static void main(String[] args) throws IOException, InterruptedException {
//Creating n threads as required
ExecutorService exec = Executors.newFixedThreadPool(4);
for(int i = 0; i < 4; i++){
exec.execute(new ProcessRunnable());
}
Thread.sleep(10000);
//whenever you want them to stop
exec.shutdownNow();
}
}
class ProcessRunnable implements Runnable{
#Override
public void run(){
do{
Process p;
try {
p = Runtime.getRuntime().exec("cd ..");
p.waitFor();
} catch (IOException e) {
//Take appropriate steps
e.printStackTrace();
} catch (InterruptedException e) {
//Take appropriate steps
e.printStackTrace();
}
}while(!Thread.interrupted());
}
}
Process#waitFor()
Causes the current thread to wait, if necessary, until the process
represented by this Process object has terminated. This method returns
immediately if the subprocess has already terminated. If the
subprocess has not yet terminated, the calling thread will be blocked
until the subprocess exits.
You should have four threads that each take an assignment from a pool, then carry it out, then when it is done, carry out the next assignment. This would be how:
class Whatever extends Thread {
public void run() {
while (!interrupted()) {
String str = listitem();
if (str == null) // there are no more commands to run
break;
Runtime.getRuntime.exec(("cmd /C start abc.bat"+str).split("\\s")).waitFor();
}
Then start four of these threads.
You can use fixed thread pool with size 4 which guarantees no more than 4 active threads at any given moment
final ExecutorService ex = Executors.newFixedThreadPool(4);
for(int i = 0; i < 100; i++) {
ex.execute(new Runnable() {
#Override
public void run() {
... run the process here and wait for it to end
}
});
}
Related
The routine myProcessToRun() needs to be executed 100 times but there needs to be about a second delay between each execution.
The following FOR Loop is used in conjunction with the ScheduledThreadPoolExecutor object.
for (int n=0; n<100; n++)
{
final ScheduledThreadPoolExecutor executor = new ScheduledThreadPoolExecutor(1);
executor.schedule(new Runnable() {
#Override
public void run() {
myProcessToRun();
}
}, (n+2), TimeUnit.SECONDS);
}
This actually works fine but the threads still remain.
Using JVisualVM the number of Threads increases by 100 threads when the routine is executed. When the routine finishes, the 100 threads still remain.
Clicking the "Perform GC" button doesn't clean them up so Java still believe they should exist.
How do these threads get cleaned up using an example like above?
---Edited---
I noticed the ScheduledThreadPoolExecutor was being instantiated within the Loop which was a terrible idea. After moving it outside the LOOP the threads created weren't so bad.
After attempting to implement the solution there was unexpected behavior.
final ScheduledThreadPoolExecutor executor = new ScheduledThreadPoolExecutor(10);
for (int n=0; n<100; n++)
{
//final ScheduledThreadPoolExecutor executor = new ScheduledThreadPoolExecutor(2);
executor.schedule(new Runnable() {
#Override
public void run() {
doAddNewCondSet();
}
}, (n+2), TimeUnit.SECONDS);
}
try
{
executor.shutdown();
if (!executor.awaitTermination(400, TimeUnit.SECONDS))
executor.shutdownNow();
} catch (InterruptedException e1)
{
e1.printStackTrace();
}
With the modified code, it would immediate stop all the processes with the shutdown and nothing was executed.
With the executor.shutdown(); commented out and just using the awaitTermination(), the program just hung and after a few minutes, all the processes kicked off at the same time without delay which resulted in errors.
I suspect my implementation was wrong.
There are a number of ways that you can do this. You can view some of them here:
https://www.baeldung.com/java-executor-wait-for-threads
My personal favorite is the CountDownLatch:
Next, let’s look at another approach to solving this problem – using a
CountDownLatch to signal the completion of a task.
We can initialize it with a value that represents the number of times
it can be decremented before all threads, that have called the await()
method, are notified.
For example, if we need the current thread to wait for another N
threads to finish their execution, we can initialize the latch using
N:
ExecutorService WORKER_THREAD_POOL
= Executors.newFixedThreadPool(10);
CountDownLatch latch = new CountDownLatch(2);
for (int i = 0; i < 2; i++) {
WORKER_THREAD_POOL.submit(() -> {
try {
// ...
latch.countDown();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
});
}
// wait for the latch to be decremented by the two remaining threads
latch.await();
This code I have is not executing tasks in parallel,
it only executes the code in this case once (whatever is in the for loop, but it should be 2) :
public class mqDirect {
public static void main(String args[]) throws Exception {
int parallelism = 2;
ExecutorService executorService =
Executors.newFixedThreadPool(parallelism);
Semaphore semaphore = new Semaphore(parallelism);
for (int i = 0; i < 1; i++) {
try {
semaphore.acquire();
// snip ... do stuff..
semaphore.release();
} catch (Throwable throwable) {
semaphore.release();
}
executorService.shutdownNow();
}
}
}
In Java the main way to make code work in parallel is to create a Thread with a new Runnable as a constructor parameter. You then need to start it.
There are many tutorials to help you get this to happen properly.
As your code stands you are merely creating an ExecutorService (and not using it), creating a Semaphore (which should be done in the thread but isn't), performing some process and then shutting down the Executor.
BTW: ShutDownNow is probably not what you want, you should just use ShutDown.
OK, So I found this good tutorial
http://programmingexamples.wikidot.com/threadpoolexecutor
And I have done something like
public class mqDirect {
int poolSize = 2;
int maxPoolSize = 2;
long keepAliveTime = 10;
ThreadPoolExecutor threadPool = null;
final ArrayBlockingQueue<Runnable> queue = new ArrayBlockingQueue<Runnable>(
5);
public mqDirect()
{
threadPool = new ThreadPoolExecutor(poolSize, maxPoolSize,
keepAliveTime, TimeUnit.SECONDS, queue);
}
public void runTask(Runnable task)
{
threadPool.execute(task);
System.out.println("Task count.." + queue.size());
}
public void shutDown()
{
threadPool.shutdown();
}
public static void main (String args[]) throws Exception
{
mqDirect mtpe = new mqDirect();
// start first one
mtpe.runTask(new Runnable()
{
public void run()
{
for (int i = 0; i < 2; i++)
{
try
{
System.out.println("First Task");
runMqTests();
Thread.sleep(1000);
} catch (InterruptedException ie)
{
}
}
}
});
// start second one
/*
* try{ Thread.sleep(500); }catch(InterruptedException
* ie){}
*/
mtpe.runTask(new Runnable()
{
public void run()
{
for (int i = 0; i < 2; i++)
{
try
{
System.out.println("Second Task");
runMqTests();
Thread.sleep(1000);
} catch (InterruptedException ie)
{
}
}
}
});
mtpe.shutDown();
// runMqTests();
}
And it works !
But the problem is , this duplicated code ... runMqtests() is the same task, is there a way to specify it to run in parallel without duplicating the code?
The example I based this off is assuming each task is different.
This code I have is not executing tasks in parallel, it only executes the code in this case once (whatever is in the for loop, but it should be 2) :
Just because you instantiate an ExecutorService instance doesn't mean that things magically run in parallel. You actually need to use that object aside from just shutting it down.
If you want the stuff in the loop to run in the threads in the service then you need to do something like:
int parallelism = 2;
ExecutorService executorService = Executors.newFixedThreadPool(parallelism);
for (int i = 0; i < parallelism; i++) {
executorService.submit(() -> {
// the code you want to be run by the threads in the exector-service
// ...
});
}
// once you have submitted all of the jobs, you can shut it down
executorService.shutdown();
// you might want to call executorService.awaitTermination(...) here
It is important to note that this will run your code in the service but there are no guarantees that it will be run "in parallel". This depends on your number of processors and the race conditions inherent with threads. For example, the first task might start up, run, and finish its code before the 2nd one starts. That's the nature of threaded programs which are by design asynchronous.
If, however, you have at least 2 cores, and the code that you submit to be run by the executor-service takes a long time to run then most likely they will be running at the same time at some point.
Lastly, as #OldCurmudgeon points out, you should call shutdown() on the service which allows current jobs already submitted to the service to run as opposed to shutdownNow() which cancels and queued jobs and also calls thread.interrupt() on any running jobs.
Hope this helps.
This method repeatedly reads commands and execute them until the game is over. The finished variable, when true, means that the player/user has hit Quit and wants to end the game - thus exiting the loop and executing till the end of the method.
import org.apache.commons.lang3.time.StopWatch;
private Parser parser;
private Command command;
private StopWatch stopWatch;
public void play()
{
stopWatch = new StopWatch();
stopWatch.start();
long timeLimit = 5000;
boolean finished = false;
printWelcome();
while (finished == false)
{
System.out.println(stopWatch.getTime() + " milliseconds");
if (stopWatch.getTime() >= timeLimit)
{
System.out.println("Time limit of " + timeLimit + " milli seconds has been reached. Good bye!");
return;
}
else
{
command = parser.getCommand();
finished = processCommand(command);
}
}
System.out.println("Thank you for playing. Good bye.");
}
But I'm observing a strange behaviour with the loop. It loops perfectly fine (displaying the continuous count of stopWatch.getTime() when the following lines are omitted :
else
{
command = parser.getCommand();
finished = processCommand(command);
}
But when I put them back in, it stops displaying the continuous incrementing time of the stopWatch as it increases towards the timeLimit (at which point, it should stop). Even if the player hasn't entered any command or input.
The stopWatch is clearly running in the background, but it doesn't display the count.
Please advise. Thanks.
Even if the player hasn't entered any command or input.
If Parser#getCommand is a method that is waiting for user input, execution will block until the user enters input. This means the while loop won't run and you won't see it printing out the new time.
Doing what you ask would make a very jarring experience for the user anyway, since it would mean that it would keep printing while they are typing in the command, and that's not very user-friendly.
You can use a producer-consumer approach for this. I have adapted the code from the linked question:
// Shared queue
final Queue<String> commands = new ConcurrentLinkedQueue<>();
// Non-blocking consumer
ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor();
executor.scheduleAtFixedRate(new Runnable() {
#Override
public void run() {
// non-blocking
if((String command = commands.poll()) != null) {
processCommand(command);
}
}
}, 0, 10, TimeUnit.MILLISECONDS);
// Blocking producer (this is basically the logic from Parser#getCommand)
Scanner sc = new Scanner(System.in);
while(sc.hasNext()) {
commands.add(sc.next());
}
However, this solution doesn't really let you implement the timeout. So you would have to create a new class that "wraps" the Runnable so that you can cancel it after a timeout (similar to this approach):
public class TimeOutableRunnable implements Runnable {
private final Queue<String> commands;
private final Runnable wrapped;
private volatile ScheduledFuture<?> self;
public TimeOutableRunnable(Runnable wrapped, Queue<String> commands) {
this.commands = commands;
this.wrapped = wrapped;
}
#Override
public void run() {
if(commands.isEmpty()) {
self.cancel(false);
} else {
wrapped.run();
}
}
public void runWithTimeout(ScheduledExecutorService executor, long timeout, TimeUnit unit) {
self = executor.scheduleAtFixedRate(this, 0, timeout, unit);
}
}
Then you could do:
final Queue<String> commands = new ConcurrentLinkedQueue<>();
ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor();
new TimeOutableRunnable(new Runnable() {
#Override
public void run() {
// non-blocking
while((String command = commands.poll()) != null) {
processCommand(command);
}
}
}, commands).runWithTimeout(executor, timeLimit, TimeUnit.MILLISECONDS);
I haven't tested this out and I wouldn't rate myself very high on concurrency so let me know if there is something fundamentally wrong with this approach.
I presume your parser.getCommand(); is blocking until there is input. As this while loop runs in one thread, the program is stopped at this point.
The easiest way to check if I'm right is to enter any command, and you should get some output from this line:
System.out.println(stopWatch.getTime() + " milliseconds");
You either need to implement a timeout in this method, or have a second thread counting time and interrupting the wait.
Perhaps I'm misunderstanding your question, but it sounds like your problem that it isn't printing continuously? If so, its because your parer is waiting for user input. Threading could fix this problem, run your timer and print in a seperate thread
I have hundreds of files to process. I do each file one at a time and it takes 30 minutes.
I'm thinking I can do this processing in 10 simultaneous threads, 10 files at a time, and I might be able to do it in 3 minutes instead of 30.
My question is, what is the "correct" way to manage my 10 threads? And when one is done, create a new one to a max number of 10.
This is what I have so far ... is this the "correct" way to do it?
public class ThreadTest1 {
public static int idCounter = 0;
public class MyThread extends Thread {
private int id;
public MyThread() {
this.id = idCounter++;
}
public void run() {
// this run method represents the long-running file processing
System.out.println("I'm thread '"+this.id+"' and I'm going to sleep for 5 seconds!");
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("I'm thread '"+this.id+"' and I'm done sleeping!");
}
}
public void go() {
int MAX_NUM_THREADS = 10;
List<MyThread> threads = new ArrayList<MyThread>();
// this for loop represents the 200 files that need to be processed
for (int i=0; i<200; i++) {
// if we've reached the max num of threads ...
while (threads.size() == MAX_NUM_THREADS) {
// loop through the threads until we find a dead one and remove it
for (MyThread t : threads) {
if (!t.isAlive()) {
threads.remove(t);
break;
}
}
}
// add new thread
MyThread t = new MyThread();
threads.add(t);
t.start();
}
}
public static void main(String[] args) {
new ThreadTest1().go();
}
}
You can use ExecutorService to manage you threads.
And you can add while loop to thread run method to execute file processing task repeatedly.
Also you can read about BlockingQueue usage. I think it will fit perfectly to allocate new files (tasks) between threads.
I would suggest using Camel's File component if you are open to it. The component will handle all the issues with concurrency to ensure that multiple threads don't try to process the same file. The biggest challenge with making your code multi-threaded is making sure the threads don't interact. Let a framework take care of this for you.
Example:
from("file://incoming?maxMessagesPerPoll=1&idempotent=true&moveFailed=failed&move=processed&readLock=none")
.threads(10).process()
I want to start two process at the same time and make sure complete them all before proceeding other steps. Can you help? I already tried Thread, it can't start two at the same time and wait until been done.
final CyclicBarrier gate = new CyclicBarrier(3);
Thread r2 = new Thread()
{
public void run()
{
try
{
int i = 0;
while (i < 3)
{
System.out.println("Goodbye, " + "cruel world!");
Thread.sleep(2000L);
i++;
gate.await();
}
}
catch (InterruptedException | BrokenBarrierException iex)
{
}
}
};
Thread r3 = new Thread()
{
public void run()
{
try
{
int i = 0;
while (i < 3)
{
System.out.println("Goodbye, " + "cruel world!");
Thread.sleep(2000L);
i++;
gate.await();
}
}
catch (InterruptedException | BrokenBarrierException iex)
{
}
}
};
r2.start();
r3.start();
gate.await();
System.out.println("Donew");
You can use Thread.join()to wait until your subprocesses/threads have finished.
You should not need CyclicBarrier.
Your problem is that you are repeatedly waiting for three parties, but only two threads are calling await() repeatedly. I would expect your code to immediately print, "Goodbye, cruel world!" twice, and "Done", then hang, because the loops are waiting for a third thread to invoke await() again, but the main thread has now terminated.
One solution is for your main thread to loop, invoking await() the same number of times that your task does. But that would be kind of ugly.
I'd suggest using the invokeAll() method of an ExecutorService. This will submit your tasks to the service at (approximately) the same time, then block until all tasks complete. If you want to try to improve the simultaneity of the task commencing, you could add a CyclicBarrier, but it looks like you are more concerned with when the tasks end, and invokeAll() will take care of that for you.
final class Sample
implements Callable<Void>
{
private static final int ITERATIONS = 3;
private static final long AVG_TIME_MS = 2000;
public static void main(String[] args)
throws InterruptedException
{
List<Sample> tasks = Arrays.asList(new Sample(), new Sample());
ExecutorService workers = Executors.newFixedThreadPool(tasks.size());
for (int i = 1; i <= ITERATIONS; ++i) {
/* invokeAll() blocks until all tasks complete. */
List<Future<Void>> results = workers.invokeAll(tasks);
for (Future<?> result : results) {
try {
result.get();
}
catch (ExecutionException ex) {
ex.getCause().printStackTrace();
return;
}
}
System.out.printf("Completed iteration %d.%n", i);
}
workers.shutdown();
System.out.println("Done");
}
#Override
public Void call()
throws InterruptedException
{
/* The average wait time will be AVG_TIME_MS milliseconds. */
ThreadLocalRandom random = ThreadLocalRandom.current();
long wait = (long) (-AVG_TIME_MS * Math.log(1 - random.nextDouble()));
System.out.printf("Goodbye, cruel world! (Waiting %d ms)%n", wait);
Thread.sleep(wait);
return null;
}
}
Notice how I spiced things up with a random wait time. Yet, invokeAll() waits until all of tasks in that iteration complete.
It's impossible for the single processor machines.
Even if you find lot of answers on threads its not gonna start two process at the same time
If you accept the Relative Simultanity that will be easy.