I am trying to understand countDownLatch and I have this program but I do not know why my program is not returning and not finishing.
package countDownLatches;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
class Processor implements Runnable {
CountDownLatch latch;
public Processor(CountDownLatch latch) {
this.latch = latch;
}
public void run() {
System.out.println("thread started: ");
try {
Thread.sleep(3000);
} catch (InterruptedException e) {
e.printStackTrace();
}
latch.countDown();
}
}
public class App {
public static void main(String args[]) {
CountDownLatch latch = new CountDownLatch(3);
ExecutorService executorService = Executors.newFixedThreadPool(3);
for (int i = 0; i < 3; i++) {
executorService.submit(new Processor(latch));
}
try {
latch.await();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println("task completed");
}
}
You need to shut down the executor service. Add this line after the for loop:
executorService.shutdown();
An alternative method that waits for all actively executing tasks to terminate is
executorService.awaitTermination();
You can read more in the ExecutorService Javadoc.
You need to shutdown your executor service properly. Or it will wait for new tasks indefinitely.
Add:
executorService.shutdown();
Before:
System.out.println("task completed");
I think, that the main reason, why it was done like that in java api, is that executorService may receive tasks from multiple threads (other than main), so - why should it stop, when there is no more actions in the main thread? Yes, I believe, it should not.
Related
This question already has answers here:
ThreadPoolExecutor Block When its Queue Is Full?
(10 answers)
Closed 3 months ago.
We have a large text file in which each line requires intensive process. The design is to have a class that reads the file and delegates the processing of each line to a thread, via thread pool. The file reader class should be blocked from reading the next line once there is no free thread in the pool to do the processing. So i need a blocking thread pool
In the current implementation ThreadPoolExecutor.submit() and ThreadPoolExecutor.execute() methods throw RejectedExecutionException exception after the configured # of threads get busy as i showed in code snippet below.
public class BlockingTp {
public static void main(String[] args) {
BlockingQueue blockingQueue = new ArrayBlockingQueue(3);
ThreadPoolExecutor executorService=
new ThreadPoolExecutor(1, 3, 30, TimeUnit.SECONDS, blockingQueue);
int Jobs = 10;
System.out.println("Starting application with " + Jobs + " jobs");
for (int i = 1; i <= Jobs; i++)
try {
executorService.submit(new WorkerThread(i));
System.out.println("job added " + (i));
} catch (RejectedExecutionException e) {
System.err.println("RejectedExecutionException");
}
}
}
class WorkerThread implements Runnable {
int job;
public WorkerThread(int job) {
this.job = job;
}
public void run() {
try {
Thread.sleep(1000);
} catch (Exception excep) {
}
}
}
Output of above program is
Starting application to add 10 jobs
Added job #1
Added job #2
Added job #3
Added job #4
Added job #5
Added job #6
RejectedExecutionException
RejectedExecutionException
RejectedExecutionException
RejectedExecutionException
Can some one throw some light i.e how i can implement blocking thread pool.
Can some one throw some light i.e how i can implement blocking thread pool.
You need to set a rejection execution handler on your executor service. When the thread goes to put the job into the executor, it will block until there is space in the blocking queue.
BlockingQueue arrayBlockingQueue = new ArrayBlockingQueue(3);
ThreadPoolExecutor executorService =
new ThreadPoolExecutor(1, 3, 30, TimeUnit.SECONDS, arrayBlockingQueue);
// when the blocking queue is full, this tries to put into the queue which blocks
executorService.setRejectedExecutionHandler(new RejectedExecutionHandler() {
#Override
public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {
try {
// block until there's room
executor.getQueue().put(r);
// check afterwards and throw if pool shutdown
if (executor.isShutdown()) {
throw new RejectedExecutionException(
"Task " + r + " rejected from " + executor);
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new RejectedExecutionException("Producer interrupted", e);
}
}
});
So instead of the TRE throwing a RejectedExecutionException, it will call the rejection handler which will in turn try to put the job back on the queue. This blocks the caller.
Lets have a look at your code again:
for (int i = 1; i <= Jobs; i++)
try {
tpExe.submit(new WorkerThread(i));
System.out.println("job added " + (i));
} catch (RejectedExecutionException e) {
System.err.println("RejectedExecutionException");
}
So - when you try to submit, and the pool is busy, that exception is thrown. If you want to wrap around that, it could look like:
public void yourSubmit(Runnable whatever) {
boolean submitted = false;
while (! submitted ) {
try {
tpExe.submit(new WorkerThread(whatever));
submitted = true;
} catch (RejectedExecutionException re) {
// all threads busy ... so wait some time
Thread.sleep(1000);
}
In other words: use that exception as "marker" that submits are currently not possible.
You can use semaphore for to control the resource.Reader will read and create asynchronous task by acquiring semaphore.If every thread is busy the reader thread will wait till thread is available.
public class MyExecutor {
private final Executor exec;
private final Semaphore semaphore;
public BoundedExecutor(Executor exec, int bound) {
this.exec = exec;
this.semaphore = new Semaphore(bound);
}
public void submitTask(final Runnable command)
throws InterruptedException, RejectedExecutionException {
semaphore.acquire();
try {
exec.execute(new Runnable() {
public void run() {
try {
command.run();
} finally {
semaphore.release();
}
}
});
} catch (RejectedExecutionException e) {
semaphore.release();
throw e;
}
}
}
Here is a RejectedExecutionHandler that supports the desired behavior. Unlike other implementations, it does not interact with the queue directly so it should be compatible with all Executor implementations and will not deadlock.
import java.util.concurrent.Executor;
import java.util.concurrent.RejectedExecutionException;
import java.util.concurrent.RejectedExecutionHandler;
import java.util.concurrent.ThreadLocalRandom;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.function.BiFunction;
import static com.github.cowwoc.requirements.DefaultRequirements.assertThat;
import static com.github.cowwoc.requirements.DefaultRequirements.requireThat;
/**
* Applies a different rejection policy depending on the thread that requested execution.
*/
public final class ThreadDependantRejectionHandler implements RejectedExecutionHandler
{
private final ThreadLocal<Integer> numberOfRejections = ThreadLocal.withInitial(() -> 0);
private final BiFunction<Thread, Executor, Action> threadToAction;
/**
* #param threadToAction indicates what action a thread should take when execution is rejected
* #throws NullPointerException if {#code threadToAction} is null
*/
public ThreadDependantRejectionHandler(BiFunction<Thread, Executor, Action> threadToAction)
{
requireThat(threadToAction, "threadToAction").isNotNull();
this.threadToAction = threadToAction;
}
#SuppressWarnings("BusyWait")
#Override
public void rejectedExecution(Runnable r, ThreadPoolExecutor executor)
{
if (executor.isShutdown())
return;
Thread currentThread = Thread.currentThread();
Action action = threadToAction.apply(currentThread, executor);
if (action == Action.RUN)
{
r.run();
return;
}
if (action == Action.REJECT)
{
throw new RejectedExecutionException("The thread pool queue is full and the current thread is not " +
"allowed to block or run the task");
}
assertThat(action, "action").isEqualTo(Action.BLOCK);
int numberOfRejections = this.numberOfRejections.get();
++numberOfRejections;
this.numberOfRejections.set(numberOfRejections);
if (numberOfRejections > 1)
return;
try
{
ThreadLocalRandom random = ThreadLocalRandom.current();
while (!executor.isShutdown())
{
try
{
Thread.sleep(random.nextInt(10, 1001));
}
catch (InterruptedException e)
{
throw new WrappingException(e);
}
executor.submit(r);
numberOfRejections = this.numberOfRejections.get();
if (numberOfRejections == 1)
{
// Task was accepted, or executor has shut down
return;
}
// Task was rejected, reset the counter and try again.
numberOfRejections = 1;
this.numberOfRejections.set(numberOfRejections);
}
throw new RejectedExecutionException("Task " + r + " rejected from " + executor + " because " +
"the executor has been shut down");
}
finally
{
this.numberOfRejections.set(0);
}
}
public enum Action
{
/**
* The thread should run the task directly instead of waiting for the executor.
*/
RUN,
/**
* The thread should block until the executor is ready to run the task.
*/
BLOCK,
/**
* The thread should reject execution of the task.
*/
REJECT
}
}
This works for me.
class handler implements RejectedExecutionHandler{
#Override
public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {
try {
executor.getQueue().put(r);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
This question already has answers here:
ThreadPoolExecutor Block When its Queue Is Full?
(10 answers)
Closed 3 months ago.
We have a large text file in which each line requires intensive process. The design is to have a class that reads the file and delegates the processing of each line to a thread, via thread pool. The file reader class should be blocked from reading the next line once there is no free thread in the pool to do the processing. So i need a blocking thread pool
In the current implementation ThreadPoolExecutor.submit() and ThreadPoolExecutor.execute() methods throw RejectedExecutionException exception after the configured # of threads get busy as i showed in code snippet below.
public class BlockingTp {
public static void main(String[] args) {
BlockingQueue blockingQueue = new ArrayBlockingQueue(3);
ThreadPoolExecutor executorService=
new ThreadPoolExecutor(1, 3, 30, TimeUnit.SECONDS, blockingQueue);
int Jobs = 10;
System.out.println("Starting application with " + Jobs + " jobs");
for (int i = 1; i <= Jobs; i++)
try {
executorService.submit(new WorkerThread(i));
System.out.println("job added " + (i));
} catch (RejectedExecutionException e) {
System.err.println("RejectedExecutionException");
}
}
}
class WorkerThread implements Runnable {
int job;
public WorkerThread(int job) {
this.job = job;
}
public void run() {
try {
Thread.sleep(1000);
} catch (Exception excep) {
}
}
}
Output of above program is
Starting application to add 10 jobs
Added job #1
Added job #2
Added job #3
Added job #4
Added job #5
Added job #6
RejectedExecutionException
RejectedExecutionException
RejectedExecutionException
RejectedExecutionException
Can some one throw some light i.e how i can implement blocking thread pool.
Can some one throw some light i.e how i can implement blocking thread pool.
You need to set a rejection execution handler on your executor service. When the thread goes to put the job into the executor, it will block until there is space in the blocking queue.
BlockingQueue arrayBlockingQueue = new ArrayBlockingQueue(3);
ThreadPoolExecutor executorService =
new ThreadPoolExecutor(1, 3, 30, TimeUnit.SECONDS, arrayBlockingQueue);
// when the blocking queue is full, this tries to put into the queue which blocks
executorService.setRejectedExecutionHandler(new RejectedExecutionHandler() {
#Override
public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {
try {
// block until there's room
executor.getQueue().put(r);
// check afterwards and throw if pool shutdown
if (executor.isShutdown()) {
throw new RejectedExecutionException(
"Task " + r + " rejected from " + executor);
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new RejectedExecutionException("Producer interrupted", e);
}
}
});
So instead of the TRE throwing a RejectedExecutionException, it will call the rejection handler which will in turn try to put the job back on the queue. This blocks the caller.
Lets have a look at your code again:
for (int i = 1; i <= Jobs; i++)
try {
tpExe.submit(new WorkerThread(i));
System.out.println("job added " + (i));
} catch (RejectedExecutionException e) {
System.err.println("RejectedExecutionException");
}
So - when you try to submit, and the pool is busy, that exception is thrown. If you want to wrap around that, it could look like:
public void yourSubmit(Runnable whatever) {
boolean submitted = false;
while (! submitted ) {
try {
tpExe.submit(new WorkerThread(whatever));
submitted = true;
} catch (RejectedExecutionException re) {
// all threads busy ... so wait some time
Thread.sleep(1000);
}
In other words: use that exception as "marker" that submits are currently not possible.
You can use semaphore for to control the resource.Reader will read and create asynchronous task by acquiring semaphore.If every thread is busy the reader thread will wait till thread is available.
public class MyExecutor {
private final Executor exec;
private final Semaphore semaphore;
public BoundedExecutor(Executor exec, int bound) {
this.exec = exec;
this.semaphore = new Semaphore(bound);
}
public void submitTask(final Runnable command)
throws InterruptedException, RejectedExecutionException {
semaphore.acquire();
try {
exec.execute(new Runnable() {
public void run() {
try {
command.run();
} finally {
semaphore.release();
}
}
});
} catch (RejectedExecutionException e) {
semaphore.release();
throw e;
}
}
}
Here is a RejectedExecutionHandler that supports the desired behavior. Unlike other implementations, it does not interact with the queue directly so it should be compatible with all Executor implementations and will not deadlock.
import java.util.concurrent.Executor;
import java.util.concurrent.RejectedExecutionException;
import java.util.concurrent.RejectedExecutionHandler;
import java.util.concurrent.ThreadLocalRandom;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.function.BiFunction;
import static com.github.cowwoc.requirements.DefaultRequirements.assertThat;
import static com.github.cowwoc.requirements.DefaultRequirements.requireThat;
/**
* Applies a different rejection policy depending on the thread that requested execution.
*/
public final class ThreadDependantRejectionHandler implements RejectedExecutionHandler
{
private final ThreadLocal<Integer> numberOfRejections = ThreadLocal.withInitial(() -> 0);
private final BiFunction<Thread, Executor, Action> threadToAction;
/**
* #param threadToAction indicates what action a thread should take when execution is rejected
* #throws NullPointerException if {#code threadToAction} is null
*/
public ThreadDependantRejectionHandler(BiFunction<Thread, Executor, Action> threadToAction)
{
requireThat(threadToAction, "threadToAction").isNotNull();
this.threadToAction = threadToAction;
}
#SuppressWarnings("BusyWait")
#Override
public void rejectedExecution(Runnable r, ThreadPoolExecutor executor)
{
if (executor.isShutdown())
return;
Thread currentThread = Thread.currentThread();
Action action = threadToAction.apply(currentThread, executor);
if (action == Action.RUN)
{
r.run();
return;
}
if (action == Action.REJECT)
{
throw new RejectedExecutionException("The thread pool queue is full and the current thread is not " +
"allowed to block or run the task");
}
assertThat(action, "action").isEqualTo(Action.BLOCK);
int numberOfRejections = this.numberOfRejections.get();
++numberOfRejections;
this.numberOfRejections.set(numberOfRejections);
if (numberOfRejections > 1)
return;
try
{
ThreadLocalRandom random = ThreadLocalRandom.current();
while (!executor.isShutdown())
{
try
{
Thread.sleep(random.nextInt(10, 1001));
}
catch (InterruptedException e)
{
throw new WrappingException(e);
}
executor.submit(r);
numberOfRejections = this.numberOfRejections.get();
if (numberOfRejections == 1)
{
// Task was accepted, or executor has shut down
return;
}
// Task was rejected, reset the counter and try again.
numberOfRejections = 1;
this.numberOfRejections.set(numberOfRejections);
}
throw new RejectedExecutionException("Task " + r + " rejected from " + executor + " because " +
"the executor has been shut down");
}
finally
{
this.numberOfRejections.set(0);
}
}
public enum Action
{
/**
* The thread should run the task directly instead of waiting for the executor.
*/
RUN,
/**
* The thread should block until the executor is ready to run the task.
*/
BLOCK,
/**
* The thread should reject execution of the task.
*/
REJECT
}
}
This works for me.
class handler implements RejectedExecutionHandler{
#Override
public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {
try {
executor.getQueue().put(r);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
i have to finish an exercise where i have to go find ".java" files in my folder path using the producer/consumer pattern with at least one producer thread and x consumer threads.
ProducerConsumer-class:
First i tried to stop the consumer when the producer is finished finding files with setting a while loop from true to false which doesn't work. It doesn't work because the threads are still running obviously just not doing anything useful. Now i use a closePool() function (as well).
So the function does work if i dont put up with my locks called locka. And thats basically something i don't understand.
So if i have
loka.lock();
ende = false;
loka.unlock();
and
while(ende){
loka.lock();
System.out.println(xy.getQueue());
loka.unlock();
}
the closePool() function will never get called. And this is something i don't understand. If i put away the locks in the while loop it does work and the threads do stop.
questions:
1) The ende parameter will be set false anyway so the lock will be finally released.
2) Secondly i did only lock a part of the method and not the object?! As far as i understand it other code in other methods in the same object will still work at the same time. Or is the lock like synchronized and i synchronize the whole object while it is in the lock state?
In my understanding the while loop in the consumer-thread is locked but the producer-thread will still call closePool();
on a extra note: maybe i didn't even design my Producer/Consumer pattern the right way.
import java.io.IOException;
import java.nio.file.FileVisitResult;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.SimpleFileVisitor;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
public class FindJavaVisitorp extends SimpleFileVisitor<Path> {
private BlockingQueue<String> xxx = new ArrayBlockingQueue<String>(10);
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) {
if (file.toString().endsWith(".java")) {
try {
xxx.put(file.toString());
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
return FileVisitResult.CONTINUE;
}
public String getQueue() throws InterruptedException {
return xxx.take();
}
}
public class ProducerConsumer {
private volatile boolean ende = true;
private Path path;
private FindJavaVisitorp xy;
private Lock loka = new ReentrantLock();
private ExecutorService pepe;
public ProducerConsumer(Path path, FindJavaVisitorp xy, ExecutorService xyz) {
this.path = path;
this.xy = xy;
pepe = xyz;
}
public void produce() throws IOException, InterruptedException {
Files.walkFileTree(path, xy);
loka.lock();
ende = false;
loka.unlock();
closePool();
}
public void consume() throws InterruptedException {
while (ende) {
loka.lock();
System.out.println(xy.getQueue());
loka.unlock();
}
}
public void closePool() {
pepe.shutdown();
try {
if (!pepe.awaitTermination(60, TimeUnit.SECONDS)) {
pepe.shutdownNow();
if (!pepe.awaitTermination(60, TimeUnit.SECONDS)) {
System.err.println("Pool couldn't be terminated!");
}
}
} catch (InterruptedException e) {
pepe.shutdownNow();
}
}
}
public class Test {
public static void main(String[] args) {
Path startingDir = Paths.get("/usr/local/");
FindJavaVisitorp x = new FindJavaVisitorp();
ExecutorService exec = Executors.newCachedThreadPool();
final ProducerConsumer pp = new ProducerConsumer(startingDir, x, exec);
exec.submit(new Runnable() {
public void run() {
try {
pp.produce();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
});
// x.printQueue();
for (int j = 0; j < 5; j++) {
exec.submit(new Runnable() {
public void run() {
try {
pp.consume();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
});
}
exec.shutdown();
}
}
Yes, but only if thread goes to check it and not waiting for something else, which in your case is happening. In the while loop threads are waiting for BlockingQueue and if queue is empty then then will not get a chance to check the flag variable. Also you don't need locks as you're already using BlockingQueue. In your example there's no relation between the two critical sections.
.
Following code will try to guard loka=false from any concurrent access.
loka.lock();
ende = false;//critical section
loka.unlock();
Following code will be free from concurrent access and mutually exclusive from the above critical section.
while(ende){
loka.lock();
System.out.println(xy.getQueue());//critical section
loka.unlock();
}
As there's nothing common between these two critical sections, mutual exclusion is doing nothing. Since ende is volatile guarding it with locks doesn't do anything as primitive types already have atomic access.
Reads and writes are atomic for reference variables and for most primitive variables (all types except long and double).
Reads and writes are atomic for all variables declared volatile (including long and double variables).
Only code inside the guarded block by lock() andunlock()` will be locked from concurrent access. Object itself is free to do any concurrent simultaneous (to the locked block) task outside these blocks.
And finally follow proper naming conventions and give your variables meaningful names.
Main answer to your problem why your threads are still running is because they're waiting on the blockingQueue.takeItem() and they can not be released from it unless queue is filled again, however since Producer is finished there's no possibility of that happening.
How to avoid this behavior
There are no methods on BlockingQueue which allow immediate release of waiting threads
One thing we can do is make producer put a LAST_ITEM and have consumers check if the item they got is LAST_ITEM and thus they can release themselves.
Following is working code. I have made some modifications to the variable and method names to make them more meaningful.
JavaFileVisitor
package filevisitor;
import java.nio.file.FileVisitResult;
import java.nio.file.Path;
import java.nio.file.SimpleFileVisitor;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
public class JavaFileVisitor extends SimpleFileVisitor<Path> {
private BlockingQueue<String> blockingQueue = new ArrayBlockingQueue<String>(10);
public static String NO_MORE_ITEMS = "### NO MORE ITEMS ###";
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) {
if (file.toString().endsWith(".java")) {
try {
blockingQueue.put(file.toString());
} catch (InterruptedException e) {
e.printStackTrace();
}
}
return FileVisitResult.CONTINUE;
}
public String getQueueItem() throws InterruptedException {
String item = blockingQueue.take();
if(NO_MORE_ITEMS.equals(item)) {
setNoMoreItems();
}
return item;
}
public void setNoMoreItems() {
try {
blockingQueue.put(NO_MORE_ITEMS);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
ProducerConsumer
package filevisitor;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
public class ProducerConsumer {
private Path path;
private JavaFileVisitor fileVisitor;
public ProducerConsumer(Path path, JavaFileVisitor visitor) {
this.path = path;
this.fileVisitor = visitor;
}
public void produce() throws IOException, InterruptedException {
Files.walkFileTree(path, fileVisitor);
fileVisitor.setNoMoreItems();
}
public void consume() throws InterruptedException {
while (true) {
String item = fileVisitor.getQueueItem();
if(JavaFileVisitor.NO_MORE_ITEMS.equals(item)) {
break;
}
System.out.println(item);
}
}
}
ProducerConsumerMain
package filevisitor;
import java.io.IOException;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
public class ProducerConsumerMain {
public static void main(String[] args) {
Path startingDir = Paths.get("src/filevisitor");
JavaFileVisitor fileVisitor = new JavaFileVisitor();
ExecutorService executor = Executors.newCachedThreadPool();
final ProducerConsumer producerConsumer = new ProducerConsumer(startingDir, fileVisitor);
executor.submit(new Runnable() {
public void run() {
System.out.println("Producer started");
try {
producerConsumer.produce();
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Producer finished");
}
});
for (int j = 0; j < 5; j++) {
executor.submit(new Runnable() {
public void run() {
String threadName = Thread.currentThread().getName();
System.out.println(threadName + " Consumer Started");
try {
producerConsumer.consume();
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println(threadName + " Consumer finished");
}
});
}
executor.shutdown();
System.out.println("Executor shutdown, waiting for threads to finish");
try {
executor.awaitTermination(60, TimeUnit.SECONDS);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Exiting main");
}
}
Output
Producer started
pool-1-thread-3 Consumer Started
pool-1-thread-2 Consumer Started
Executor shutdown, waiting for threads to finish
pool-1-thread-5 Consumer Started
pool-1-thread-6 Consumer Started
pool-1-thread-4 Consumer Started
src\filevisitor\JavaFileVisitor.java
src\filevisitor\ProducerConsumerMain.java
src\filevisitor\ProducerConsumer.java
pool-1-thread-6 Consumer finished
pool-1-thread-4 Consumer finished
pool-1-thread-3 Consumer finished
pool-1-thread-5 Consumer finished
Producer finished
pool-1-thread-2 Consumer finished
Exiting main
I am using single thread executor for long-running threads like this:
executor = Executors.newSingleThreadExecutor(THREAD_FACTORY);
executor.submit(new LongRunnable());
which checks a flag to be stopped:
private class LongRunnable implements Runnable {
#Override
public void run() {
while (isRunning.get()) {
try {
doSomething();
} catch (InterruptedException e) {
...
}
}
}
}
and whole execution is interrupted that way:
#Override
public void close() throws Exception {
isRunning.set(false);
executor.shutdownNow();
}
Still I can see some threads not gc-ed in profiler (while by logs, runnable they were executing has quit outermost while loop).
Question: does provided working with threads strategy memory-leak-free and thread-leak-free?
I am not able to see any issue with executor or shutDownNow. Probably you are looking at different threads in your profiler.
Try this program which is similar to the one in your question and you can see the thread is no longer there after successful shutdown.
public class ExecutorShutdownTest {
private static ExecutorService executor;
private static AtomicLong executorThreadId = new AtomicLong(0);
public static void main(String[] args) {
// get thread MX bean
ThreadMXBean threadMXBean = ManagementFactory.getThreadMXBean();
// create an executor and start the task
executor = Executors.newSingleThreadExecutor(new TestThreadFactory());
LongRunnable runnable = new LongRunnable();
executor.submit(runnable);
// main thread: keep running for sometime
int count = 5;
while (count-- > 0) {
try {
Thread.sleep(1000);
System.out.println(String.valueOf(threadMXBean.getThreadInfo(executorThreadId.longValue())).replace("\r", "").replace(
"\n", ""));
} catch (InterruptedException e) {
e.printStackTrace();
}
}
// main thread: stop the task
try {
runnable.close();
System.out.println(String.valueOf(threadMXBean.getThreadInfo(executorThreadId.longValue())).replace("\r", "").replace("\n", ""));
} catch (Exception e) {
e.printStackTrace();
}
// main thread: run some more time to verify the executor thread no longer exists
count = 5;
while (count-- > 0) {
try {
Thread.sleep(1000);
System.out.println(String.valueOf(threadMXBean.getThreadInfo(executorThreadId.longValue())).replace("\r", "").replace("\n", ""));
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
private static class LongRunnable implements Runnable {
private volatile boolean isRunning = true;
#Override
public void run() {
while (isRunning) {
System.out.println("Running");
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
//ignore
}
}
System.out.println("Stopped");
}
public void close() throws Exception {
System.out.println("Stopping");
isRunning = false;
executor.shutdownNow();
}
}
private static class TestThreadFactory implements ThreadFactory {
private static final AtomicInteger poolNumber = new AtomicInteger(1);
private final ThreadGroup group;
private final AtomicInteger threadNumber = new AtomicInteger(1);
private final String namePrefix;
TestThreadFactory() {
SecurityManager s = System.getSecurityManager();
group = (s != null) ? s.getThreadGroup() : Thread.currentThread().getThreadGroup();
namePrefix = "pool-" + poolNumber.getAndIncrement() + "-thread-";
}
public Thread newThread(Runnable r) {
Thread t = new Thread(group, r, namePrefix + threadNumber.getAndIncrement(), 0) {
#Override protected void finalize() throws Throwable {
super.finalize();
// probably bad idea but lets see if it gets here
System.out.println("Executor thread removed from JVM");
}
};
if (t.isDaemon())
t.setDaemon(false);
if (t.getPriority() != Thread.NORM_PRIORITY)
t.setPriority(Thread.NORM_PRIORITY);
executorThreadId.set(t.getId());
System.out.println("Executor thread created");
return t;
}
}
}
Here's a sample program using the single-thread Executor that manages to strand a thread so that the JVM can't shut down, but it only manages to do it by not calling shutdownNow:
import java.util.concurrent.*;
public class Exec {
public static void main(String[] args) throws Exception {
ExecutorService executor = Executors.newSingleThreadExecutor();
executor.submit(new MyTask());
Thread.sleep(20000L);
// executor.shutdownNow();
int retryCount = 4;
while (!executor.isTerminated() && retryCount > 0) {
System.out.println("waiting for tasks to terminate");
Thread.sleep(500L);
retryCount -= 1;
}
}
}
class MyTask implements Runnable {
public void run() {
int count = 0;
try {
while (!Thread.currentThread().isInterrupted() && count < 10) {
Thread.sleep(1000L);
count += 1;
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
System.out.println("all done");
}
}
The thread used by the executor has a separate life cycle from the task, this example shows how the task finishes but the thread goes on. Uncommenting the shutdownNow results in the executor's thread terminating. Otherwise the main thread sleeps for a while and exits, leaving the executor's thread hanging out, preventing the JVM from exiting.
My guess is that your close method isn't getting called and your executor never gets shut down. To get more useful answers please add a MVCE so that we can reproduce the problem.
Consider that with interruption there's no need to keep a reference to the Runnable to set the flag. As I read the question the task not finishing is not an issue here, but it would still be better to make the Runnable respond to interruption and lose the flag, just because having less things to keep track of is always an improvement.
I have a requirement threading where I need to initiate a thread which will run continuously doing some DB operations . A second thread will be present which needs to run for every 30 secs. The job of the second thread will be killing the first thread and start a new instance of the first thread.
I tried several ways to achieve this but I am not able to do the same.
public class ThreadMain {
public static void main(String[] args) throws InterruptedException, BrokenBarrierException{
final CyclicBarrier gate = new CyclicBarrier(3);
Thread t1 = new Thread(){
public void run(){
try {
gate.await();
while(true)
{
System.out.println("Thread1");
break;
}
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (BrokenBarrierException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}};
Thread t2 = new Thread(){
public void run(){
try {
gate.await();
while(true)
{
System.out.println("Continiously running thread:-Thread2");
}
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (BrokenBarrierException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}};
t1.start();
t2.start();
This seems to work nicely:
// Thread that runs forever.
volatile static Thread forEverThread = null;
static class ForEver implements Runnable {
#Override
public void run() {
try {
while (true) {
Thread.sleep(1000);
System.out.println("For Ever!");
}
} catch (InterruptedException ex) {
// Just quit if I was interrupted.
}
}
}
// Stop the thread if it is running.
private static void stopForeverThread() throws InterruptedException {
// Skip if non-existent.
if (forEverThread != null) {
// Make sure no-one else is already doing it.
synchronized (forEverThread) {
// Still not null?
if (forEverThread != null) {
// Interrupt it.
forEverThread.interrupt();
// Wait for it to finish.
forEverThread.join();
// Clear it.
forEverThread = null;
}
}
}
}
private static void restartForeverThread() throws InterruptedException {
System.out.println("Restarting...");
// Stop it if it is running.
stopForeverThread();
// Start it again.
forEverThread = new Thread(new ForEver());
forEverThread.start();
System.out.println("Restarted");
}
public static void start() throws InterruptedException {
// Start it all up.
restartForeverThread();
// Timed event to restart it.
Timer restartTimer = new Timer(true);
restartTimer.scheduleAtFixedRate(
new TimerTask() {
#Override
public void run() {
try {
// Restart every few seconds.
restartForeverThread();
} catch (InterruptedException ex) {
// We were interrupted during restart - Log it.
}
}
// Every few seconds.
}, 0, 10 * 1000);
}
public static void main(String args[]) {
try {
// Start it all up.
start();
// Hang around for a while - to see what happens.
Thread.sleep(60 * 1000);
} catch (Throwable t) {
t.printStackTrace(System.err);
}
}
If your database task is interruptible (i.e. it reacts on thread interruption and hence can be cancelled by that), the best strategy is to use an ScheduledExecutorService for both, the database task itself and the restart task that runs periodically.
Note that task and thread are two different things: While a task is a piece of work that should be run, threads are the mechanism to do this in parallel.
static class DatabaseTask implements Runnable {
public void run() {
...
}
}
static class RestartTask implements Runnable {
private final ExecutorService executor;
private volatile Future<Void> future;
public RestartTask(ExecutorService executor) {
this.executor = executor;
}
public void run() {
if (future != null) {
future.cancel(true);
}
future = executor.submit(new DatabaseTask());
}
}
ScheduledExecutorService executor = Executors.newScheduledThreadPool(1);
executor.scheduleAtFixedRate(new RestartTask(executor), 0, 30, TimeUnit.SECONDS);
Note that if your DatabaseTask is NOT sensitive to thread interruption and continues to perform database operations, the number of threads executing database tasks will grow continously - probably not what you want. So make sure, all blocking database operations are either interruptible, or terminate within a reasonable amount of time.