I'm writing a swing application with HttpClient and I need a way to make a download list because I need to wait 1 minute (for example) before starting a new download.
So I would like to create a waiting list of threads (downloads).
I would have a class that takes a time parameter and contains a list of threads and when I add a thread in the list it starts if there is no running thread. Otherwise it waits for its turn.
Is there any tool to do that ?
Thanks a lot for your help.
Yes. ScheduledExecutorService. You can create a fixed length service via Executors.newScheduledThreadPool(corePoolSize). When you are ready to submit the task to wait the amount of time just submit it to ScheduledExecutorService.schedule
ScheduledExecutorService e = Executors.newScheduledThreadPool(10)
private final long defaultWaitTimeInMinutes = 1;
public void submitTaskToWait(Runnable r){
e.schedule(r, defaultWaitTimeInMinutes, TimeUnit.MINUTES);
}
Here the task will launch in 1 minute from the time of being submitted. And to address your last point. If there are currently tasks being downloaded (this configuration means 10 tasks being downloaded) after the 1 minute is up the runnable submitted will have to wait until one of the other downloads are complete.
Keep in mind this deviates a bit from the way you are designing it. For each new task you wouldnt create a new thread, rather you would submit to a service that already has thread(s) waiting. For instance, if you only want one task to download at a time you change from Executors.newScheduledThreadPool(10) to Executors.newScheduledThreadPool(1)
Edit: I'll leave my previous answer but update it with a solution to submit a task to start exactly 1 minute after the previous task completes. You would use two ExecutorServices. One to submit to the scheuled Executor and the other to do the timed executions. Finally the first Executor will wait on the completion and continue with the other tasks queued up.
ExecutorService e = Executors.newSingleThreadExecutor();
ScheduledExecutorService scheduledService = Executors.newScheduledThreadPool(1)
public void submitTask(final Runnable r){
e.submit(new Runnable(){
public void run(){
ScheduledFuture<?> future= scheduledService.schedule(r, defaultWaitTimeInMinutes, TimeUnit.MINUTES);
future.get();
}
});
}
Now when the future.get(); completes the next Runnable submitted through submitTask will be run and then scheduled for a minute. Finally this will work only if you require the task to wait the 1 minute even if there is no other tasks submitted.
I think this would be a wrong way of going about the problem. A bit more logical way would be to create "download job" objects which will be added to a job queue. Create a TimerTask which would query this "queue" every 1 minute, pick up the Runnable/Callable jobs and submit them to the ExecutorService.
You could use the built-in ExecutorService. You can queue up tasks as Runnables and they will run on the available threads. If you want only a single task to run at a time use newFixedThreadPool(1);
ExecutorService executor = Executors.newFixedThreadPool(1);
You could then append an artificial Thread.sleep at the beginning of each Runnable run method to ensure that it waits the necessary amount of time before starting (not the most elegant choice, I know).
The Java Concurrency package contains classes for doing what you ask. The general construct you're talking about is an Executor which is backed by a ThreadPool. You generate a list of Runables and send them to an Executor. The Executor has a ThreadPool behind it which will run the Runnables as the threads become available.
So as an example here, you could have a Runnable like:
private static class Downloader implements Runnable {
private String file;
public Downloader(String file) {
this.file = file;
}
#Override
public void run() {
// Use HttpClient to download file.
}
}
Then You can use it by creating Downloader objects and submitting it to an ExecutorService:
public static void main(String[] args) throws Exception {
ExecutorService executorService = Executors.newFixedThreadPool(5);
for (String file : args) {
executorService.submit(new Downloader(file));
}
executorService.awaitTermination(100, TimeUnit.SECONDS);
}
It is maybe not the best solution but here is what I came up with thanks to the answer of John Vint. I hope it will help someone else.
package tests;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
public class RunnableQueue
{
private long waitTime;
private TimeUnit unit;
ExecutorService e;
public RunnableQueue(long waitTime, TimeUnit unit) {
e = Executors.newSingleThreadExecutor();
this.waitTime = waitTime;
this.unit = unit;
}
public void submitTask(final Runnable r){
e.submit(new Runnable(){
public void run(){
Thread t = new Thread(r);
t.start();
try {
t.join();
Thread.sleep(unit.toMillis(waitTime));
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
}
public static void main(String[] args) {
RunnableQueue runQueue = new RunnableQueue(3, TimeUnit.SECONDS);
for(int i=1; i<11; i++)
{
runQueue.submitTask(new DownloadTask(i));
System.out.println("Submitted task " + i);
}
}
}
Related
I have a recursive bulk task that I put into execution in a ForkJoinPool thread pool.
public class SomeTask extends RecursiveAction {
#Override
protected void compute() {
//Some recursive logic...
}
}
public class Main {
public static void main(String[] args) {
startForkJoinPool();
}
private void startForkJoinPool() {
SomeTask someTask = new SomeTask();
ForkJoinPool pool = new ForkJoinPool(4);
pool.invoke(someTask);
pool.shutdown();
}
}
Now I need to execute this logic in two more parallel threads.
I decided to try to use the ExecutorService thread pool, and when I put entities into it for execution, I found out that they are not executed in parallel, but, as it were, one of the threads is parked for the duration of the first one.
public class SomeTask extends RecursiveAction {
#Override
protected void compute() {
//Some recursive logic...
}
}
public class Main {
public static void main(String[] args) {
List<Thread> threadList = new ArrayList<>();
threadList.add(new Thread(() -> startForkJoinPool()));
threadList.add(new Thread(() -> startForkJoinPool()));
ExecutorService executorService = Executors.newFixedThreadPool(2);
threadList.forEach(executorService::execute);
executorService.shutdown();
}
private void startForkJoinPool() {
SomeTask someTask = new SomeTask();
ForkJoinPool pool = new ForkJoinPool(4);
pool.invoke(someTask);
pool.shutdown();
}
}
Tell me, please, what can I do wrong?
Many thanks in advance to anyone who can point me in the right direction.
It is a bit strange that your code example contains ExecutorService.invoke and ExecutorService.execute where you should be calling ExecutorService.submit.
The ExecutorService documentation also contains a usage example.
And then there is some nice Java Tutorial on the topic.
Don't create any threads, that is the job of the ExecutorService.
You don't want to execute the task, you want to submit the task. Then you get a Future<T> returned.
As you already have a collection, consider invokeAll that returns List<Future<T>> and saves you iterating through your list.
tl;dr
Change execute to submit.
Await termination when shutting down an executor service.
Executor#execute is not necessarily async
You need to carefully read the documentation. This is a tricky area.
The ExecutorService#execute method is inherited from the super interface Executor.
The Javadoc for Executor says:
However, the Executor interface does not strictly require that execution be asynchronous
So any Runnable you pass to execute may or may not be run on a background thread.
ExecutorService#submit is always async
If you definitely want your Runnable to run on a background thread, pass to ExecutorService#submit.
Change this line:
threadList.forEach(executorService::execute);
… to this:
threadList.forEach(executorService::submit);
Shutdown
Your call to ExecutorService#shutdown is insufficient. You need to wait for submitted tasks to complete.
See the boilerplate method shutdownAndAwaitTermination given to you in the Javadoc for ExecutorService.
Or, in Java 19+, call ExecutorService#close.
Good day.
I have blocker issue with my web crawler project.
Logic is simple. First creates one Runnable, it downloads html document, scans all links and then on all funded links it creates new Runnable objects. Each new created Runnable in its turn creates new Runnable objects for each link and execute them.
Problem is that ExecutorService never stops.
CrawlerTest.java
public class CrawlerTest {
public static void main(String[] args) throws InterruptedException {
new CrawlerService().crawlInternetResource("https://jsoup.org/");
}
}
CrawlerService.java
import java.io.IOException;
import java.util.Collections;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
public class CrawlerService {
private Set<String> uniqueUrls = Collections.newSetFromMap(new ConcurrentHashMap<String, Boolean>(10000));
private ExecutorService executorService = Executors.newFixedThreadPool(8);
private String baseDomainUrl;
public void crawlInternetResource(String baseDomainUrl) throws InterruptedException {
this.baseDomainUrl = baseDomainUrl;
System.out.println("Start");
executorService.execute(new Crawler(baseDomainUrl)); //Run first thread and scan main domain page. This thread produce new threads.
executorService.awaitTermination(10, TimeUnit.MINUTES);
System.out.println("End");
}
private class Crawler implements Runnable { // Inner class that encapsulates thread and scan for links
private String urlToCrawl;
public Crawler(String urlToCrawl) {
this.urlToCrawl = urlToCrawl;
}
public void run() {
try {
findAllLinks();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private void findAllLinks() throws InterruptedException {
/*Try to add new url in collection, if url is unique adds it to collection,
* scan document and start new thread for finded links*/
if (uniqueUrls.add(urlToCrawl)) {
System.out.println(urlToCrawl);
Document htmlDocument = loadHtmlDocument(urlToCrawl);
Elements findedLinks = htmlDocument.select("a[href]");
for (Element link : findedLinks) {
String absLink = link.attr("abs:href");
if (absLink.contains(baseDomainUrl) && !absLink.contains("#")) { //Check that we are don't go out of domain
executorService.execute(new Crawler(absLink)); //Start new thread for each funded link
}
}
}
}
private Document loadHtmlDocument(String internetResourceUrl) {
Document document = null;
try {
document = Jsoup.connect(internetResourceUrl).ignoreHttpErrors(true).ignoreContentType(true)
.userAgent("Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0")
.timeout(10000).get();
} catch (IOException e) {
System.out.println("Page load error");
e.printStackTrace();
}
return document;
}
}
}
This app need about 20 secs to scan jsoup.org for all unique links. But it just wait 10 minutes executorService.awaitTermination(10, TimeUnit.MINUTES);
and then I see dead main thread and still working executor.
Threads
How to force ExecutorService work correctly?
I think problem is that it invoke executorService.execute inside another task instead in main thread.
You are misusing awaitTermination. According to javadoc you should call shutdown first:
Blocks until all tasks have completed execution after a shutdown request, or the timeout occurs, or the current thread is interrupted, whichever happens first.
To achieve your goal I'd suggest to use CountDownLatch (or latch that support increments like this one) to determine exact moment when there is no tasks left so you safely can do shutdown.
I see your comment from earlier:
I can't use CountDownLatch because I don't know beforehand how many unique links I will collect from resource.
First off, vsminkov is spot on with the answer as to why awaitTermniation will sit and wait for 10 minutes. I will offer an alternate solution.
Instead of using a CountDownLatch use a Phaser. For each new task, you can register, and await completion.
Create a single phaser and register each time a execute.submit is invoked and arrive each time a Runnable completes.
public void crawlInternetResource(String baseDomainUrl) {
this.baseDomainUrl = baseDomainUrl;
Phaser phaser = new Phaser();
executorService.execute(new Crawler(phaser, baseDomainUrl));
int phase = phaser.getPhase();
phase.awaitAdvance(phase);
}
private class Crawler implements Runnable {
private final Phaser phaser;
private String urlToCrawl;
public Crawler(Phaser phaser, String urlToCrawl) {
this.urlToCrawl = urlToCrawl;
this.phaser = phaser;
phaser.register(); // register new task
}
public void run(){
...
phaser.arrive(); //may want to surround this in try/finally
}
You are not calling shutdown.
This may work - An AtomicLong variable in the CrawlerService. Increment before every new sub task is submitted to executor service.
Modify your run() method to decrement this counter and if 0, shutdown the executor service
public void run() {
try {
findAllLinks();
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
//decrements counter
//If 0, shutdown executor from here or just notify CrawlerService who would be doing wait().
}
}
In the "finally", reduce the counter and when the counter is zero, shutdown executor or just notify CrawlerService. 0 means, this is the last one, no other is running, none pending in queue. No task will submit any new sub tasks.
How to force ExecutorService work correctly?
I think problem is that it invoke executorService.execute inside another task instead in main thread.
No. The problem is not with ExecutorService. You are using APIs in incorrect manner and hence not getting right result.
You have to use three APIs in a certain order to get right result.
1. shutdown
2. awaitTermination
3. shutdownNow
Recommended way from oracle documentation page of ExecutorService:
void shutdownAndAwaitTermination(ExecutorService pool) {
pool.shutdown(); // Disable new tasks from being submitted
try {
// Wait a while for existing tasks to terminate
if (!pool.awaitTermination(60, TimeUnit.SECONDS)) {
pool.shutdownNow(); // Cancel currently executing tasks
// Wait a while for tasks to respond to being cancelled
if (!pool.awaitTermination(60, TimeUnit.SECONDS))
System.err.println("Pool did not terminate");
}
} catch (InterruptedException ie) {
// (Re-)Cancel if current thread also interrupted
pool.shutdownNow();
// Preserve interrupt status
Thread.currentThread().interrupt();
}
shutdown(): Initiates an orderly shutdown in which previously submitted tasks are executed, but no new tasks will be accepted.
shutdownNow():Attempts to stop all actively executing tasks, halts the processing of waiting tasks, and returns a list of the tasks that were awaiting execution.
awaitTermination():Blocks until all tasks have completed execution after a shutdown request, or the timeout occurs, or the current thread is interrupted, whichever happens first.
On a different note: If you want to wait for all tasks to complete, refer to this related SE question:
wait until all threads finish their work in java
I prefer using invokeAll() or ForkJoinPool(), which are best suited for your use case.
I have multiple tasks/Runnable (i.e. downloading images from internet), which are generated as the user scrolls through a list in a Android App.
I cannot control how many tasks/Runnable are generated at a time, this could in 100. But I want to execute only n(10) tasks in parallel. So, I am planning to build a design, where as soon as a new task/runnable is generated, it will be added to a queue (List<Runnable>) and through Executors.newFixedThreadPool(10), I will execute only first 10 runnable tasks in parallel. Now as soon as the tasks/Runnable are completed, I should be able to remove them from queue (List<Runnable>) and should be able to execute new tasks/Runnable which are in queue, in FIFO.
I have two classes for this design. First is ExecutorManager which is a singleton class and manages the executions of 10 parallels tasks and second is the ImageDownloader class which implements runnable and is responsible to download the image. I am not sure what is the best way to inform the ExecutorManager that the task/download is completed and it can execute new task from the queue. I am following FIFO, so I will always start execution from first 10 tasks in the queue, but how will I get to know, which task is completed and which one to remove from the Queue?
public class ImageDownloader implements Runnable{
DownloadListener mDownloadListener;
public ImageDownloader(DownloadListener mDownloadListener, String URL){
this.mDownloadListener = mDownloadListener;
}
#Override
public void run() {
//Download the Image from Internet
//ToDo
//if Success in download
mDownloadListener.onDownloadComplete();
//if Error in download
mDownloadListener.onDownloadFailure();
//Inform the Executor Manager that the task is complete and it can start new task
incrementCount();
}
private static synchronized void incrementCount(){
ExecutorManager.getInstance().OnTaskCompleted();// is there a better way to do it
}
}
public class ExecutorManager {
private static ExecutorManager Instance;
ExecutorService executor = Executors.newFixedThreadPool(Constants.NumberOfParallelThread);
ArrayList<Runnable> ExecutorQueue = new ArrayList<Runnable>();
int ActiveNumberOfThread = 0;
private ExecutorManager(){
}
public static ExecutorManager getInstance(){
if(Instance==null){
Instance = new ExecutorManager();
}
return Instance;
}
private void executeTask(){
if(ExecutorQueue.size()>0 && ActiveNumberOfThread < Constants.NumberOfParallelThread){
++ActiveNumberOfThread;
executor.execute(ExecutorQueue.get(0));//Execute the First Task in Queue
}
}
public void enQueueTask(Runnable Task){
ExecutorQueue.add(Task);
executeTask();
}
public void removeFromQueue(){
//How to know, which task to remove?
ExecutorQueue.remove(0);
}
public void OnTaskCompleted(){
--ActiveNumberOfThread;
removeFromQueue();
executeTask();
}
}
Well, you're in luck. You don't have to tell the ExecutorManager anything at all. An ExecutorService with a BlockingQueue handles queues for you. All you have to do is submit the Runnable to the ExecutorService. It will hold on to it. If there are any open threads, it will run it instantly. Else, it will wait until one of the other Runnables finishes execution. Once it finishes, it will take the next one.
If you look at the source code for Executors#newFixedThreadPool, it actually just creates a ThreadPoolExecutor with nThreads threads backed by a LinkedBlockingQueue like so:
public static ExecutorService newFixedThreadPool(int nThreads) {
return new ThreadPoolExecutor(nThreads, nThreads,
0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>());
}
I currently have the main method which collects tasks. After the tasks are collected, the ExecutorService is being called with fixed thread pool size. The tasks are iterated and submitted to the executor.
But I need to refresh the tasks and if there is any new tasks available I am adding it to the executor. But if one of the thread is free without any tasks to be allocated from the queue, I want that thread to manually notify my main thread to refresh the events and submit to executor even before the manual refresh is happening from my end. How can i achieve this. Thanks
Sample Code
public class Sample{
Map<String, List<Integer>> tasks;
ThreadPoolExecutor executor;
public static void main(String[] args) {
executor = Executors.newFixedThreadPool(2);
tasks = Collections.synchronizedMap(new HashMap<String, List<Integer>>());
tasks = Tasks.refresh(); //This will get me a new set of data to be processed
invokeexecutor();
}
public void invokeexecutor(){
for(String key: tasks.keyset())
{
executor.submit(new TaskRunnable(tasks.get(key));
}
tasks.clear(); //Remove the allocated tasks from the collection
}
}
public class TaskRunnable implements Runnable{
public void run(){
//Do some logic
}
}
In this case I want my data in the tasks to be continuously refreshed after 10 seconds lets say or if any of the executor thread is free this refresh must happen and a new runnable mus tbe assinged to the thread.
But if one of the thread is free without any tasks to be allocated from the queue, I want that thread to manually notify my main thread to refresh the events and submit to executor even before the manual refresh is happening from my end. How can i achieve this.
There are a couple of ways you can achieve this easily. One way is to create the `ThreadPoolExecutor' yourself.
ThreadPoolExecutor executor = new ThreadPoolExecutor(2, 2, 0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>());
Then have a polling thread that watches the ThreadPoolExecutor class to determine if there are any free threads. Something like:
while (!Thread.currentThread().isInterrupted()) {
// sleep a bit
Thread.sleep(1000);
if (executor.getActiveCount() < 2) {
// add tasks here
}
}
A polling thread is a bit gross however. Another idea which is a bit simpler would be to use a fixed size queue of tasks and then always be trying to add the tasks to the list. This will block if the queue is full. Something like:
// create a limited blocking queue
ThreadPoolExecutor executor = new ThreadPoolExecutor(2, 2, 0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>(10));
while (!Thread.currentThread().isInterrupted()) {
// add tasks here which will block if too many in the queue
}
You can try to override afterExecute method in ThreadPoolExecutor. It is called when a thread in a pool executed a task.
class MyThreadPoolExecutor extends ThreadPoolExecutor {
public MyThreadPoolExecutor {
super(/*Call one of TheadPoolExecutor constructors*/)
}
protected afterExecute(Runnable r, Throwable t) {
// Notify main thread here
}
}
Anybody able to provide me with an example of getting a RejectedExecutionException
Possibly a real life example.
Thanks in advance.
Anybody able to provide me with an example of getting a RejectedExecutionException Possibly a real life example.
Sure. The following code submits 2 jobs into a thread-pool with only 1 thread running. It uses a SynchronousQueue which means that no jobs will be stored in the job queue.
Since each job takes a while to run, the 2nd execute fills the queue and throws a RejectedExecutionException.
// create a 1 thread pool with no buffer for the runnable jobs
ExecutorService threadPool =
new ThreadPoolExecutor(1, 1, 0L, TimeUnit.MILLISECONDS,
new SynchronousQueue<Runnable>());
// submit 2 jobs that take a while to run
/// this job takes the only thread
threadPool.execute(new SleepRunnable());
// this tries to put the job into the queue, throws RejectedExecutionException
threadPool.execute(new SleepRunnable());
public class SleepRunnable implements Runnable {
public void run() {
try {
// this just sleeps for a while which pauses the thread
Thread.sleep(10000);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
return;
}
}
}
Sending tasks to an executor after calling shutdown( on it will throw this exception.
In addition, if the executor uses a bounded blocking queue if the queue is full submitting the task will not block but will fail-fast with the exception.
This question has already been asked and answered :
What could be the cause of RejectedExecutionException
Submitting tasks to a thread-pool gives RejectedExecutionException
This code gives you the error because we try to launch the task but the executor is shut down you can refer to the link above for further explications the answer looked pretty complete:
public class Executorz {
public static void main(String[] args) {
Executorz ex = new Executorz();
ExecutorService es = Executors.newFixedThreadPool(10);
for (int i = 0; i<100 ; i++){
System.out.println("Executed");
es.execute(ex.getNewCountin());
if (i==20)
es.shutdown();
}
}
public Countin getNewCountin(){
return new Countin();
}
public class Countin implements Runnable {
#Override
public void run() {
for (double i =0; i<1000000000 ; i++){
}
System.out.println("Done");
}
}
}