We Keep Coding

Java best practice for handling high amounts of stateful events

I'm looking for a utility class or a best practice pattern to handle lot's of incoming stateful events in my application.
Imagine a producer that produces many events that are then consumed by an application that acts upon these events. Now in some situation the producer is producing more events than the consumer can actually handle, but because all events are stateful, it doesn't matter if some events would be missed, because the latest event contains all the information the previous events conveyed.
I have now written the following java code to handle these situations, but I'm unsure if this is the correct way of doing this, and if there isn't an easier, nicer, more secure way of doing this.
private static ScheduledThreadPoolExecutor executorService = new ScheduledThreadPoolExecutor(1);
private final static Object lock = new Object();
private static List<EventData> lastEventData = null;
static {
executorService.scheduleWithFixedDelay(new Runnable() {
#Override
public void run() {
synchronized(lock) {
while(lastEventData == null && !executorService.isShutdown()) {
try {
lock.wait();
} catch (InterruptedException ex) { ... }
}
try {
actUponEvent(lastEventData);
} catch (Throwable ex) { ... }
lastEventData = null;
}
}
}, 250, 250, TimeUnit.MILLISECONDS);
}
public synchronized update(final List<EventData> data) {
synchronized(lock) {
lastEventData = data;
lock.notifyAll();
}
}
public void dispose() {
executorService.shutdown();
}
In order words, I'd like to get event notifications as soon as the arrive, but rate limit them to one event every 250ms and I'm only interested in the last incoming event.
I looked through java.util.concurrent for some hints / pre existing solutions but couldn't find anything that would fit my problem. The BlockingQueue seems to be very nice at first because it blocks if empty, but on the other hand, the queue itself is not important for me, as I'm only interested in the latest event anyway and the blocking on insert if full is not what I'm looking for either.

The following model can support very high update rates, (into the tens of millions per second) but you only need to keep the latest in memory.
If you are taking a snapshot every N ms, you can use this approach.
final AtomicReference<ConcurrentHashMap<Key, Event>> mapRef =
When you have an update, add it to a ConcurrentMap. The keys are chosen so that an event which should replace a previous one has the same key.
Key key = keyFor(event);
mapRef.get().put(key, event);
This way to map has the latest update for any key at a moment.
Have a task which runs every N ms. This task when it runs can swap the map for another one (or a previous empty one to avoid creating new ones)
ConcurrentMap<Key, Event> prev = mapRef.set(prevEmptyMap);
for(Event e: prev.values())
process(e);
prev.clear();
this.prevEmptymap = prev;

Java LinkedBlockingQueue with ability to signal when done?

I have a situation of a single producer and single consumer working with a queue of objects. There are two situations when the queue might be empty:
The consumer handled the objects quicker than the producer was capable of generating new objects (producer uses I/O before generating objects).
The producer is done generating objects.
If the queue is empty, I want the consumer to wait until a new object is available or until the producer signals that it is done.
My research so far got me no where because I still ended up with a loop that checks both the queue and a separate boolean flag (isDone). Given that there's no way of waiting on multiple locks (thought of waiting on the queue AND the flag), what can be done to solve this?

First of all, the suggestion that using a wrapper is "too much overhead" is a guess, and IMO a very bad one. This assumption should be measured with a performance test with actual requirements. If and only if the test fails, then verify using a profiler that wrapping the queue object is why.
Still if you do that and wrapping the queue object (in this case a String) really is the cause of unacceptable performance, then you can use this technique: create a known, unique string to serve as an "end of messages" message.
public static final String NO_MORE_MESSAGES = UUID.randomUUID().toString();
Then when retrieving Strings from the queue, just check (it can be an reference check) if the String is NO_MORE_MESSAGES. If so, then you're done processing.

Simple. Define a special object that the producer can send to signal "done".

One option is to wrap your data in a holder object, which can be used to signal the end of processing.
For example:
public class QueueMessage {
public MessageType type;
public Object thingToWorkOn;
}
where MessageType is an enum defining a "work" message or a "shutdown" message.

You could use LinkedBlockingQueues poll(long timeout, TimeUnit unit) -method in the consumer, and if it returns null (the timout elapsed), check the boolean flag. Another way would be passing some special "EndOfWork"-object into the queue as the last one, so the consumer knows that it's the end of work.
Yet another way would be interrupting the consumer thread from the producer thread, but this would require the producer thread to be aware of the consumer. If they both would be implemented as nested classes, you could use the parent class to hold a boolean running-value, which both could access, and terminate both threads with single boolean.

The following option has been raised too (not sure if this should be in an answer to myself but couldn't find a better place to write this):
Create a wrapper for the queue. This wrapper will have a monitor that will be waited on when reading by the consumer and will be notified by the producer whenever either a new object is added or the flag of isDone is raised.
When the consumer reads objects from the queue, these objects will be wrapped with something similar to what #yann-ramin suggested above. To reduce overhead though, the consumer will provide a single, reusable, instance of QueueMessage upon every read call (it will always be the same instance). The queue wrapper will update the fields accordingly before returning the instance to the consumer.
This avoids any use of timeouts, sleeps, etc.
EDITED
This is a proposed implementation:
/**
* This work queue is designed to be used by ONE producer and ONE consumer
* (no more, no less of neither). The work queue has certain added features, such
* as the ability to signal that the workload generation is done and nothing will be
* added to the queue.
*
* #param <E>
*/
public class DefiniteWorkQueue<E> {
private final E[] EMPTY_E_ARRAY;
private LinkedBlockingQueue<E> underlyingQueue = new LinkedBlockingQueue<E>();
private boolean isDone = false;
// This monitor allows for flagging when a change was done.
private Object changeMonitor = new Object();
public DefiniteWorkQueue(Class<E> clazz) {
// Reuse this instance, makes calling toArray easier
EMPTY_E_ARRAY = (E[]) Array.newInstance(clazz, 0);
}
public boolean isDone() {
return isDone;
}
public void setIsDone() {
synchronized (changeMonitor) {
isDone = true;
changeMonitor.notifyAll();
}
}
public int size() {
return underlyingQueue.size();
}
public boolean isEmpty() {
return underlyingQueue.isEmpty();
}
public boolean contains(E o) {
return underlyingQueue.contains(o);
}
public Iterator<E> iterator() {
return underlyingQueue.iterator();
}
public E[] toArray() {
// The array we create is too small on purpose, the underlying
// queue will extend it as needed under a lock
return underlyingQueue.toArray(EMPTY_E_ARRAY);
}
public boolean add(E o) {
boolean retval;
synchronized (changeMonitor) {
retval = underlyingQueue.add(o);
if (retval)
changeMonitor.notifyAll();
}
return retval;
}
public boolean addAll(Collection<? extends E> c) {
boolean retval;
synchronized (changeMonitor) {
retval = underlyingQueue.addAll(c);
if (retval)
changeMonitor.notifyAll();
}
return retval;
}
public void remove(RemovalResponse<E> responseWrapper) throws InterruptedException {
synchronized (changeMonitor) {
// If there's nothing in the queue but it has not
// ended yet, wait for someone to add something.
if (isEmpty() && !isDone())
changeMonitor.wait();
// When we get here, we've been notified or
// the current underlying queue's state is already something
// we can respond about.
if (!isEmpty()) {
responseWrapper.type = ResponseType.ITEM;
responseWrapper.item = underlyingQueue.remove();
} else if (isDone()) {
responseWrapper.type = ResponseType.IS_DONE;
responseWrapper.item = null;
} else {
// This should not happen
throw new IllegalStateException(
"Unexpected state where a notification of change was made but " +
"nothing is in the queue and work is not done.");
}
}
}
public static class RemovalResponse<E> {
public enum ResponseType {
/**
* Used when the response contains the first item of the queue.
*/
ITEM,
/**
* Used when the work load is done and nothing new will arrive.
*/
IS_DONE
};
private ResponseType type;
private E item;
public ResponseType getType() {
return type;
}
public void setType(ResponseType type) {
this.type = type;
}
public E getItem() {
return item;
}
public void setItem(E item) {
this.item = item;
}
}
}

Extending java's ThreadLocal to allow the values to be reset across all threads

After looking at this question, I think I want to wrap ThreadLocal to add a reset behavior.
I want to have something similar to a ThreadLocal, with a method I can call from any thread to set all the values back to the same value. So far I have this:
public class ThreadLocalFlag {
private ThreadLocal<Boolean> flag;
private List<Boolean> allValues = new ArrayList<Boolean>();
public ThreadLocalFlag() {
flag = new ThreadLocal<Boolean>() {
#Override protected Boolean initialValue() {
Boolean value = false;
allValues.add(value);
return value;
}
};
}
public boolean get() {
return flag.get();
}
public void set(Boolean value) {
flag.set(value);
}
public void setAll(Boolean value) {
for (Boolean tlValue : allValues) {
tlValue = value;
}
}
}
I'm worried that the autoboxing of the primitive may mean the copies I've stored in the list will not reference the same variables referenced by the ThreadLocal if I try to set them. I've not yet tested this code, and with something tricky like this I'm looking for some expert advice before I continue down this path.
Someone will ask "Why are you doing this?". I'm working in a framework where there are other threads that callback into my code, and I don't have references to them. Periodically I want to update the value in a ThreadLocal variable they use, so performing that update requires that the thread which uses the variable do the updating. I just need a way to notify all these threads that their ThreadLocal variable is stale.
I'm flattered that there is new criticism recently regarding this three year old question, though I feel the tone of it is a little less than professional. The solution I provided has worked without incident in production during that time. However, there are bound to be better ways to achieve the goal that prompted this question, and I invite the critics to supply an answer that is clearly better. To that end, I will try to be more clear about the problem I was trying to solve.
As I mentioned earlier, I was using a framework where multiple threads are using my code, outside my control. That framework was QuickFIX/J, and I was implementing the Application interface. That interface defines hooks for handling FIX messages, and in my usage the framework was configured to be multithreaded, so that each FIX connection to the application could be handled simultaneously.
However, the QuickFIX/J framework only uses a single instance of my implementation of that interface for all the threads. I'm not in control of how the threads get started, and each is servicing a different connection with different configuration details and other state. It was natural to let some of that state, which is frequently accessed but seldom updated, live in various ThreadLocals that load their initial value once the framework has started the thread.
Elsewhere in the organization, we had library code to allow us to register for callbacks for notification of configuration details that change at runtime. I wanted to register for that callback, and when I received it, I wanted to let all the threads know that it's time to reload the values of those ThreadLocals, as they may have changed. That callback comes from a thread I don't control, just like the QuickFIX/J threads.
My solution below uses ThreadLocalFlag (a wrapped ThreadLocal<AtomicBoolean>) solely to signal the other threads that it may be time to update their values. The callback calls setAll(true), and the QuickFIX/J threads call set(false) when they begin their update. I have downplayed the concurrency issues of the ArrayList because the only time the list is added to is during startup, and my use case was smaller than the default size of the list.
I imagine the same task could be done with other interthread communication techniques, but for what it's doing, this seemed more practical. I welcome other solutions.

Interacting with objects in a ThreadLocal across threads
I'll say up front that this is a bad idea. ThreadLocal is a special class which offers speed and thread-safety benefits if used correctly. Attempting to communicate across threads with a ThreadLocal defeats the purpose of using the class in the first place.
If you need access to an object across multiple threads there are tools designed for this purpose, notably the thread-safe collections in java.util.collect.concurrent such as ConcurrentHashMap, which you can use to replicate a ThreadLocal by using Thread objects as keys, like so:
ConcurrentHashMap<Thread, AtomicBoolean> map = new ConcurrentHashMap<>();
// pass map to threads, let them do work, using Thread.currentThread() as the key
// Update all known thread's flags
for(AtomicBoolean b : map.values()) {
b.set(true);
}
Clearer, more concise, and avoids using ThreadLocal in a way it's simply not designed for.
Notifying threads that their data is stale
I just need a way to notify all these threads that their ThreadLocal variable is stale.
If your goal is simply to notify other threads that something has changed you don't need a ThreadLocal at all. Simply use a single AtomicBoolean and share it with all your tasks, just like you would your ThreadLocal<AtomicBoolean>. As the name implies updates to an AtomicBoolean are atomic and visible cross-threads. Even better would be to use a real synchronization aid such as CyclicBarrier or Phaser, but for simple use cases there's no harm in just using an AtomicBoolean.
Creating an updatable "ThreadLocal"
All of that said, if you really want to implement a globally update-able ThreadLocal your implementation is broken. The fact that you haven't run into issues with it is only a coincidence and future refactoring may well introduce hard-to-diagnose bugs or crashes. That it "has worked without incident" only means your tests are incomplete.
First and foremost, an ArrayList is not thread-safe. You simply cannot use it (without external synchronization) when multiple threads may interact with it, even if they will do so at different times. That you aren't seeing any issues now is just a coincidence.
Storing the objects as a List prevents us from removing stale values. If you call ThreadLocal.set() it will append to your list without removing the previous value, which introduces both a memory leak and the potential for unexpected side-effects if you anticipated these objects becoming unreachable once the thread terminated, as is usually the case with ThreadLocal instances. Your use case avoids this issue by coincidence, but there's still no need to use a List.
Here is an implementation of an IterableThreadLocal which safely stores and updates all existing instances of the ThreadLocal's values, and works for any type you choose to use:
import java.util.Iterator;
import java.util.concurrent.ConcurrentMap;
import com.google.common.collect.MapMaker;
/**
* Class extends ThreadLocal to enable user to iterate over all objects
* held by the ThreadLocal instance. Note that this is inherently not
* thread-safe, and violates both the contract of ThreadLocal and much
* of the benefit of using a ThreadLocal object. This class incurs all
* the overhead of a ConcurrentHashMap, perhaps you would prefer to
* simply use a ConcurrentHashMap directly instead?
*
* If you do really want to use this class, be wary of its iterator.
* While it is as threadsafe as ConcurrentHashMap's iterator, it cannot
* guarantee that all existing objects in the ThreadLocal are available
* to the iterator, and it cannot prevent you from doing dangerous
* things with the returned values. If the returned values are not
* properly thread-safe, you will introduce issues.
*/
public class IterableThreadLocal<T> extends ThreadLocal<T>
implements Iterable<T> {
private final ConcurrentMap<Thread,T> map;
public IterableThreadLocal() {
map = new MapMaker().weakKeys().makeMap();
}
#Override
public T get() {
T val = super.get();
map.putIfAbsent(Thread.currentThread(), val);
return val;
}
#Override
public void set(T value) {
map.put(Thread.currentThread(), value);
super.set(value);
}
/**
* Note that this method fundamentally violates the contract of
* ThreadLocal, and exposes all objects to the calling thread.
* Use with extreme caution, and preferably only when you know
* no other threads will be modifying / using their ThreadLocal
* references anymore.
*/
#Override
public Iterator<T> iterator() {
return map.values().iterator();
}
}
As you can hopefully see this is little more than a wrapper around a ConcurrentHashMap, and incurs all the same overhead as using one directly, but hidden in the implementation of a ThreadLocal, which users generally expect to be fast and thread-safe. I implemented it for demonstration purposes, but I really cannot recommend using it in any setting.

It won't be a good idea to do that since the whole point of thread local storage is, well, thread locality of the value it contains - i.e. that you can be sure that no other thread than your own thread can touch the value. If other threads could touch your thread local value, it won't be "thread local" anymore and that will break the memory model contract of thread local storage.
Either you have to use something other than ThreadLocal (e.g. a ConcurrentHashMap) to store the value, or you need to find a way to schedule an update on the threads in question.
You could use google guava's map maker to create a static final ConcurrentWeakReferenceIdentityHashmap with the following type: Map<Thread, Map<String, Object>> where the second map is a ConcurrentHashMap. That way you'd be pretty close to ThreadLocal except that you can iterate through the map.

I'm disappointed in the quality of the answers received for this question; I have found my own solution.
I wrote my test case today, and found the only issue with the code in my question is the Boolean. Boolean is not mutable, so my list of references wasn't doing me any good. I had a look at this question, and changed my code to use AtomicBoolean, and now everything works as expected.
public class ThreadLocalFlag {
private ThreadLocal<AtomicBoolean> flag;
private List<AtomicBoolean> allValues = new ArrayList<AtomicBoolean>();
public ThreadLocalFlag() {
flag = new ThreadLocal<AtomicBoolean>() {
#Override protected AtomicBoolean initialValue() {
AtomicBoolean value = new AtomicBoolean();
allValues.add(value);
return value;
}
};
}
public boolean get() {
return flag.get().get();
}
public void set(boolean value) {
flag.get().set(value);
}
public void setAll(boolean value) {
for (AtomicBoolean tlValue : allValues) {
tlValue.set(value);
}
}
}
Test case:
public class ThreadLocalFlagTest {
private static ThreadLocalFlag flag = new ThreadLocalFlag();
private static boolean runThread = true;
#AfterClass
public static void tearDownOnce() throws Exception {
runThread = false;
flag = null;
}
/**
* #throws Exception if there is any issue with the test
*/
#Test
public void testSetAll() throws Exception {
startThread("ThreadLocalFlagTest-1", false);
try {
Thread.sleep(1000L);
} catch (InterruptedException e) {
//ignore
}
startThread("ThreadLocalFlagTest-2", true);
try {
Thread.sleep(1000L);
} catch (InterruptedException e) {
//ignore
}
startThread("ThreadLocalFlagTest-3", false);
try {
Thread.sleep(1000L);
} catch (InterruptedException e) {
//ignore
}
startThread("ThreadLocalFlagTest-4", true);
try {
Thread.sleep(8000L); //watch the alternating values
} catch (InterruptedException e) {
//ignore
}
flag.setAll(true);
try {
Thread.sleep(8000L); //watch the true values
} catch (InterruptedException e) {
//ignore
}
flag.setAll(false);
try {
Thread.sleep(8000L); //watch the false values
} catch (InterruptedException e) {
//ignore
}
}
private void startThread(String name, boolean value) {
Thread t = new Thread(new RunnableCode(value));
t.setName(name);
t.start();
}
class RunnableCode implements Runnable {
private boolean initialValue;
RunnableCode(boolean value) {
initialValue = value;
}
#Override
public void run() {
flag.set(initialValue);
while (runThread) {
System.out.println(Thread.currentThread().getName() + ": " + flag.get());
try {
Thread.sleep(4000L);
} catch (InterruptedException e) {
//ignore
}
}
}
}
}

What is the correct way to pass data to a running thread

In most cases when you create your thread you can prepare the data beforehand and pass it into the constructor or method.
However in cases like an open socket connection you will typically already have a thread created but wish to tell it to perform some action.
Basic idea:
C#
private Thread _MyThread = new Thread(MyMethod);
this._MyThread.Start(param);
Java
private Thread _MyThread = new Thread(new MyRunnableClass(param));
this._MyThread.start();
Now what?
So what is the correct way to pass data to a running thread in C# and Java?

One way to pass data to a running thread is by implementing Message Queues. The thread that wants to tell the listening thread to do something would add an item to the queue of the listening thread. The listening thread reads from this thread in a blocking fashion. Causing it to wait when there are no actions to perform. Whenever another thread puts a message in the queue it will fetch the message, depending on the item and it's content you can then do something with it.
This is some Java / pseudo code:
class Listener
{
private Queue queue;
public SendMessage(Message m)
{
// This will be executed in the calling thread.
// The locking will be done either in this function or in the Add below
// depending on your Queue implementation.
synchronize(this.queue)
{
this.queue.put(m);
}
}
public Loop()
{
// This function should be called from the Listener thread.
while(true)
{
Message m = this.queue.take();
doAction(m);
}
}
public doAction(Message m)
{
if (m is StopMessage)
{
...
}
}
}
And the caller:
class Caller
{
private Listener listener;
LetItStop()
{
listener.SendMessage(new StopMessage());
}
}
Of course, there are a lot of best practices when programming paralllel/concurrent code. For example, instead of while(true) you should at the least add a field like run :: Bool that you can set to false when you receive a StopMessage. Depending on the language in which you want to implement this you will have other primitives and behaviour to deal with.
In Java for example you might want to use the java.util.Concurrent package to keep things simple for you.

Java
You could basically have a LinkedList (a LIFO) and proceed (with something) like this (untested) :
class MyRunnable<T> implements Runnable {
private LinkedList<T> queue;
private boolean stopped;
public MyRunnable(LinkedList<T> queue) {
this.queue = queue;
this.stopped = false;
}
public void stopRunning() {
stopped = true;
synchronized (queue) {
queue.notifyAll();
}
}
public void run() {
T current;
while (!stopped) {
synchronized (queue) {
queue.wait();
}
if (queue.isEmpty()) {
try { Thread.sleep(1); } catch (InterruptedException e) {}
} else {
current = queue.removeFirst();
// do something with the data from the queue
}
Thread.yield();
}
}
}
As you keep a reference to the instance of the LinkedList given in argument, somewhere else, all you have to do is :
synchronized (queue) {
queue.addLast(T); // add your T element here. You could even handle some
// sort of priority queue by adding at a given index
queue.notifyAll();
}

Edit: Misread question,
C#
What I normally do is create a Global Static Class and then set the values there. That way you can access it from both threads. Not sure if this is the preferred method and there could be cases where locking occurs (correct me if I'm wrong) which should be handled.
I haven't tried it but It should work for for the threadpool/backgroundworker as well.

One way I can think of is through property files.

Well, it depends a lot on the work that the thread is supposed to do.
For example, you can have a thread waiting for a Event (e.g. ManualResetEvent) and a shared queue where you put work items (can be data structures to be processed, or more clever commands following a Command pattern). Somebody adds new work to the queue ad signals the event, so the trhread awakes, gets work from the queue and start performing its task.
You can encapsulate this code inside a custom queue, where any thread that calls the Deque methods stops until somebody calls Add(item).
On the other hand, maybe you want to rely on .NET ThreadPool class to issue tasks to execute by the threads on the pool.
Does this example help a bit?

You can use delegate pattern where child threads subscribes to an event and main thread raises an event, passing the parameters.

You could run your worker thread within a loop (if that makes sense for your requirement) and check a flag on each execution of the loop. The flag would be set by the other thread to signal the worker thread that some state had changed, it could also set a field at the same time to pass the new state.
Additionally, you could use monitor.wait and monitor.pulse to signal the state changes between the threads.
Obviously, the above would need synchronization.

Queue with notifications on isEmpty() changes

I have an BlockingQueue<Runnable>(taken from ScheduledThreadPoolExecutor) in producer-consumer environment. There is one thread adding tasks to the queue, and a thread pool executing them.
I need notifications on two events:
First item added to empty queue
Last item removed from queue
Notification = writing a message to database.
Is there any sensible way to implement that?

A simple and naïve approach would be to decorate your BlockingQueue with an implementation that simply checks the underlying queue and then posts a task to do the notification.
NotifyingQueue<T> extends ForwardingBlockingQueue<T> implements BlockingQueue<T> {
private final Notifier notifier; // injected not null
…
#Override public void put(T element) {
if (getDelegate().isEmpty()) {
notifier.notEmptyAnymore();
}
super.put(element);
}
#Override public T poll() {
final T result = super.poll();
if ((result != null) && getDelegate().isEmpty())
notifier.nowEmpty();
}
… etc
}
This approach though has a couple of problems. While the empty -> notEmpty is pretty straightforward – particularly for a single producer case, it would be easy for two consumers to run concurrently and both see the queue go from non-empty -> empty.
If though, all you want is to be notified that the queue became empty at some time, then this will be enough as long as your notifier is your state machine, tracking emptiness and non-emptiness and notifying when it changes from one to the other:
AtomicStateNotifier implements Notifier {
private final AtomicBoolean empty = new AtomicBoolean(true); // assume it starts empty
private final Notifier delegate; // injected not null
public void notEmptyAnymore() {
if (empty.get() && empty.compareAndSet(true, false))
delegate.notEmptyAnymore();
}
public void nowEmpty() {
if (!empty.get() && empty.compareAndSet(false, true))
delegate.nowEmpty();
}
}
This is now a thread-safe guard around an actual Notifier implementation that perhaps posts tasks to an Executor to asynchronously write the events to the database.

The design is most likely flawed but you can do it relatively simple:
You have a single thread adding, so you can check before adding. i.e. pool.getQueue().isEmpty() - w/ one producer, this is safe.
Last item removed cannot be guaranteed but you can override beforeExecute and check the queue again. Possibly w/ a small timeout after isEmpty() returns true. Probably the code below will be better off executed in afterExecute instead.
protected void beforeExecute(Thread t, Runnable r) {
if (getQueue().isEmpty()){
try{
Runnable r = getQueue().poll(200, TimeUnit.MILLISECONDS);
if (r!=null){
execute(r);
} else{
//last message - or on after execute by Setting a threadLocal and check it there
//alternatively you may need to do so ONLY in after execute, depending on your needs
}
}catch(InterruptedException _ie){
Thread.currentThread().interrupt();
}
}
}
sometime like that
I can explain why doing notifications w/ the queue itself won't work well: imagine you add a task to be executed by the pool, the task is scheduled immediately, the queue is empty again and you will need notification.