GAE just blocks forever when I try to terminate an ExecutorService. Small sample below:
ThreadFactory threadFactory = ThreadManager.currentRequestThreadFactory();
ExecutorService pool = Executors.newSingleThreadExecutor(threadFactory);
Future<String> future = pool.submit(new Callable<String>() {
public String call() throws Exception {
return "Hello from Thread";
}
});
LOG.info("Result is: [" + future.get() + "]. Pool expected to be idle now");
pool.shutdown();
if (!pool.awaitTermination(1, TimeUnit.SECONDS)) {
LOG.info("Pool does not like shutdown()");
pool.shutdownNow();
if (!pool.awaitTermination(1, TimeUnit.SECONDS)) {
LOG.info("Pool does not even like shutdownNow()");
}
}
The same code works without blocking when running locally, it just blocks without terminating when running deployed on AppEngine. The timeout can be increased until the 60 second request limit forces the code to interrupt.
This seems to be a subtle yet dangerous difference to a standard JVM. Code found regularly to clean up can essentially kill your service. ThreadManager documentation mentions that the threads are a bit special but they are -as far as I understand - interruptible and meant to terminate.
Is it just me (some library messing with threads)?
Is it a bug / feature / somewhere documented?
Since waiting for termination is just pointless, is it okay to just call pool.shutdown(), then assume all is going to be okay? Running threads are a good way to leak memory..
Update #1
I'm even more confused after some more testing. All works fine when using a Thread directly. Slightly convoluted example:
final CountDownLatch threadEnter = new CountDownLatch(1);
final Object wait4Interrupt = new Object();
Runnable task = new Runnable() {
public void run() {
synchronized (wait4Interrupt) {
threadEnter.countDown();
try {
wait4Interrupt.wait();
} catch (InterruptedException e) {
// expected to happen since nothing is going to notify()
LOG.info("Thread got interrupted.");
Thread.currentThread().interrupt();
}
}
}
};
Thread thread = ThreadManager.createThreadForCurrentRequest(task);
// not started state
LOG.info("Thread log #1: " + thread + " " + thread.getState());
thread.start();
threadEnter.await();
// thread is inside synchronized / already waiting
synchronized (wait4Interrupt) {
// => guaranteed that thread is in waiting state here
LOG.info("Thread log #2: " + thread + " " + thread.getState());
thread.interrupt();
}
thread.join(1000);
// thread is dead
LOG.info("Thread log #3: " + thread + " " + thread.getState());
Logs produced:
I 16:08:37.213 Thread log #1: Thread[Thread-7,5,Request #0] NEW
I 16:08:37.216 Thread log #2: Thread[Thread-7,5,Request #0] WAITING
I 16:08:37.216 Thread got interrupted.
I 16:08:37.217 Thread log #3: Thread[Thread-7,5,] TERMINATED
The thread returned by the factory isn't started, it supports wait & interrupt just fine and it can be join()'d and is terminated afterwards. What else would an ExecutorService want to do?
Update #2
pool.toString() from example #1 after shutdown() results in
java.util.concurrent.ThreadPoolExecutor#175434a
[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1]
which also indicates that it's not an issue caused by unterminated threads since it states active threads = 0.
Update #3
Pools do shutdown nicely when being told to do so before they finished their task. The following terminates correctly after 500 ms. Adding future.get() will show the original problem again.
Future<String> future = pool.submit(new Callable<String>() {
public String call() throws Exception {
// sleep a bit so pool is "busy" when we're trying to shutdown.
Thread.sleep(500);
return "Hello from Thread";
}
});
// get here = evil
pool.shutdown();
pool.awaitTermination(2, TimeUnit.SECONDS);
=> Issue seems to occur on idle pools only. Busy pool can be shutdown.
You are right, Threads on App Engine are interruptible. Quoting from the official docs:
An application can perform operations against the current thread, such as thread.interrupt().
Since it is working fine locally, it is a difference between the development server and the sandbox at production environment.
I think the development server allows multi-threaded execution if not disabled while the production environment requires to explicitly state it in the application config file (appengine-web.xml):
<threadsafe>true</threadsafe>
Unless you explicitly state your app is thread-safe, serving a request can only use 1 thread therefore your ExecutorService cannot start a new Thread to execute the task you submitted and therefore future.get() will block. It would block until the "current" thread would end but obviously that could only happen after serving the request, so you have a deadlock here.
Related
In the following code two consumer threads start and become waiting. The producer thread starts (very likely) after that and calls 'notify'. All threads use the producer as monitor.
Thread producer = new Thread() {
#Override
public void run() {
synchronized (this) {
System.out.printf("notify at %d %n", getId());
notify();
}
}
};
Runnable consumer = () -> {
try {
synchronized (producer) {
long id = Thread.currentThread().getId();
System.out.printf("wait at %d %n", id);
producer.wait();
System.out.printf("awakened: %d %n", id);
}
} catch (InterruptedException e) {
e.printStackTrace();
}
};
Stream.generate( () -> consumer )
.limit(2)
.map(Thread::new)
.forEach(Thread::start);
Thread.sleep(3000); // consumer threads are (likely) waiting
producer.start();
From javadoc for Object.notify:
Wakes up a single thread that is waiting on this object's monitor.
The code produces this (or similar) output:
wait at 13
wait at 14
notify at 12
awakened: 13
awakened: 14
The point is that both consumer threads are awakened, not just one of them. Why?
Compiled and tested with OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.5+10) under Windows 10, 64-bit.
Thanks in advance!
The issue is that instead of an arbitrary Object you use a Thread as the monitor.
Thread uses signals internally as documented in Thread.join:
As a thread terminates the this.notifyAll method is invoked. It is recommended that applications not use wait, notify, or notifyAll on Thread instances.
The general suggestion is to always use dedicated objects to wait/notify on that no other code can get access to to avoid "spurious" notifies or waits like this.
I'm just exploring method scheduleAtFixedRate of class ScheduledExecutorService in Java.
Here is my suspicious code:
ScheduledExecutorService scheduledExecutorService = Executors.newScheduledThreadPool(5);
Runnable command = () -> {
System.out.println("Yo");
try {
Thread.sleep(4000);
} catch (InterruptedException e) {
e.printStackTrace();
}
};
scheduledExecutorService.scheduleAtFixedRate(command, 0, 1, TimeUnit.SECONDS);
I expected that every 1 second scheduledExecutorService will try to take new thread from the pool and start it.
API says: "scheduledExecutorService creates and executes a periodic action that becomes enabled first after the given initial delay, and subsequently with the given period. /(unimportant deleted)/ If any execution of this task takes longer than its period, then subsequent executions may start late, but will not concurrently execute."
Result - every new thread starts every 4 seconds.
So, the questions:
What's the catch - Does Thread.sleep() stop all threads or nuance in this behavior - "If any execution of this task takes longer than its period, then subsequent executions may start late, but will not concurrently execute"?
If "will not concurrently execute" is true in this situation - why do we need this pool of several threads if every thread will start after execution of previous thread?
Is there any simple valid example of usage of scheduleAtFixedRate, where one thread starts while previous still executes?
The answer is in the quote you provided. Executor waits until the task finishes before launching this task again. It prevents concurrent execution of many instances of one task - in most cases this behaviour is needed. In your case Executor starts a task, then waits 1 second of delay, then waits 3 more seconds until current task is done and only then starts this task again (It does not necessarily start new thread, it may start the task in the same thread).
Your code does not use thread pool at all - you can get exactly same result using single thread executor.
If you want to get this behaviour:
I expected that every 1 second scheduledExecutorService will try to take new thread
from the pool and start it.
Then you may write is like this:
ScheduledExecutorService scheduledExecutorService = Executors.newScheduledThreadPool(5);
Runnable command = () -> {
System.out.println("Yo");
try {
Thread.sleep(4000);
} catch (InterruptedException e) {
e.printStackTrace();
}
};
Runnable commandRunner = () -> {
scheduledExecutorService.schedule(command, 0, TimeUnit.SECONDS);
}
scheduledExecutorService.scheduleAtFixedRate(commandRunner, 0, 1, TimeUnit.SECONDS);
(It's better to create a single-threaded ScheduledExecutorService that runs commandRunner and create a thread pool based ExecutorService that is used by commandRunner to execute command)
What's the catch - Does Thread.sleep() stop all threads or nuance in
this behavior - "If any execution of this task takes longer than its
period, then subsequent executions may start late, but will not
concurrently execute"?
I didn't quite understand what you mean here. But, essentially speaking, in the code that you have shared, Thread.sleep() is just making the thread execution take 4 seconds, which is longer than the set period of 1 second. Thus, subsequent threads will not execute after 1 second, but only after ~4 seconds of execution of the previous thread.
If "will not concurrently execute" is true in this situation - why do
we need this pool of several threads if every thread will start after
execution of previous thread?
You may want to schedule some other type of threads (which do a different job) in the same executor, which may run in parallel to the code which you have shared. Your current code only needs 1 thread in the pool though, since you are scheduling only one job (Runnable).
Is there any simple valid example of usage of scheduleAtFixedRate,
where one thread starts while previous still executes?
As stated in the documentation, concurrent execution will not happen for the job that you scheduled at fixed rate (with the current code)
public class Poll {
ScheduledFuture<?> future;
static int INIT_DELAY = 1;
static int REPEAT_PERIOD = 2;
static int MAX_TRIES = 3;
int tries = 1;
Runnable task = () -> {
System.out.print( tries + ": " + Thread.currentThread().getName() + " " );
if ( ++tries > MAX_TRIES ) {
future.cancel( false );
}
};
void poll() {
ScheduledExecutorService executor = Executors.newScheduledThreadPool(1);
future = executor.scheduleAtFixedRate( task, INIT_DELAY, REPEAT_PERIOD, TimeUnit.SECONDS );
System.out.println( "Start: " + tries + ": " + Thread.currentThread().getName() + " " );
try {
future.get();
} catch ( InterruptedException | ExecutionException e ) {
System.out.println( e.getMessage() );
} catch ( CancellationException e ) {
System.out.println( "Regular End Of Scheduled Task as Designed.");
} finally {
executor.shutdown();
executor.shutdownNow();
}
System.out.println( "Return The Result." );
}
// The Driver
public static void main( String[] args ) {
new Poll().poll();
}
}
How does Executor.newSingleThreadExecutor() behave if I am frequently scheduling tasks to run that are being cancelled with future.cancel(true);?
Does the single thread spawned by the executor get interrupted (so the future code needs to clear the interrupt), or does the interrupt flag get automatically cleared when the next future starts up.
Does the Executor need to spawn an additional thread on every interrupt to be used by the remaining task queue?
Is there a better way?
Good question, I don't find this documented anywhere, so I would say it is implementation dependent.
For example OpenJDK does reset the interrupted flag before every executed task:
// If pool is stopping, ensure thread is interrupted;
// if not, ensure thread is not interrupted. This
// requires a recheck in second case to deal with
// shutdownNow race while clearing interrupt
if ((runStateAtLeast(ctl.get(), STOP) ||
(Thread.interrupted() &&
runStateAtLeast(ctl.get(), STOP))) &&
!wt.isInterrupted())
wt.interrupt();
Snippet from from OpenJDK jdk8u ThreadPoolExecutor#runWorker source.
The following sample program demonstrates that the interrupt is called on the thread if you call the cancel method with true. You can even see that it is reusing the same thread. The cancel returns a boolean which indicates if the cancellation was successful. The javadoc of this method is also clear enough.
class Task implements Callable<String> {
#Override
public String call() throws Exception {
try {
System.out.println("Thread name = " + Thread.currentThread().getName());
Thread.sleep(Integer.MAX_VALUE);
} catch (InterruptedException e) {
System.out.println("Interrupted");
return "Interruped";
}
return "X";
}
}
public class Testy {
public static void main(String[] args) throws InterruptedException {
ExecutorService executorService =
Executors.newSingleThreadExecutor();
int count = 0;
while (true) {
System.out.println("Iteration " + count++);
Future<String> submit = executorService.submit(new Task());
Thread.sleep(500);
submit.cancel(true);
}
}
}
Output looks like below
Iteration 0
Thread name = pool-1-thread-1
Iteration 1
Interrupted
Thread name = pool-1-thread-1
Iteration 2
Interrupted
The javadoc for ExecutorService sometimes refers to the case when a Thread terminates 'due to failure'. However, it is not clear what kind of failure does this refer to.
For instance, the single thread executor documentation says that
if this single thread terminates due to a failure during execution
prior to shutdown, a new one will take its place if needed to execute
subsequent tasks
I would have thought that this situation might happen in case of an Exception, or maybe a RuntimeException, but it does not seem to be the case. Running the following code seems to be giving the same thread name and thread ID.
ExecutorService executor = Executors.newSingleThreadExecutor();
executor.submit(() -> {
System.out.println("Hello from " + Thread.currentThread().getName()+ " " + Thread.currentThread().getId());
throw new NullPointerException("Test");
});
executor.submit(() -> {
System.out.println("Hello 2 from " + Thread.currentThread().getName() + " " + Thread.currentThread().getId());
});
The output of this code is:
Hello from pool-1-thread-1 12
Hello 2 from pool-1-thread-1 12
It seems that the same thread is being reused even in the case of NullPointerException.
So what kind of 'failure' is the Javadoc referring to?
This is an interesting question. Following the code in ThreadPoolExecutor the thread is discarded when a Runnable is passed to the execute() method.
When you call submit() the executor creates a wrapper for the callable/runnable of type FutureTask. FutureTask.run() has some logic to catch exceptions and store them (so then, you can query this from the Future). In this case, the exception never reaches the ThreadPool, so the thread is not discarded.
Augusto is right. Runnable tasks should have discarded the Thread after encountering the exception when they have passed as parameter in execute() method.
I have found concrete evidence regarding swallowing of exceptions by Future tasks at this article and Future Task source code
**Inside FutureTask$Sync**
void innerRun() {
if (!compareAndSetState(READY, RUNNING))
return;
runner = Thread.currentThread();
if (getState() == RUNNING) { // recheck after setting thread
V result;
try {
result = callable.call();
} catch (Throwable ex) {
setException(ex);
return;
}
set(result);
} else {
releaseShared(0); // cancel
}
}
protected void setException(Throwable t) {
sync.innerSetException(t);
}
There are few more interesting questions in SE around this topic.
Catching thread exceptions from Java ExecutorService
Choose between ExecutorService's submit and ExecutorService's execute
EDIT:
Thread failure or termination will happen when an exception is uncaught in the thread code. If you submit task by execute() instead of submit(), exception won't be caught unless you catch the exception. Uncaught exception by the thread code will result thread to terminate or failure and new thread will be created by Executor.
If you submit the task through submit(), a FutureTask will be created and that task will swallow uncaught exception by the code. Since the exception was caught in FutureTask, the thread won't be discarded.
As the title showed, If Future.get(timeout) timeout, does the thread continue running,
ExecutorService executor = Executors.newFixedThreadPool(n);
Callable<Object> task = new Callable<Object>() {
public Object call() {
//...
}
}
Future<Object> future = executor.submit(task);
try {
Object result = future.get(5, TimeUnit.SECONDS);
} catch (TimeoutException ex) {
// handle the timeout
}
If the thread continue to run and get blocked due to some IO, etc, then when the threadpool get full, not new task can be sumitted, which means the trheadpool gets stuck, since all the threads in the pool are blocked, right?
The call to future.get(..) will block the thread running it for up to 5 seconds. The task executed by the thread pool will be unaffected, and will continue running until a graceful termination / exception / interruption.
Regarding the submission of new tasks when the thread pool is in full capacity, in your case the tasks WILL be submitted (releasing the submitter thread immediately), but will wait in the thread pool queue for execution. The API documentation of Executors.newFixedThreadPool(..) specifies this clearly.
Right, underlaying thread will be live until IO thrown an exception or ended.