I have a multithreaded code in java.
I debug with intellij. I know how to switch debug contexts between threads.
Nevertheless the console shows only exception thrown for the main thread.
How can i see the exceptions thrown from any thread?
basically when I run integration test with an Executor, some exception is thrown from secondary thread.
But nothing is printed to the debug-console.
Actually the the code flies and never reach to the try catch i have.
It's like I cannot step-next after the code that throws an exception
try {
while (true) {
logger.debug("New thread. polling on DB, polling on deployment service");
ConfigAction configAction = configActionFactory.get(ConfigActionsEnum.PushToDeployment);
while (configAction != null) {
configAction = configAction.execute() ? configAction.nextActivity() : null;
}
}
} catch (Exception e) {
e.printStackTrace();
logger.error("DeploymentContextListener run failed", e);
}
whereas when I run with the main thread, the same code I see the exception:
public void run() {
try {
while (true) {
logger.debug("New thread. polling on DB, polling on deployment service");
ConfigAction configAction = configActionFactory.get(ConfigActionsEnum.PushToDeployment);
while (configAction != null) {
configAction = configAction.execute() ? configAction.nextActivity() : null;
}
}
} catch (Exception e) {
e.printStackTrace();
logger.error("DeploymentContextListener run failed", e);
}
it is written in one of the try-catch i have
} catch (Exception ex) {
logger.error("failed to get from DB: getLatestConfigVersionWaitingForDeploy", ex);
}
My problem was that i used:
ExecutorService executor = Executors.newSingleThreadExecutor();
executor.submit(() -> { }
from what I have read:
invokeAll - call “future.get()”. Wait for all threads to finish.
Submit -> attach the errors to Future objects, not the the std-error
Execute -> the errors are thrown to the std-error
Related
In the below code snipped when dispose() is called, then the emitter thread is interrupted (InterruptedException is thrown out of sleep method).
Observable<Integer> obs = Observable.create(emitter -> {
for (int i = 0; i < 10; i++) {
if (emitter.isDisposed()) {
System.out.println("> exiting.");
emitter.onComplete();
return;
}
emitter.onNext(i);
System.out.println("> calculation = " + i);
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
emitter.onComplete();
});
Disposable disposable = obs
.subscribeOn(Schedulers.computation())
.subscribe(System.out::println);
try {
Thread.sleep(2000);
} catch (InterruptedException e) {
e.printStackTrace();
}
disposable.dispose();
From debugging session I see that the interrupt origins from FutureTask which is cancelled during disposal. In there, the thread that is calling dispose() is checked against runner thread, and if it does not match, the emitter is interrupted. The thread is different since I used computation Scheduler.
Is there any way to make dispose not interrupt such emitter or is it how this actually should always be handled? An issue I see with this approach is when I would have an interruptible operation (simulated here by sleep) that I would want to complete normally before calling onComplete().
Please refer to What's different in 2.0 - Error handling.
One important design requirement for 2.x is that no Throwable errors should be swallowed. This means errors that can't be emitted because the downstream's lifecycle already reached its terminal state or the downstream cancelled a sequence which was about to emit an error.
So you can either wrap everything inside a try/catch and properly emit the error:
Observable<Integer> obs = Observable.create(emitter -> {
try {
// ...
} catch (InterruptedException ex) {
// check if the interrupt is due to cancellation
// if so, no need to signal the InterruptedException
if (!disposable.isDisposed()) {
observer.onError(ex);
}
}
});
or setup a global error consumer to ignore it:
RxJavaPlugins.setErrorHandler(e -> {
// ..
if (e instanceof InterruptedException) {
// fine, some blocking code was interrupted by a dispose call
return;
}
// ...
Log.warning("Undeliverable exception received, not sure what to do", e);
});
We have a spark streaming program which pull messages from kafka and process each individual message using forEachPartiton transformation.
If case if there is specific error in the processing function we would like to throw the exception back and halt the program. The same seems to be not happening. Below is the code we are trying to execute.
JavaInputDStream<KafkaDTO> stream = KafkaUtils.createDirectStream( ...);
stream.foreachRDD(new Function<JavaRDD<KafkaDTO>, Void>() {
public Void call(JavaRDD<KafkaDTO> rdd) throws PropertiesLoadException, Exception {
rdd.foreachPartition(new VoidFunction<Iterator<KafkaDTO>>() {
#Override
public void call(Iterator<KafkaDTO> itr) throws PropertiesLoadException, Exception {
while (itr.hasNext()) {
KafkaDTO dto = itr.next();
try{
//process the message here.
} catch (PropertiesLoadException e) {
// throw Exception if property file is not found
throw new PropertiesLoadException(" PropertiesLoadException: "+e.getMessage());
} catch (Exception e) {
throw new Exception(" Exception : "+e.getMessage());
}
}
}
});
}
}
In the above code even if we throw a PropertiesLoadException the program doesn't halt and streaming continues. The max retries we set in Spark configuration is only 4. The streaming program continues even after 4 failures. How should the exception be thrown to stop the program?
I am not sure if this is the best approach but we surrounded the main batch with try and catch and when I get exception I just call close context. In addition you need to make sure that stop gracfully is off (false).
Example code:
try {
process(dataframe);
} catch (Exception e) {
logger.error("Failed on write - will stop spark context immediately!!" + e.getMessage());
closeContext(jssc);
if (e instanceof InterruptedException) {
Thread.currentThread().interrupt();
}
throw e;
}
And close function:
private void closeContext(JavaStreamingContext jssc) {
logger.warn("stopping the context");
jssc.stop(false, jssc.sparkContext().getConf().getBoolean("spark.streaming.stopGracefullyOnShutdown", false));
logger.error("Context was stopped");
}
In config :
spark.streaming.stopGracefullyOnShutdown false
I think that with your code it should look like this:
JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, streamBatch);
JavaInputDStream<KafkaDTO> stream = KafkaUtils.createDirectStream( jssc, ...);
stream.foreachRDD(new Function<JavaRDD<KafkaDTO>, Void>() {
public Void call(JavaRDD<KafkaDTO> rdd) throws PropertiesLoadException, Exception {
try {
rdd.foreachPartition(new VoidFunction<Iterator<KafkaDTO>>() {
#Override
public void call(Iterator<KafkaDTO> itr) throws PropertiesLoadException, Exception {
while (itr.hasNext()) {
KafkaDTO dto = itr.next();
try {
//process the message here.
} catch (PropertiesLoadException e) {
// throw Exception if property file is not found
throw new PropertiesLoadException(" PropertiesLoadException: " + e.getMessage());
} catch (Exception e) {
throw new Exception(" Exception : " + e.getMessage());
}
}
}
});
} catch (Exception e){
logger.error("Failed on write - will stop spark context immediately!!" + e.getMessage());
closeContext(jssc);
if (e instanceof InterruptedException) {
Thread.currentThread().interrupt();
}
throw e;
}
}
}
In addition please note that my stream is working on spark 2.1 Standalone (not yarn / mesos) client mode. In addition I implement the stop gracefully my self using ZK.
Recently, I find some BlockingOperationException in my netty4 project.
Some people said that when using the sync() method of start netty's ServerBootstrap can cause dead lock, because sync() will invoke await() method, and there is a method called 'checkDeadLock' in await().
But I don't think so. ServerBootstrap use the EventLoopGroup called boosGroup, and Channel use the workerGroup to operation IO, I don't think they will influence each other, they have different EventExecutor.
And in my practice, Deadlock exception doesn't appear in the Netty startup process, most of which occurs after the Channel of the await writeAndFlush.
Analysis source code, checkDeadLock, BlockingOperationException exception thrown is when the current thread and executor thread of execution is the same.
My project code is blow:
private void channelWrite(T message) {
boolean success = true;
boolean sent = true;
int timeout = 60;
try {
ChannelFuture cf = cxt.write(message);
cxt.flush();
if (sent) {
success = cf.await(timeout);
}
if (cf.isSuccess()) {
logger.debug("send success.");
}
Throwable cause = cf.cause();
if (cause != null) {
this.fireError(new PushException(cause));
}
} catch (LostConnectException e) {
this.fireError(new PushException(e));
} catch (Exception e) {
this.fireError(new PushException(e));
} catch (Throwable e) {
this.fireError(new PushException("Failed to send message“, e));
}
if (!success) {
this.fireError(new PushException("Failed to send message"));
}
}
I know Netty officials advise not to use sync() or await() method, but I want to know what situation will causes deadlocks in process and the current thread and executor thread of execution is the same.
I change my project code.
private void pushMessage0(T message) {
try {
ChannelFuture cf = cxt.writeAndFlush(message);
cf.addListener(new ChannelFutureListener() {
#Override
public void operationComplete(ChannelFuture future) throws PushException {
if (future.isSuccess()) {
logger.debug("send success.");
} else {
throw new PushException("Failed to send message.");
}
Throwable cause = future.cause();
if (cause != null) {
throw new PushException(cause);
}
}
});
} catch (LostConnectException e) {
this.fireError(new PushException(e));
} catch (Exception e) {
this.fireError(new PushException(e));
} catch (Throwable e) {
this.fireError(new PushException(e));
}
}
But I face a new problem, I can't get the pushException from the ChannelHandlerListener.
BlockingOperationException will be throw by netty if you call sync*or await* on a Future in the same thread that the EventExecutor is using and to which the Future is tied to. This is usually the EventLoop that is used by the Channel itself.
Can not call await in IO thread is understandable. However, there are 2 points.
1. If you call below code in channel handler, no exception will be reported, because the the most of the time the check of isDone in await returns true, since you are in IO thread, and IO thread is writing data synchronously. the data has been written when await is called.
ChannelPromise p = ctx.writeAndFlush(msg);
p.await()
If add a handler in different EventExecutorGroup, this check is not necessary, since that executor is newly created and is not the same one with the channel's IO executor.
I am trying to use Java's ExecutorService to send out concurrent REST requests which make various logs of system information (coming from a controller), however am running into a bug. About half of my requests successfully make it to the target, but the other half appear as if they are sent, but are not found on the server they were sent to. I think I may have a flaw in the logic of setting up my ExecutorService. The function log() shown below can be called from a REST call to the controller, and is supposed to create a new thread which sends out a separate HTTP request, and continue with the main thread so as not to wait for the network I/O. After much searching, I believe I have the ExecutorService shutdown properly an wait for the thread to complete. Can anybody see some type of error in the logic of my thread creation, as multiple requests from the controller can continue to come in?
//Controller
//code
#RequestMapping(value="/log", method= RequestMethod.GET)
public String log()
{
genomicsLogger.log(Severity.err, Category.LOG, "This is a log from the reporting manager!");
return "Hopefully logged";
}
//ClassB
public String log(String trns , String user, Severity severity, Category category, String msg) {
trnsField = trns;
userField = user;
...
...
...
ExecutorService executor = Executors.newFixedThreadPool(1);
Runnable task = () -> {
try {
System.out.println("Started thread: " + Thread.currentThread().getName());
restService.consumeRest(true, instance.getUri().toString(), LOG_URI, list, log, HttpMethod.POST, new HttpHeaders(), String.class);
System.out.println("SENT REST REQUEST");
} catch (URISyntaxException e) {
e.printStackTrace();
} catch (KeyStoreException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} catch (CertificateException e) {
e.printStackTrace();
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
} catch (UnrecoverableKeyException e) {
e.printStackTrace();
} catch (KeyManagementException e) {
e.printStackTrace();
}
}
};
executor.submit(task);
try {
System.out.println("attempt to shutdown executor");
executor.shutdown();
executor.awaitTermination(5, TimeUnit.SECONDS);
}
catch (InterruptedException e) {
System.err.println("tasks interrupted");
}
finally {
if (!executor.isTerminated()) {
System.err.println("cancel non-finished tasks");
}
executor.shutdownNow();
System.out.println("shutdown finished");
}
return "";
}
You are creating executor service with each log and killing it.
This is not the way to use it, it is meant to be reused, make it e.g. a field in this class and set some number of threads that you are willing to use for it (probably higher than 1). And don't do shutdown on it until you are really sure it won't be used (e.g. during application shutdown).
I use ScheduledExecutorService to schedule some tasks which need to run periodically.
I want to know whether this code works to recover the schedule when an exception happens.
ScheduledExecutorService service = Executors.newScheduledThreadPool(1);
this.startMemoryUpdateSchedule(service);//See below method
//Recursive method to handle exception when run schedule task
private void startMemoryUpdateSchedule(ScheduledExecutorService service) {
ScheduledFuture<?> future = service.scheduleWithFixedDelay(new MemoryUpdateThread(), 1, UPDATE_MEMORY_SCHEDULE, TimeUnit.MINUTES);
try {
future.get();
} catch (ExecutionException e) {
e.printStackTrace();
logger.error("Exception thrown for thread",e);
future.cancel(true);
this.startMemoryUpdateSchedule(service);
} catch(Exception e) {
logger.error("Other exception ",e);
}
}
You should probably enclose the try block in a while(true) loop because if the first run does not throw an exception, you will exit your method and if the second call throws one, you won't catch it.
I would also run the recursive call in its own thread to avoid the risk of a StackOverFlow error if things go bad.
So it would look like this:
private void startMemoryUpdateSchedule(final ScheduledExecutorService service) {
final ScheduledFuture<?> future = service.scheduleWithFixedDelay(new MemoryUpdateThread(), 1, UPDATE_MEMORY_SCHEDULE, TimeUnit.MINUTES);
Runnable watchdog = new Runnable() {
#Override
public void run() {
while (true) {
try {
future.get();
} catch (ExecutionException e) {
//handle it
startMemoryUpdateSchedule(service);
return;
} catch (InterruptedException e) {
//handle it
return;
}
}
}
};
new Thread(watchdog).start();
}
ScheduledExecutorService.scheduleWithFixedDelay(Runnable, long, long, TimeUnit) throws RejectedExecutionException (a child of RuntimeException) ==> We can catch it & retry submission once more.
Now as future.get() is supposed to return the result of one execution, we need to invoke it in a loop.
Also, the failure of one execution does not affect the next scheduled execution, which differentiates the ScheduledExecutorService from the TimerTask which executes the scheduled tasks in the same thread => failure in one execution would abort the schedule in case of TimerTask (http://stackoverflow.com/questions/409932/java-timer-vs-executorservice)
We just need to catch all the three exceptions thrown by Future.get(), but we can not rethrow them, then we won't be able to get the result of the subsequent executions.
The code could be:
public void startMemoryUpdateSchedule(final ScheduledExecutorService service) {
final ScheduledFuture<?> future;
try {
future = service.scheduleWithFixedDelay(new MemoryUpdateThread(),
1, UPDATE_MEMORY_SCHEDULE, TimeUnit.SECONDS);
} catch (RejectedExecutionException ree) {
startMemoryUpdateSchedule(service);
return;
}
while (true) {
try {
future.get();
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
} catch (ExecutionException ee) {
Throwable cause = ee.getCause();
// take action, log etc.
} catch (CancellationException e) {
// safety measure if task was cancelled by some external agent.
}
}
}
Try to use VerboseRunnable class from jcabi-log, which is designed exactly for this purpose:
import com.jcabi.log.VerboseRunnable;
Runnable runnable = new VerboseRunnable(
Runnable() {
public void run() {
// do business logic, may Exception occurs
}
},
true // it means that all exceptions will be swallowed and logged
);
Now, when anybody calls runnable.run() no exceptions are thrown. Instead, they are swallowed and logged (to SLF4J).
I've added the loop as discussed.
public void startMemoryUpdateSchedule(final ScheduledExecutorService service) {
boolean retry = false;
do {
ScheduledFuture<?> future = null;
try {
retry = false;
future = service.scheduleWithFixedDelay(new MemoryUpdateThread(), 1, UPDATE_MEMORY_SCHEDULE, TimeUnit.SECONDS);
future.get();
} catch (ExecutionException e) {
// handle
future.cancel(true);
retry = true;
} catch(Exception e) {
// handle
}
} while (retry);
}