Having a hard time to do a reliable retry of a background task which sends request to let's say mail service in order to get latest emails. Once emails successfully received the execution should continue in thenAccept() block - persist emails, however if exception occurs I have to rerun mail retrieval until successful attempt and on success should persist mails and stop. Please take a look and advice if I do it wrong.
private void retrieveMailsAsync(User user) {
CompletableFuture.supplyAsync(() -> {
try {
return mailService.getEmails(user.getName(), user.getPassword());
} catch (InvalidAuthentication | TimeoutException | BadGatewayException e) {
throw new CompletionException(e);
}
}).thenAccept(email -> {
mailService.persist(email);
}).exceptionally(ex -> {
log.log(Level.SEVERE, "Exception retrieveMailsAsync emails, Retrying retrieveMailsAsync:: ", ex.getCause());
retrieveMailsAsync(user);
return null;
});
}
P.S please also take a look at how I'm handling checked exception wrapping it into CompletionException and rethrowing - the main idea here to handle all exceptions (defined checked and runtime) in one exceptionally() block rather than logging them in catch block and return null.
Thanks guys in advance, hope I'm not doing pretty stupid stuff, or at least there is already reliable solutions exists for Java 8.
What I meant in my comment was this:
private void retrieveMailsAsync(User user) {
CompletableFuture.supplyAsync(() -> {
while (continueQuery()) { // true for infinite retries, or some other logic
try {
return mailService.getEmails(user.getName(), user.getPassword());
} catch (InvalidAuthentication | TimeoutException | BadGatewayException e) {
log.log(Level.SEVERE, "Exception retrieveMailsAsync emails, Retrying retrieveMailsAsync: ", e);
}
}
return null;
}).thenAccept(email -> {
mailService.persist(email);
});
}
Ie you just retry in the submitted runnable until you don't get an exception anymore.
I think you can achieve that via :
public static void main(String[] args) {
String result = call(new User().setName("name").setPassword("p")).join();
System.out.println(result);
}
private static CompletableFuture<String> call(User user) {
CompletableFuture<String> future = CompletableFuture.supplyAsync(() -> retrieveMailsAsync(user));
return future.handleAsync((String result, Throwable ex) -> {
// or any other Predicate that is satisfied against ex
if(ex != null) {
return call(user);
} else {
return future;
}
}).thenCompose(Function.identity());
}
EDIT
So what stays in your way to change the code above, for example, to:
static ExecutorService service = Executors.newFixedThreadPool(1);
public static void main(String[] args) {
call(new User().setName("name").setPassword("p"))
// chain any other action here, like mailService.persist(email);
.thenAcceptAsync(
System.out::println,
service
);
System.out.println("Continue main thread");
}
private static CompletableFuture<String> call(User user) {
CompletableFuture<String> future = CompletableFuture.supplyAsync(() -> retrieveMailsAsync(user), service);
return future.handleAsync((String result, Throwable ex) -> {
// or any other Predicate that is satisfied against ex
if(ex != null) {
return call(user);
} else {
return future;
}
}).thenCompose(Function.identity());
}
Related
My situation
I'm trying to craft a functionality which would execute n (where n >=0) requests to a given endpoint, but I do understand that sometimes that endpoint might not respond due to
500 error or other issue, so I want to repeat my requests to an endpoint (with a
small interval in between [not yet implemented]) till I get a response, or till I get an unknown error which would indicate what I can't repeat, because of other reasons than a crashed server.
So, I've tried to implement this piece of functionality using Executors and concurrency provided by Java 11 and it does not work as I want
I can't resubmit failed tasks till I get all the responses and I don't know why
I have a method
private void DoMyTasks(List<MyRequest> requests) {
final ExecutorService executorService = Executors.newFixedThreadPool(10);
final ExecutorCompletionService<MyReqResDto> completionService =
new ExecutorCompletionService<>(executorService);
for (final MyRequest MyRequest : requests) {
completionService.submit(new MyCallableRequest(webClient, MyRequest));
}
List<MyReqResDto> responses = new ArrayList<>();
for (int i = 0; i < requests.size(); ++i) {
try {
final Future<MyReqResDto> future = completionService.take();
if (future.get().getEx() != null) {
completionService.submit(new MyCallableRequest(webClient, future.get().getMyRequest()));
}
responses.add(future.get());
} catch (ExecutionException | InterruptedException e) {
log.warn("Error"));
} catch (Exception exception) {
log.error("Other error");
} finally {
executorService.shutdown();
try {
if (!executorService.awaitTermination(10, TimeUnit.MINUTES)) {
executorService.shutdownNow();
}
} catch (InterruptedException e) {
executorService.shutdownNow();
}
}
}
responses.size();
}
I'm trying to repeat failed tasks with
if (future.get().getEx() != null) {
completionService.submit(new MyCallableRequest(webClient, future.get().getMyRequest()));
}
and yet, at the end of execution I don't get all responses for my requests. What I get is at most 3 to 5 responses when I try executing 10 requests. Why? How to fix it?
My callable class is
public class MyCallableRequest implements Callable<MyReqResDto> {
private final WebClient webClient;
private final MyRequest myRequest;
public MyCallableRequest(WebClient webClient, MyRequest myRequest) {
this.webClient = webClient;
this.myRequest = myRequest;
}
#Override
public MyReqResDto call() throws Exception {
try {
if (new Random().nextInt(10) % 2 == 0) {
throw new TestException();
}
if (new Random().nextInt(10) % 7 == 0) {
throw new RuntimeException();
}
WebClient.UriSpec<WebClient.RequestBodySpec> uriSpec = webClient.post();
WebClient.RequestBodySpec bodySpec = uriSpec.uri(
s -> s.path("/myEndpoint").build());
MyRequestDto myMyRequestDto = new MyRequestDto();
WebClient.RequestHeadersSpec<?> headersSpec =
bodySpec.body(Mono.just(myMyRequestDto), MyRequestDto.class);
ResponseDto responseDto = headersSpec.exchangeToMono(s -> {
if (s.statusCode().equals(HttpStatus.OK)) {
return s.bodyToMono(ResponseDto.class);
} else if (s.statusCode().is1xxInformational()) {
return s.createException().flatMap(Mono::error);
} else if (s.statusCode().is3xxRedirection()) {
return s.createException().flatMap(Mono::error);
} else if (s.statusCode().is4xxClientError()) {
return s.createException().flatMap(Mono::error);
} else if (s.statusCode().is5xxServerError()) {
return s.createException().flatMap(Mono::error);
} else {
return s.createException().flatMap(Mono::error);
}
//return null;
}).block();
return new MyReqResDto(myRequest, responseDto, null);
} catch (Exception exception) {
return new MyReqResDto(myRequest, null, exception);
}
}
}
Update NO. 1
I changed a for loop to a while loop according to a comment provided by
Slaw and an answer provided by erickson. And this solutions works, meaning that
it is hammering an endpoint till all responses are received without
any errors. But I'm still not sure it feels that I'm building a sh**
tower with this solution. Is there any thread related issues that I should be aware while using executor like this?
while (true) {
Future < MyReqResDto > future = null;
try {
future = completionService.take();
if (future.get().getEx() != null /*and check exception if possible to handle, if not break from a loop*/) {
completionService.submit(new MyCallableRequest(webClient, future.get().getRequestCT());
} else {
responseDtos.add(future.get());
}
} catch (ExecutionException | InterruptedException e) {
log.warn("Error while downloading", e.getCause());
// test if I can recover from these exceptions if no
break;
}
}
if (responseDtos.size() == requests.size()) {
executorService.shutdown();
try {
if (!executorService.awaitTermination(10, TimeUnit.MINUTES)) {
executorService.shutdownNow();
}
} catch (InterruptedException e) {
executorService.shutdownNow();
}
break;
}
You are shutting down the executor as soon as you get one response. Perhaps a few more have completed in this time, but you are not allowing time for any others to complete.
Your logic here is wrong. The executor should only be shut down when you are sure no more tasks will be submitted; at soonest, that is after the loop responsible for re-submitting failures.
Here is a simplified view of your code to highlight the premature shutdown:
for (int i = 0; i < requests.size(); ++i) {
try {
final Future<MyReqResDto> future = completionService.take();
...
responses.add(future.get());
...
} finally {
executorService.shutdown();
}
}
What is the correct way to handle CompletableFuture exception in this situation?
I have a piece of code that return a CompletableFuture, but it has a potential of completeExceptionally.
I want to chain CompletableFuture, but I don't want it to execute the next method if it receive an error.
And I do not want to recover from the error either.
So my solution is throw new CompletionException(error).
public static void main(String[] args) {
calculateThings()
.handle((number, error) -> {
if(error != null) {
throw new CompletionException(error);
}
// ...
})
.thenApply(number -> {
// i don't want this part to execute when error happen!
System.out.println("should not execute");
return 100;
})
.whenComplete((result, error) -> {
if(error != null) {
System.out.println("output this line!"); // <-- output
}
});
}
public static CompletableFuture<Integer> calculateThings() {
CompletableFuture<Integer> future = new CompletableFuture<>();
CompletableFuture.runAsync(() -> {
future.completeExceptionally(new IllegalStateException());
});
return future;
}
Is this correct? I feel like this isn't a right thing to do.
I see a couple of example return failedStage or completedStage but I don't know how to unwrap the object.
Thanks!
I have 3 methods that I need to run in parallel, since they are independent to each other and combine the results of each one at the end and send it as the response. I need to handle exception as well.
In different post I found the below code and modified accordingly.
public Response getResponse() {
Response resultClass = new Response();
try {
CompletableFuture<Optional<ClassA>> classAFuture
= CompletableFuture.supplyAsync(() -> service.getClassA() );
CompletableFuture<ClassB> classBFuture
= CompletableFuture.supplyAsync(() -> {
try {
return service.getClassB();
}
catch (Exception e) {
throw new CompletionException(e);
}
});
CompletableFuture<ClassC> classCFuture
= CompletableFuture.supplyAsync(() -> {
try {
return service.getClassC();
} catch (Exception e) {
throw new CompletionException(e);
}
});
CompletableFuture<Response> responseFuture =
CompletableFuture.allOf(classAFuture, classBFuture, classCFuture)
.thenApplyAsync(dummy -> {
if (classAFuture.join().isPresent() {
ClassA classA = classAFuture.join();
classA.setClassB(classBFuture.join());
classA.setClassC(classCFuture.join());
response.setClassA(classA)
}
return response;
});
responseFuture.join();
} catch (CompletionExecution e) {
throw e;
}
return response;
}
Should the above run correctly in parallel? I see it takes some more time, and I wanted to make sure I am doing it right.
If you want to run methods in parallel you should use ExecutorService. Try something like that:
ExecutorService myExecutor = Executors.newFixedThreadPool(3);
List<Future<Object>> futures = myExecutor.invokeAll(
Arrays.asList(
() -> service.getClassA(),
() -> service.getClassB(),
() -> service.getClassC(),
)
);
myExecutor.shutdown();
The idea is correct, but this all could be done with a lot less code:
public Response getResponse() {
CompletableFuture<Optional<ClassA>> classAFuture = CompletableFuture.supplyAsync(() -> service.getClassA());
CompletableFuture<ClassB> classBFuture = CompletableFuture.supplyAsync(() -> service.getClassB());
CompletableFuture<ClassC> classCFuture = CompletableFuture.supplyAsync(() -> service.getClassC());
try {
return CompletableFuture.allOf(classAFuture, classBFuture, classCFuture)
.thenApply(() -> {
Response response = new Response();
Optional<ClassA> maybeA = classAFuture.get();
if (maybeA.isPresent()) {
ClassA classA = maybeA.get();
classA.setClassB(classBFuture.get());
classA.setClassC(classCFuture.get());
response.setClassA(classA);
}
return response;
}).get();
} catch (ExecutionException e) { // Ususally the exception is wrapped to ExecutionException by java concurrency framework itself
Throwable cause = e.getCause();
if (cause != null) {
throw cause;
} else {
throw e;
}
}
}
Main things:
You don't need to wrap your exceptions to CompletionException.
You don't need to use thenApplyAsync. Just thenApply is the same thing unless you want to be very specific on the type of thread you want to use. Check this for more information https://stackoverflow.com/a/47489654/3020903
You don't need to join() anything. By the time CompletableFuture.all has finished, you can be very sure that all the supplied jobs have finished and calling get() on then will just return the value.
As for can you be sure jobs A, B and C will be run in parallel. Yes and no. It will be run in parallel if there are enough system resources to run them in parallel. You have done your best to ask them to run in parallel. Maybe at some point you also want to supply your custom thread pool to have more control, but that's a topic for another day.
I'm using completable futures to do a bunch of thing X. X talks to the internet and can either fail or not fail. When I invoke X I pass it a value, let's call it value. X(value).
private void X(String value) {
CompletableFuture<Boolean> future = CompletableFuture.supplyAsync(()-> {
try {
Object response = talkToExternalThing(value);
} catch (InterruptedException e) {
throw new CompletionException(e.getCause());
}
return true;
}).exceptionally(ex -> false);
futures.add(future);
}
Above is a snippet of what I'm playing with. When it comes down to analyzing the result-set, I can see all values that failed/didn't fail in my test (ie. true or false).
Map<Boolean, List<CompletableFuture<Boolean>>> result = futures.stream()
.collect(Collectors.partitioningBy(CompletableFuture::isCompletedExceptionally));
My problem is, I want to not only know if it failed or did not fail, but I also want other metadata, such as the value that caused the failure. My hope is to potentially have an exception object I can analyze as a result. It's worth noting that the exception is a checked exception (interrupt).
This would be my suggestion:
ExecutorService executorService = Executors.newCachedThreadPool();
private void X(String value) {
CompletableFuture<Pair<Boolean, String>> future = new CompletableFuture<>();
executorService.execute(() -> {
try {
Object response = talkToExternalThing(value);
} catch (InterruptedException e) {
// un-successfully, with the value
future.complete(new Pair<>(false, value));
return;
}
// successfully, with the value
future.complete(new Pair<>(true, value));
});
futures.add(future);
}
I have the following code:
// How to throw the ServerException?
public void myFunc() throws ServerException{
// Some code
CompletableFuture<A> a = CompletableFuture.supplyAsync(() -> {
try {
return someObj.someFunc();
} catch(ServerException ex) {
// throw ex; gives an error here.
}
}));
// Some code
}
someFunc() throws a ServerException. I don't want to handle this here but throw the exception from someFunc() to caller of myFunc().
Your code suggests that you are using the result of the asynchronous operation later in the same method, so you’ll have to deal with CompletionException anyway, so one way to deal with it, is
public void myFunc() throws ServerException {
// Some code
CompletableFuture<A> a = CompletableFuture.supplyAsync(() -> {
try { return someObj.someFunc(); }
catch(ServerException ex) { throw new CompletionException(ex); }
});
// Some code running in parallel to someFunc()
A resultOfA;
try {
resultOfA = a.join();
}
catch(CompletionException ex) {
try {
throw ex.getCause();
}
catch(Error|RuntimeException|ServerException possible) {
throw possible;
}
catch(Throwable impossible) {
throw new AssertionError(impossible);
}
}
// some code using resultOfA
}
All exceptions thrown inside the asynchronous processing of the Supplier will get wrapped into a CompletionException when calling join, except the ServerException we have already wrapped in a CompletionException.
When we re-throw the cause of the CompletionException, we may face unchecked exceptions, i.e. subclasses of Error or RuntimeException, or our custom checked exception ServerException. The code above handles all of them with a multi-catch which will re-throw them. Since the declared return type of getCause() is Throwable, the compiler requires us to handle that type despite we already handled all possible types. The straight-forward solution is to throw this actually impossible throwable wrapped in an AssertionError.
Alternatively, we could use an alternative result future for our custom exception:
public void myFunc() throws ServerException {
// Some code
CompletableFuture<ServerException> exception = new CompletableFuture<>();
CompletableFuture<A> a = CompletableFuture.supplyAsync(() -> {
try { return someObj.someFunc(); }
catch(ServerException ex) {
exception.complete(ex);
throw new CompletionException(ex);
}
});
// Some code running in parallel to someFunc()
A resultOfA;
try {
resultOfA = a.join();
}
catch(CompletionException ex) {
if(exception.isDone()) throw exception.join();
throw ex;
}
// some code using resultOfA
}
This solution will re-throw all “unexpected” throwables in their wrapped form, but only throw the custom ServerException in its original form passed via the exception future. Note that we have to ensure that a has been completed (like calling join() first), before we query the exception future, to avoid race conditions.
For those looking for other ways on exception handling with completableFuture
Below are several ways for example handling Parsing Error to Integer:
1. Using handle method - which enables you to provide a default value on exception
CompletableFuture correctHandler = CompletableFuture.supplyAsync(() -> "A")
.thenApply(Integer::parseInt)
.handle((result, ex) -> {
if (null != ex) {
ex.printStackTrace();
return 0;
} else {
System.out.println("HANDLING " + result);
return result;
}
})
.thenAcceptAsync(s -> {
System.out.println("CORRECT: " + s);
});
2. Using exceptionally Method - similar to handle but less verbose
CompletableFuture parser = CompletableFuture.supplyAsync(() -> "1")
.thenApply(Integer::parseInt)
.exceptionally(t -> {
t.printStackTrace();
return 0;
}).thenAcceptAsync(s -> System.out.println("CORRECT value: " + s));
3. Using whenComplete Method - using this will stop the method on its tracks and not execute the next thenAcceptAsync
CompletableFuture correctHandler2 = CompletableFuture.supplyAsync(() -> "A")
.thenApply(Integer::parseInt)
.whenComplete((result, ex) -> {
if (null != ex) {
ex.printStackTrace();
}
})
.thenAcceptAsync(s -> {
System.out.println("When Complete: " + s);
});
4. Propagating the exception via completeExceptionally
public static CompletableFuture<Integer> converter(String convertMe) {
CompletableFuture<Integer> future = new CompletableFuture<>();
try {
future.complete(Integer.parseInt(convertMe));
} catch (Exception ex) {
future.completeExceptionally(ex);
}
return future;
}
Even if other's answer is very nice. but I give you another way to throw a checked exception in CompletableFuture.
IF you don't want to invoke a CompletableFuture in another thread, you can use an anonymous class to handle it like this:
CompletableFuture<A> a = new CompletableFuture<A>() {{
try {
complete(someObj.someFunc());
} catch (ServerException ex) {
completeExceptionally(ex);
}
}};
IF you want to invoke a CompletableFuture in another thread, you also can use an anonymous class to handle it, but run method by runAsync:
CompletableFuture<A> a = new CompletableFuture<A>() {{
CompletableFuture.runAsync(() -> {
try {
complete(someObj.someFunc());
} catch (ServerException ex) {
completeExceptionally(ex);
}
});
}};
I think that you should wrap that into a RuntimeException and throw that:
throw new RuntimeException(ex);
Or many be a small utility would help:
static class Wrapper extends RuntimeException {
private Wrapper(Throwable throwable) {
super(throwable);
}
public static Wrapper wrap(Throwable throwable) {
return new Wrapper(throwable);
}
public Throwable unwrap() {
return getCause();
}
}
public static void go() {
CompletableFuture<String> a = CompletableFuture.supplyAsync(() -> {
try {
throw new Exception("Just because");
} catch (Exception ex) {
throw Wrapper.wrap(ex);
}
});
a.join();
}
And then you could unwrap that..
try {
go();
} catch (Wrapper w) {
throw w.unwrap();
}