I am trying to upload to S3 within my asynchronous Java code
private void submitCallablesWithExecutor()
throws InterruptedException, ExecutionException, TimeoutException {
ExecutorService executorService = null;
try {
executorService = Executors.newCachedThreadPool();
Future<String> task1Future = executorService.submit(new Callable<String>() {
public String call() {
try {
processExportRequest(xmlPutRequest_, customizedRequest_, response_);
return "Success";
} catch (Exception ex) {
return ex.getMessage();
}
}
});
} finally {
executorService.shutdown();
try {
if (!executorService.awaitTermination(800, TimeUnit.MILLISECONDS)) {
executorService.shutdownNow();
}
} catch (InterruptedException e) {
executorService.shutdownNow();
}
}
}
within processExportRequest I am calling upload to S3. I have tried both S3Client and S3AsyncClient. In both cases, I am getting following error:
Failed to upload to S3: java.lang.IllegalStateException: Interrupted waiting to refresh the value.
I don't see anywhere in my code that's calling Thread.interrupt(), and everything else seems to work fine, just not S3 upload. Maybe the multithreaded nature of Java Future is not compatible with AWS SDK? Thanks.
I changed Future to CompletableFuture, and combine two of them (in sequence):
private CompletableFuture<PutObjectResponse> processExportAndUploadAsync()
throws IOException {
CompletableFuture<PutObjectResponse> result = processExportAsync()
.thenCompose(fileName -> uploadS3Async(fileName));
return result;
}
It seems to work.
Related
After using this as a guide I'm trying to send a FileInputStream to two APIs concurrently, if either of them fails we error out.
I have two callables
private void submitCallablesWithExecutor(final FileInputStream content)
throws InterruptedException, ExecutionException, TimeoutException {
ExecutorService executorService = null;
try {
executorService = Executors.newFixedThreadPool(5);
//send to api1
final Future<String> futureOne = executorService.submit(new Callable<String>() {
#Override
public String call() throws Exception {
final String returnedValue = implOne.storeContent(content)
return returnedValue;
}
});
//send to api2
final Future<String> futureTwo = executorService.submit(new Callable<String>() {
#Override
public String call() throws Exception {
final String returnedValue = implTwo.storeContent(content);
return returnedValue;
}
});
final String valueOne = futureOne.get(4, TimeUnit.SECONDS);
final String valueTwo = futureTwo.get(4, TimeUnit.SECONDS);
} finally {
executorService.shutdown();
}
}
within api2s implementation it tries to calculate the filesize with the below method
protected Long calculateFilesize(InputStream data) throws StoreException {
try {
return (long) data.available();
} catch (IOException e) {
LOGGER.error("Error determining filesize of InputStream");
throw new StoreException( e);
}
}
I get the following error during data.available()
java.io.IOException: Stream Closed
I suspect at some point the stream is being closed and at this point it fails to read it.
Any advice would be highly appreciated.
It's worth noting that the method that calls submitCallablesWithExecutor(content) is surrounded with a try with resource. I wonder if it's auto closing the Stream?
try (final FileInputStream content = new FileInputStream(tmpFile)) {
submitCallablesWithExecutor(content)
}
It hard to tell without seeing more code of the tasks. This is one possible scenario for the closed stream. The javadoc for executorService.shutdown(); says:
This method does not wait for previously submitted tasks to complete execution.
If your first task take longer than 4 seconds to run you will get TimeoutException when calling futureOne.get(4, TimeUnit.SECONDS) which you don't catch so it leads to your own try with resource closing the FileInputStream. However the second callable might still be running in the executor service so could get this java.io.IOException: Stream Closed exception.
You could fix by using future.get() which blocks on first task completion, but it sounds like you don't want this, so it's better to cleanly shut down your service thread.
Try adding executorService.awaitTermination(long timeout, TimeUnit unit) after shutdown() and catch the TimeoutException and handle appropriately to ensure the second task exits cleanly in this timeout/error situation, such as using latch or flags to notify to end the task.
I am trying to get the result from my CompletableFuture based on different timeouts. The second call to the CompletableFuture would ideally be fire and forget.
Example:
CompletableFuture<String> future = CompletableFuture.supplyAsync(() -> {
try {
TimeUnit.Seconds.sleep(10);
return "success"
} catch (InterruptedException e) {
return "fail";
}
}
public String method(CompletableFuture<String> future) {
try {
return future.get(1, TimeUnit.SECONDS);
} catch (Exception e) {
ExecutorService.execute(() -> {
String result = future.get(10, TimeUnit.SECONDS);
}
}
return "Initial Future timed out"
}
I still would want the Initial Future timed out string to be returned, just another thread to reprocess in a fire and forget way.
Addionally, is there a cleaner way of doing something like this. Maybe being able to wrap it all inside the initial CompletableFuture?
I'm trying to send async transaction to my Fabric network using the java gateway sdk but i receive the error Channel [CHANNEL NAME] has been shutdown.
Here some example code:
Gateway.Builder builder = Gateway.createBuilder()
.discovery(true)
.identity(wallet, user.getName())
.networkConfig([PATH TO CONNECTION PROFILE]);
try(Gateway gateway = builder.connect()) {
Network channel = gateway.getNetwork(CHANNEL_NAME);
Contract someChaincode = channel.getContract(CHAINCODE_NAME);
int coresNumber = (Runtime.getRuntime().availableProcessors());
ExecutorService executor = Executors.newFixedThreadPool(coresNumber);
for(String elemt : elements) {
CompletableFuture<String> future = CompletableFuture.supplyAsync(() -> {
try{
//Exception thrown here
byte[] res = someChaincode.submitTransaction("someFunction", elemt);
return new String(res);
} catch (ContractException e) {
e.printStackTrace();
}
}, executor);
}
} catch (Exception e) {
// Handle Exception
}
And here the exception:
java.util.concurrent.ExecutionException: org.hyperledger.fabric.gateway.GatewayRuntimeException: org.hyperledger.fabric.sdk.exception.InvalidArgumentException: Channel [CHANNEL NAME] has been shutdown.
Precisely, the exception is thrown in the method checkChannelState(). I have a sense that I'm not handling multithreading correctly.
You don't look to be waiting for completion of the futures you have created in your code snippet. So you are scheduling transaction invocations for execution on different threads but then, before this code is executed, dropping out of a try-with-resources block which closes the Gateway instance you are using to connect. Closing the Gateway causes all the associated resources and connections to be closed, including the underlying Channel. So when your transaction invocations actually get run, you have already closed the connection and resources needed for them to execute.
You need to get the results from the Future objects you have created before closing the Gateway instance; in other words, before dropping out of the try-with-resources block that creates the Gateway. Something vaguely like this:
Collection<Callable<String>> tasks = elements.stream()
.map(element -> new Callable<String>() {
public String call() throws ContractException, TimeoutException, InterruptedException {
byte[] result = contract.submitTransaction("someFunction", element);
return new String(result);
}
}).collect(Collectors.toList());
try {
Collection<String> results = new ArrayList<>();
Collection<Future<String>> futures = executor.invokeAll(tasks, timeout, timeUnit);
for (Future<String> future : futures) {
try {
String result = future.get(timeout, timeUnit);
results.add(result);
} catch (CancellationException | InterruptedException | ExecutionException | TimeoutException e) {
e.printStackTrace();
}
}
System.out.println("Results: " + results);
} catch (InterruptedException e ) {
e.printStackTrace();
}
Recently, I find some BlockingOperationException in my netty4 project.
Some people said that when using the sync() method of start netty's ServerBootstrap can cause dead lock, because sync() will invoke await() method, and there is a method called 'checkDeadLock' in await().
But I don't think so. ServerBootstrap use the EventLoopGroup called boosGroup, and Channel use the workerGroup to operation IO, I don't think they will influence each other, they have different EventExecutor.
And in my practice, Deadlock exception doesn't appear in the Netty startup process, most of which occurs after the Channel of the await writeAndFlush.
Analysis source code, checkDeadLock, BlockingOperationException exception thrown is when the current thread and executor thread of execution is the same.
My project code is blow:
private void channelWrite(T message) {
boolean success = true;
boolean sent = true;
int timeout = 60;
try {
ChannelFuture cf = cxt.write(message);
cxt.flush();
if (sent) {
success = cf.await(timeout);
}
if (cf.isSuccess()) {
logger.debug("send success.");
}
Throwable cause = cf.cause();
if (cause != null) {
this.fireError(new PushException(cause));
}
} catch (LostConnectException e) {
this.fireError(new PushException(e));
} catch (Exception e) {
this.fireError(new PushException(e));
} catch (Throwable e) {
this.fireError(new PushException("Failed to send message“, e));
}
if (!success) {
this.fireError(new PushException("Failed to send message"));
}
}
I know Netty officials advise not to use sync() or await() method, but I want to know what situation will causes deadlocks in process and the current thread and executor thread of execution is the same.
I change my project code.
private void pushMessage0(T message) {
try {
ChannelFuture cf = cxt.writeAndFlush(message);
cf.addListener(new ChannelFutureListener() {
#Override
public void operationComplete(ChannelFuture future) throws PushException {
if (future.isSuccess()) {
logger.debug("send success.");
} else {
throw new PushException("Failed to send message.");
}
Throwable cause = future.cause();
if (cause != null) {
throw new PushException(cause);
}
}
});
} catch (LostConnectException e) {
this.fireError(new PushException(e));
} catch (Exception e) {
this.fireError(new PushException(e));
} catch (Throwable e) {
this.fireError(new PushException(e));
}
}
But I face a new problem, I can't get the pushException from the ChannelHandlerListener.
BlockingOperationException will be throw by netty if you call sync*or await* on a Future in the same thread that the EventExecutor is using and to which the Future is tied to. This is usually the EventLoop that is used by the Channel itself.
Can not call await in IO thread is understandable. However, there are 2 points.
1. If you call below code in channel handler, no exception will be reported, because the the most of the time the check of isDone in await returns true, since you are in IO thread, and IO thread is writing data synchronously. the data has been written when await is called.
ChannelPromise p = ctx.writeAndFlush(msg);
p.await()
If add a handler in different EventExecutorGroup, this check is not necessary, since that executor is newly created and is not the same one with the channel's IO executor.
Iam attempting to use ExecutorService to run some code to connect my client to a server. Obviously Iam trying to get this to run on a seperate thread to the UI Thread, But my UI freezes when the code is executing.Which is not what I was excpecting. I thought ExecutorService would run on a seperate thread? below is my code
#Override
public void registerDevice() {
ExecutorService exservice = Executors.newFixedThreadPool(10);
Future<Boolean> future = exservice.submit(new Callable() {
#Override
public Boolean call() throws Exception {
android.os.Process.setThreadPriority(Process.THREAD_PRIORITY_BACKGROUND);
try {
Thread.sleep(20000);
} catch (InterruptedException e) {
e.printStackTrace();
}
reghandler.post(new Runnable() {
#Override
public void run() {
regpresenter.updateUIProgress();
}
});
return true;
}
});
exservice.shutdown();
try {
Boolean done = future.get(10, TimeUnit.SECONDS);
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
} catch (TimeoutException e) {
e.printStackTrace();
warningstr = "Server call timed out!";
}
exservice.shutDownNow();
}
If anyone can tell my why this doesnt run seperate to the UI thread I would be grateful. Maybe there is something Iam missing after reading all the docs on executor service. also I have another solution that seems to work but it creates a circular dependency with a timertask within my run method of another thread. The timertask interrupts the thread after timeout, if The thread is alive. so I dont really want to use that solution. Also iam just testing the timeout works at present. Ive left out some of the code in the method as its not really relevant to my issue. Thanks again
Below code means to wait to task result (maximum with 10 seconds timeout)
Boolean done = future.get(10, TimeUnit.SECONDS);
if you don't want to wait till task is completed, don't call get().