I want to send a ping for up to 10 users at the same time, and update the user object with the result once the ping is done.
In order to do this, I am trying to use ExecutorService.
I started with a code like this:
private void pingUsers(List<User> userList) throws ExecutionException, InterruptedException {
final int NUM_THREADS = 10;
ExecutorService executor = Executors.newFixedThreadPool(NUM_THREADS);
for (User user : userList) {
SnmpPingDevice pingUser = new PingUser(user);
Future<Boolean> isUserActive = executor.submit(pingUser);
user.isActive = isUserActive.get() ; // -- I guess it will block other pings and i'm back to my starting point where I need to run the pings in parallel.
}
executor.shutdown();
try {
executor.awaitTermination(30, TimeUnit.SECONDS);
} catch (InterruptedException e) {
logger.error("Failed to terminate executor");
}
}
This is how my PingUser class look like:
#Override
public Boolean call() {
ping = new CmdRunner(toolDir,outDir,
new UserOidWorkerPing(version,community,ip,logger));
return this.isActive();
}
public boolean isActive(){
String cmd = ping.getCmdNoRedirect();
String rc = this.cmdRunner.runShellCmd(cmd,this.outDir +"/dummy",false);
logger.debug("PING was sent with cmd:" + cmd + ",rc:" + rc);
return rc != null && !rc.contains("Timeout:") && !rc.isEmpty();
}
And back to the same issue, that the pings won't run in parallel (as soon as the loop waiting for the isUserActive.get() to end)
Any idea what I'm missing? How I can make those pings run in parallel and save the result for each user in my List<User> userList?
Future::get is a blocking operation so the invoking thread will be blocked until the call is completed. So you submit a new task only after the previous was finished.
Consider using ExecutorService::invokeAll which will return a list of Futures :
List<PingUser> pingUsers = userList.stream().map(PingUser::new).collect(Collectors.toList());
List<Future<Boolean>> results = executor.invokeAll(pingUsers);
You are blocking your execution for each call, with this line:
user.isActive = isUserActive.get() ;
This effectively waits for the call to end, and does this for each call, on at a time.
You should rather submit all tasks, and build a list of Futures, to only wait for results when all tasks have been submitted. Something like this:
List<Future<Boolean>> tasks = new ArrayList<>();
for (User user : userList) {
SnmpPingDevice pingUser = new PingUser(user);
tasks.add(executor.submit(pingUser));
}
for(Future<Boolean> task: tasks) {
//use the result... OK to get() here.
}
What you can do is add the user.isActive = future.get() into the Runnable you submit.
for (User user : userList) {
SnmpPingDevice pingUser = new PingUser(user);
executor.submit(() -> user.isActive = pingUser.call());
}
Related
We have a synchronous process that needs to call two REST endpoints, whereas the result of the first is needed for the second. Using Springs WebClient the .block() causes the following exception:
java.lang.IllegalStateException: block()/blockFirst()/blockLast() are blocking, which is not supported in thread parallel-2
How can this be prevented?
Here is a simplified code snippet:
var job = webClient.createJob().block();
if (job == null || StringUtils.isBlank(job.getId())) {
throw new Exception("WebClient did not return with a job id");
}
batchRecords(job.getId(), records);// does some additional calls to the webClient
This works in the unit test, but when called through a #RestController the above exception is thrown.
EDIT:
The batchRecords method currently also has blocking Monos in it, so we can have a delay in between:
public void batchRecords(final String jobId, final List<InventoryRecord> records)
var recordCount = 0;
var inventoryPositions = new ArrayList<InventoryPosition>();
var recordIterator = records.iterator();
while (recordIterator != null && recordIterator.hasNext()) {
var inventoryRecord = recordIterator.next();
inventoryPositions.add(mapInventoryPosition(inventoryRecord));
recordCount++;
if (inventoryPositions.size() == batchSize) {
var response = createBatch(jobId, inventoryPositions);
Thread.sleep(sleepTime);
response.block();
inventoryPositions = new ArrayList<>();
}
}
}
You should do it reactively without blocking:
webClient.createJob()
.filter(job -> !StringUtils.isBlank(job.getId()))
.flatMap(job -> batchRecords(job.getId(), records))
.switchIfEmpty(Mono.error(new Exception("WebClient did not return with a job id")));
As soon as the createJob operation is finished, the result is filtered and provided to the flatMap operator. In case of an empty response (Mono.empty()) an exception is thrown.
I want to run many GET and POST. In this example I use GET and for every task the same URL, but later it will be always a different URL for each task.
What I find that the time increases as the number of task and threads used.
num = 1 -> Done in 1846
num = 10 -> Done in 2114
num = 100 -> Done in 7204
num = 200 -> Done in 13720
If I have just 1 task I use 1 thread. If 10 tasks I use 10 threads, and so on.
I don't understand the time increase. If time for 1 task executed with 1 thread would take approx. 1 second, then for 10 tasks executed with 10 threads I would expect about the same time of 1 sec. Because on my 4-core CPU I can executed many threads concurrently.
Is it possibly that because I have only 1 network device, the requests don't get send in parallel but somehow in sequence?
// Amount of task and threads
int num = 10;
// Create many instances of the task
List<MyCallable> tasks = new ArrayList<>();
// Create num instances of MyCallable
ExecutorService executor = Executors.newFixedThreadPool(num);
List<Future<Void>> invokeAll = null;
long started = System.currentTimeMillis();
try {
invokeAll = executor.invokeAll(tasks);
} catch (InterruptedException ex) {
}
long ended = System.currentTimeMillis();
System.out.println("Done in " + (ended - started));
executor.shutdown();
private class MyCallable implements Callable<Void> {
public MyCallable() {}
#Override
public Void call() throws Exception {
int statusCode = sendGet();
return null;
}
private int sendGet() throws Exception {
CloseableHttpClient closeableHttpClient = HttpClients.createDefault();
CloseableHttpResponse closeableHttpResponse = closeableHttpClient.execute(new HttpGet("https://bing.com")); // https://www.google.com
int statusCode = closeableHttpResponse.getStatusLine().getStatusCode();
closeableHttpClient.close();
return statusCode;
}
}
I am trying to add 10 records to different server with the help of thread pool and waiting for response to get the information added on server side, but when I execute the code at my end I am getting same response for few threads, but data stored at server end are unique values per record.
ExecutorService threadPool = Executors.newFixedThreadPool(10);
List<Future<Employee>> listOfEmployee = new ArrayList<Future<Employee>>();
List<Employee> employeeList = new ArrayList<Employee>();
for(int i=0;i<10;i++)
{
listOfEmployee.add(threadPool.submit(new Callable<Employee>() {
#Override
public Employee call()
{
return employee.add();
}
}));
}
threadPool.shutdown();
threadPool.awaitTermination(100,TimeUnit.MILLISECONDS);
for(Future<Employee> future : listOfEmployee)
{
employeeList.add(future.get());
}
what can be the root cause for the problem?
My task is simple to download a file from a url using selenium. I did till clicking on the download part. Now I want to wait till the file is downloaded.Fine. I use following and done.
do {
Thread.sleep(60000);
}
while ((downloadeBuild.length()/1024) < 138900);
Now challenge is for how much time do I wait ? Can I set some threshold ? I can think of is use a counter in do while and check till counter goes to 10 or something like that ? But any other way in Java ? As such I do not have any action to do till the file is downloaded.
How about this?
I think using TimeOut is not stable since there is no need to wait for a un-predictable downloading operation.
You can just turn to CompletableFuture using supplyAsync to do the downloading and use thenApply to do the processing/converting and retrieve the result by join as follows:
public class SimpleCompletableFuture {
public static void main(String... args) {
testDownload();
}
private static void testDownload() {
CompletableFuture future = CompletableFuture.supplyAsync(() -> downloadMock())
.thenApply(SimpleCompletableFuture::processDownloaded);
System.out.println(future.join());
}
private static String downloadMock() {
try {
Thread.sleep(new Random().nextInt() + 1000); // mock the downloading time;
} catch (InterruptedException ignored) {
ignored.printStackTrace();
}
return "Downloaded";
}
private static String processDownloaded(String fileMock) {
System.out.println("Processing " + fileMock);
System.out.println("Done!");
return "Processed";
}
}
You can use guava Stopwatch
Stopwatch stopwatch = Stopwatch.createStarted();
while ((downloadeBuild.length()/1024) < 138900 && topWatch.elapsed(TimeUnit.SECONDS) < 60);
If what you want is a time out practice, may be you can try code below:
long timeout = 10 * 60 * 1000;
long start = System.currentTimeMillis();
while(System.currentTimeMillis() - timeout <= start ){
//Not timeout yet, wait
}
//Time out, continue
It's quite common in java library.
I have an HTML which is output (displaying the results of the threads) and displayed after all threads complete (I wait for completion using a join)
Sometimes individual threads can have exceptions.
If I don't have any exceptions in any threads, i want to display the HTML in my browser.
If I do have an exception in all threads then I want to NOT display the HTML
If I have an exception in some but not all threads then I want TO display the HTML
What's the easiest way (least amount of code) to implement something that can track if a thread has failed or not?
You can use CompletableFuture for this purpose, example:
val future1: CompletableFuture<String> = CompletableFuture.supplyAsync {
println("This is your thread 1 code")
"<html><head><title>"
}
val future2: CompletableFuture<String> = CompletableFuture.supplyAsync {
println("This is your thread 2 code")
if (Random().nextBoolean()) throw RuntimeException("Failed")
"Title!</title></html></head>"
}
future1.thenCombine(future2, {result1, result2 -> result1 + result2}).whenComplete { s, throwable ->
if (throwable != null) {
println("failed")
} else {
println("done with $s")
}
}
And in Kotlin 1.1 you will be able write this code in more readable way:
async {
try {
val s1 = await(future1)
val s2 = await(future2)
println(s1 + s2)
} catch (e: Exception) {
println("failed")
}
}