I have a list of objects for which I submit an update DB statement as below.
Current code:
List<MyObj> updateObj = ....<logic here>;
updateObj.stream().forEach(obj -> updateMyDB(obj));
Here each obj is getting executed sequentially and taking more time. Now I wish to parallelize it. What is the best way to do this?
Option 1:
updateObj.parallelStream().foreach(obj->updateMyObj(obj));
Option 2:
ExecutorService executor = new FixedThreaadPool(10);
updateObj.stream().foreach(obj -> completableFuture.runAsync(()-> updateMyObj(obj),executor));
Intention here is to parallize operations. Any ideas to achieve this?
Related
I have a workflow like: startevent -> task1(Assignee:Tom) -> choose sequence flow "agree" ->task2(Assignee:Jerry) -> choose sequence flow "disagree" -> task1
When the flow arrive to task1, i want to set assignee to "Tom" again.
Now i have an idea like:
When the flow arrive to task1, i use complete method, after the complete method, set a local variable "pre_task_id(task1's taskid)" in task2 so that i can use task1's taskid to search in "act_hi_taskinst" table for assignee(Tom), but this method taskService.setVariableLocal(taskId, variableName, value) need task2's taskid, how can i get the task2's taskid after complete method?
#Test
public void testCompleteTask() {
Task task = taskService.createTaskQuery().taskAssignee("Tom").singleResult();
if (task == null) {
System.out.println("no task!!!");
return;
}
String preTaskId = task.getId();
HashMap <String,Object> variables = new HashMap<>();
variables.put("userId", "Jerry");
variables.put("oper", "saolu");
taskService.complete(task.getId(),variables);
//don't konw how to get the taskId
//taskService.setVariableLocal(taskId, "pre_task_id", preTaskId);
}
I am using activiti6
Or please let me konw if there are any better solutions
workflow.png
If I understand properly. You need to keep track which task is executed most recently.
In that case you can use a stack. When ever you execute any task you just call the push with the task id.
When ever you call pop it will give you the task id of the most recent task executed.
Just be careful about how you want to clear the stack as well.
I would recommend to always call pop before calling push.
In the past I have always used a process variable for this purpose, Have never attempted the "push" operation suggested above but I would be concerned about getting the correct value back if multiple parallel processes are all pushing/pulling to and from the stack at once. The process variable is simple and I know it works.
I have a List<HystrixCommand<?>>> commands, what is the best way to execute these commands and collect the results such that the commands run in parallel?
I have tried something like this:
List<Future<?>> futures = commands.stream()
.map(HystrixCommand::queue)
.collect(Collectors.toList());
List<?> results = futures.stream()
.map(Future::get)
.collect(Collectors.toList());
Does this run the commands in parallel?
I.e. when calling HystrixCommand.queue() followed by Future.get() on the same thread, the .get() call does not block on some command and delay the other commands?
I ask because I couldn't find any documentation for this.
I have also looked at HystrixCollapser, but this still requires creating and running the individual commands (like above) in the createCommand method.
Ok I have investigated this and figured it out... by creating some simple examples rather than debugging production code...
My initial code was correct:
List<Request> requests = ...; // some expensive requests
List<HystrixCommand<?>>> commands = getCommands(requests);
List<Future<?>> futures = commands.stream()
.map(HystrixCommand::queue)
.collect(Collectors.toList());
List<?> results = futures.stream()
.map(Future::get)
.collect(Collectors.toList());
The commands do indeed run in parallel.
The .get() method does block, but since all the commands have been queued (prior to any .get() call) they are all running (or queued to run).
Say the second command is faster to completion than the first. The first .get() will block, but when it eventually returns, the second .get() call will return immediately, as the second command was able to complete while the first command was blocking. (Assuming core size >=2.)
In terms of HystrixCollapser, I misunderstood the API. HystrixCollapser is used to combine many HystrixCollapser instances into one HystrixCommand not the other way around. So I had to modify my code to wrap my requests with HystrixCollapserrather than HystrixCommand:
List<Request> requests = ...; // some expensive requests
List<HystrixCollapser<?>>> commands = getCommands(requests);
HystrixRequestContext context = HystrixRequestContext.initializeContext();
try {
List<Future<?>> futures = commands.stream()
.map(HystrixCollapser::queue)
.collect(Collectors.toList());
List<?> results = futures.stream()
.map(Future::get)
.collect(Collectors.toList());
} finally {
context.shutdown();
}
JMH benchmarks and full example source here
I am experimenting with CompletableFutures and hope someone could help turn head around. My algorithm below:
I have list of Articles.
I call REST API to get Categories for each Article.
I save Categories to DB.
I want perform steps 2 and 3 in parallel manner.
I want to wait until all Articles are processed, so there are no work done asynchronously after 2nd log.info().
My code is below:
log.info("Starting CustomCommandLineRunner");
List<Article> articles = articleService.getArticlesAvaialbleForAnalysis();
List<CompletableFuture<Article>> futureArticlesList = articles.stream()
.map(article -> {
CompletableFuture<Map<String, Double>> futureCategoriesMap = categoryService.getArticleCategories(article.getUrl());
return futureCategoriesMap.thenApply(categoriesMap -> articleService.setCatgoriesForArticle(article, categoriesMap));
})
.collect(Collectors.toList());
CompletableFuture<Void> allArticlesFuture = CompletableFuture.allOf(futureArticlesList.toArray(new CompletableFuture[futureArticlesList.size()]));
allArticlesFuture.get();
log.info("Ended CustomCommandLineRunner");
Unfortunately articleService.setCatgoriesForArticle() invoked twice. Could you please push me into right direction?
I apologize for false fire drill. Code in questions works as expected. articleService.setCatgoriesForArticle() is not invoked twice, but Spring #Transactional does not play well with CompletableFutures and this leads to issues with double save for same entity.
Lesson learned: do not mix #Transactional with asynchronous code.
I manage to write a REST API using Stripe Framework. Inside my API, I have several tasks which need to execute and combine their results. I come up with an approach, borrowed from JavaScript, which will spawn tasks into several threads and join rather than chronological implementation. Thus, I used ExecutorService but I found a bottleneck on the implementation when the number of requests is quite big, tasks are finished on a longer time than I expect.
My question is related to an alternate way to achieve the same purpose.
How can I create an Executors per request
How can I expand Executors' size
To demonstrate, let consider this way on Javascript
import Promise from 'bluebird';
let tasks = [];
tasks.push(task01);
tasks.push(task02);
Promise.all(tasks).then(results => { do_sth_here!} )
Bring this idea to Java, I have implemented like below
ExecutorService exec = Executors.newCachedThreadPool();
List<Callable<Promise>> tasks = new ArrayList<>();
List<Future<Promise>> PromiseAll;
try {
tasks.add(() -> TaskPromises(Input));
tasks.add(() -> TaskPromise(Input));
PromiseAll = exec.invokeAll(tasks);
for (Future<Promise> fr : PromiseAll) {
// do_some_thing_next
}
}
I want to work with two threads in my Java program for a tiny part. I need to give the first call to a database and the second call to an API, both calls with same input, and then work with the output of whichever thread finishes first.
It's my first time programming with threads and I'm very confused. I've seen tutorials and they mainly explain how to get two separate things done with threads so I'm a little lost.
Can someone please help or re-direct me to any useful link they may have?
So far, as I understand it, should it look something like this? :
Thread thread1 = new Thread(func1());
Thread thread2 = new Thread(func2());
thread1.start();
thread2.start();
But then how do I extract the output of the functions? How would I know which one has finished first?
-----------UPDATE 1---------
After trying CompletableFuture (thanks for the help Johan!) I have something like this:
CompletableFuture<Object> getData = CompletableFuture.anyOf(
CompletableFuture.runAsync(() -> getDataFromDB(clientData)),
CompletableFuture.runAsync(() -> getDataFromApi(clientData))
);
getData.thenApply(dataObject -> {
// Cast the returned Object to the actual type of your data,
// assuming both getDataFromDb and getDataFromApi
// return the same result type
Object data = (String) dataObject;
// Work with the returned data.
result = (String) data;
});
But I get this error for getData.thenApply():
The method thenApply(Function) in the type CompletableFuture is not applicable for the arguments (( dataObject) -> {})
Since I know that getData in of type String, would it be okay to just convert it to String and store the result?
As #Johan Hirsch suggests try with CompletableFuture. I've just try this and it works:
CompletableFuture.anyOf(
CompletableFuture.supplyAsync(() -> getDataFromDB(clientData)),
CompletableFuture.supplyAsync(() -> getDataFromApi(clientData)))
.thenApply(item -> (String) item)
.thenAccept(result -> {
// Consume the data
System.out.println(result);
});
Beware that I'm currently consuming the data so it doesn't return anything. If you just want to pass the result to another CompletableFuture change the thenAccept method for a thenApply
Java 8 provides a very nice utility class called CompletableFuture, which can help in your case.
Create two CompletableFuture, one for each of your tasks, and then use the CompletableFuture.anyOf method to wait for either one to finish.
CompletableFuture<TData> getData = CompletableFuture.anyOf(
CompletableFuture.runAsync(() -> getDataFromDb()),
CompletableFuture.runAsync(() -> getDataFromApi())
);
getData.thenApply(dataObject -> {
// Cast the returned Object to the actual type of your data,
// assuming both getDataFromDb and getDataFromApi
// return the same result type
TData data = (TData)dataObject;
// Work with the returned data.
processData(data);
});
You can use ExecutorService.invokeAny
Executes the given tasks, returning the result of one that has completed successfully (i.e., without throwing an exception), if any do. Upon normal or exceptional return, tasks that have not completed are cancelled. The results of this method are undefined if the given collection is modified while this operation is in progress.