I have a hashmap that contains ip/port information and a message that has to be sent to the whole list.
So I decided to create a small method that accepts the hashmap and the message and does this. It looks something like this:
public static ChannelFuture sendMessageTo(Map<JsonElement, JsonObject> list, String message) {
Set<JsonElement> keys = list.keySet();
for (JsonElement key : keys) { //iterate through the map
ChannelInboundHandler[] handlers = {
new MessageCompletenessHandler(),
new MessageRequest(message),
};
JsonObject identity = list.get(key);
ChannelFuture f = connectWithHandler(identity.get("ip").getAsString(), identity.get("port").getAsInt(), handlers); //to the following ip/port send message and return ChannelFuture
}
return result; //here result should be a ChannelFuture that when .addListener is added it should be called only when ALL the ChannelFuture-s from the for loop have finished(a.k.a. all messages have been sent)
}
The comments should explain the situation clearly enough.
The question is how do I implement this ChannelFuture result.
I know I can .sync() the ChannelFuture-s, but this defeats the purpose of async networking.
P.S.: I essentially want to have the functionality described here https://twistedmatrix.com/documents/16.2.0/api/twisted.internet.defer.DeferredList.html but am failing to find an equivalent.
In general what you're trying to achieve is not really correct async way. However netty has an utility class for that kind of task - DefaultChannelGroupFuture.java. However it is package private and used only for DefaultChannelGroup.java which indeed does what you described in your code. So you could easily copy this DefaultChannelGroupFuture and use it. To be more specific :
Collection<ChannelFuture> futures = ArrayList<ChannelFuture>();
...
//here you add your futures
ChannelFuture f = connectWithHandler(identity.get("ip").getAsString(), identity.get("port").getAsInt(), handlers);
futures.add(f);
...
DefaultChannelGroupFuture groupOfFutures = new DefaultChannelGroupFuture(futures, executor);
if (groupOfFutures.sync().isSuccess()) {
}
Have in mind you'll need to change DefaultChannelGroupFuture for your needs.
Related
I'm using Mutiny extension (for Quarkus) and I don't know how to manage this problem.
I want to send many request in an async way so I've read about Mutiny extension. But the server closes the connection because it receives thousand of them.
So I need:
Send the request by blocks
After all request are sent, do things.
I've been using Uni object to combine all the responses as this:
Uni<Map<Integer, String>> uniAll = Uni.combine()
.all()
.unis(list)
.combinedWith(...);
And then:
uniAll.subscribe()
.with(...);
This code, send all the request in paralell so the server closes the connection.
I'm using group of Multi objects, but I don't know how to use it (in Mutiny docs I can't found any example).
This is the way I'm doing now:
//Launch 1000 request
for (int i=0;i<1000;i++) {
multi = client.getAbs("https://api.*********.io/jokes/random")
.as(BodyCodec.jsonObject())
.send()
.onItem().transformToMulti(
array -> Multi.createFrom()
.item(array.body().getString("value")))
.group()
.intoLists()
.of(100)
.subscribe()
.with(a->{
System.out.println("Value: "+a);
});
}
I think that the subscription doesn't execute until there are "100" groups of items, but I guess this is not the way because it doesn't work.
Does anybody know how to launch 1000 of async requests in blocks of 100?
Thanks in advance.
UPDATED 2021-04-19
I've tried with this approach:
List<Uni<String>> listOfUnis = new ArrayList<>();
for (int i=0;i<1000;i++) {
listOfUnis.add(client
.getAbs("https://api.*******.io/jokes/random")
.as(BodyCodec.jsonObject())
.send()
.onItem()
.transform(item -> item
.body()
.getString("value")));
}
Multi<Uni<String>> multiFormUnis = Multi.createFrom()
.iterable(listOfUnis);
List<String> listOfResponses = new ArrayList<>();
List<String> listOfValues = multiFormUnis.group()
.intoLists()
.of(100)
.onItem()
.transformToMultiAndConcatenate(listOfOneHundred ->
{
System.out.println("Size: "+listOfOneHundred.size());
for (int index=0;index<listOfOneHundred.size();index++) {
listOfResponses.add(listOfOneHundred.get(index)
.await()
.indefinitely());
}
return Multi.createFrom()
.iterable(listOfResponses);
})
.collectItems()
.asList()
.await()
.indefinitely();
for (String value : listOfValues) {
System.out.println(value);
}
When I put this line:
listOfResponses.add(listOfOneHundred.get(index)
.await()
.indefinitely());
The responses are printed one after each other, and when the first 100s group of items ends, it prints the next group. The problem? There are sequential requests and it takes so much time
I think I am close to the solution, but I need to know, how to send the parallel request only in group of 100s, because if I put:
subscribe().with()
All the request are sent in parallel (and not in group of 100s)
I think you create the multy wrong, it would be much easier to use this:
Multi<String> multiOfJokes = Multi.createFrom().emitter(multiEmitter -> {
for (int i=0;i<1000;i++) {
multiEmitter.emit(i);
}
multiEmitter.complete();
}).onItem().transformToUniAndMerge(index -> {
return Uni.createFrom().item("String" + index);
})
With this approach it should mace the call parallel.
Now is the question of how to make it to a list.
The grouping works fine
I run it with this code:
Random random = new Random();
Multi<Integer> multiOfInteger = Multi.createFrom().emitter(multiEmitter -> {
for (Integer i=0;i<1000;i++) {
multiEmitter.emit(i);
}
multiEmitter.complete();
});
Multi<String> multiOfJokes = multiOfInteger.onItem().transformToUniAndMerge(index -> {
if (index % 10 == 0 ) {
Duration delay = Duration.ofMillis(random.nextInt(100) + 1);
return Uni.createFrom().item("String " + index + " delayed").onItem()
.delayIt().by(delay);
}
return Uni.createFrom().item("String" + index);
}).onCompletion().invoke(() -> System.out.println("Completed"));
Multi<List<String>> multiListJokes = multiOfJokes
.group().intoLists().of(100)
.onCompletion().invoke(() -> System.out.println("Completed"))
.onItem().invoke(strings -> System.out.println(strings));
multiListJokes.collect().asList().await().indefinitely();
You will get a list of your string.
I don't know, how you intend to send the list to backend.
But you can either to it with:
call (executed asynchronously)
write own subscriber (implements Subscriber) the methods are straight forward.
As you need for your bulk request.
I hope you understand it better afterward.
PS: link to guide where I learned all of it:
https://smallrye.io/smallrye-mutiny/guides
So in short you want to batch parallel calls to the server, without hitting it with everything at once.
Could this work for you? It uses merge. In my example, it has a parallelism of 2.
Multi.createFrom().range(1, 10)
.onItem()
.transformToUni(integer -> {
return <<my long operation Uni>>
})
.merge(2) //this is the concurrency
.collect()
.asList();
I'm not sure if merge was added later this year, but this seems to do what you want. In my example, the "long operation producing Uni" is actually a call to the Microprofile Rest Client which produces a Uni, and returns a string. After the merge you can put another onItem to perform something with the response (it's a plain Multi after the merge), instead of collecting everything as list.
According to the business logic, the output of one of the method is used as an input to another. The logic has linear flow.
To emulate the behaviour, now there is a controller class which has everything.
It is very messy, too much loc and hard to modify. Also the exception handling is very complex. The individual method does some handling but the global exceptions bubble up and which involves a lot of try catch statements.
Does there exists a design pattern to address this problem?
Example Controller Class Code
try{
Logic1Inputs logic1_inputs = new Logic1Inputs( ...<some other params>... );
Logic1 l = new Logic1(logic1_inputs);
try{
Logic1Output l1Output = l.execute();
} catch( Logic1Exception l1Exception) {
// exception handling
}
Logic2Inputs logic2_inputs = new Logic2Inputs(l1Output);
Logic2 l2 = new Logic2(logic2_inputs);
try{
Logic2Output l2Output = l2.execute();
} catch( Logic2Exception l2Exception) {
// exception handling
}
Logic3Inputs logic3_inputs = new Logic3Inputs(l1Output, l2Output);
Logic3 l3 = new Logic3(logic2_inputs);
try{
Logic3Output l3Output = l3.execute();
} catch( Logic3Exception l3Exception) {
// exception handling
}
} catch(GlobalException globalEx){
// exception handling
}
I think this is called pipeline: http://en.wikipedia.org/wiki/Pipeline_%28software%29 This pattern is used for algorithms in which data flows through a sequence of tasks or stages.
You can search for a library that does this( http://code.google.com/p/pipelinepattern ) or try your own java implementation
Basically you have all you objects in a list and the output from one si passed to the next. This is a naive implementation but you can add generics and all you need
public class BasicPipelinePattern {
List<Filter> filters;
public Object process(Object input) {
for (Filter c : filters) {
try {
input = c.apply(input);
} catch (Exception e) {
// exception handling
}
}
return input;
}
}
public interface Filter {
public Object apply(Object o);
}
When faced with problems like this, I like to see how other programming languages might solve it. Then I might borrow that concept and apply it to the language that I'm using.
In javascript, there has been much talk of promises and how they can simplify not only asynchronous processing, but error handling. This page is a great introduction to the problem.
Then approach has been called using "thenables". Here's the pseudocode:
initialStep.execute().then(function(result1){
return step2(result1);
}).then(function(result2){
return step3(result3);
}).error(function(error){
handle(error);
}).done(function(result3){
handleResult(result3)
});
The advantage of this pattern is that you can focus on the processing and effectively handle errors in one place without needing to worry about checking for success at each step.
So how would this work in java? I would take a look at one of the promises/futures libraries, perhaps jdeferred. I would expect that you could put something like this together (assuming java 8 for brevity):
initialPromise.then( result1 -> {
Logic2 logic2 = new Logic2(new Logic2Inputs(result1));
return logic2.execute();
}).then(result2 -> {
Logic3 logic3 = new Logic3(new Logic3Inputs(result2));
return logic2.execute();
}).catch(exception -> {
handleException(exception)
}).finally( result -> {
handleResult(result);
});
This does, of course gloss over a hidden requirement in your code. You mention that in step 3 you need the output for both step 1 and step 2. If you were writing scala, there is syntactic sugar that would handle this for you (leaving out error handling for the moment):
for(result1 <- initialStep.execute();
Logic2 logic2 = new Logic2(Logic2Input(result1));
result2 <- logic2.execute();
Logic3 logic3 = new Logic3(Logic3Input(result1, result2));
result3 <- logic3.execute()) yield result3;
But since you don't have the ability here, then you are left to the choices of being refactoring each step to take only the output of the previous step, or nesting the processing so that result1 is still in scope when you need to set up step 3.
The classic alternative to this, as #user1121883 mentioned would be to use a Pipeline processor. The downside to this approach is that it works best if your input and output are the same type. Otherwise you are going to have to push Object around everywhere and do a lot of type checking.
Another alternative would be to expose a fluent interface for the pipeline. Again, you'd want to do some refactoring, perhaps to have a parameter-less constructor and a consistent interface for inputs and outputs:
Pipeline p = new Pipeline();
p.then(new Logic1())
.then(new Logic2())
.then(new Logic3())
.addErrorHandlder(e->handleError(e))
.complete();
This last option is more ideomatic java, but retains many of the advantages of the thenables processing, so it's probably the way that I would go.
I have vertx server application where I am getting single client requests and from the server, I need to make two blocking calls. For instance, one call to back-end system A and another call to back-end system B. I am looking to make two concurrent calls to both the systems. I need to wait for the responses from both the calls and then merge two data from both the calls and then send the response back to client. I am unable to figure out how to do this in worker verticle.
Could anyone recommend what would be the best approach in vertx?
This sounds like a good use case for Promises. Give the module vertx-promises a try.
create a CompositeFuture from your launched Futures and handle it normally.
public Future<JsonArray> getEntitiesByIndFields(String keyspace, String entidad, String field1, String field2) {
Promise<JsonArray> p = Promise.promise();
// launch in parallel
Future<JsonArray> f1 = getEntitiesByIndField1(keyspace, entidad, field1);
Future<JsonArray> f2 = getEntitiesByIndField2(keyspace, entidad, field2);
CompositeFuture.all(f1, f2).setHandler(done ->
{
if (done.failed()) {
p.fail(done.cause());
return;
}
List<JsonArray> ja = done.result().list();
JsonArray finalarray = ja.get(0);
ja.get(1).forEach(jo ->
{ // add one by one, don't duplicate ids
long id = ((JsonObject) jo).getLong("id");
if (!containsKey(finalarray, id)) {
finalarray.add(jo);
}
});
;
p.complete(finalarray); // send union of founds
});
return p.future();
}
I have multiple clients that send files to a server. For one set of data there are two files that contain information about that data, each with the same name. When a file is received, the server sends a message out to my queue containing the file path, file name, ID of the client, and the "type" of file it is (all have same file extension but there are two "types," call them A and B).
The two files for one set of data have the same file name. As soon as the server has received both of the files I need to start a program that combines the two. Currently I have something that looks like this:
from("jms:queue.name").aggregate(header("CamelFileName")).completionSize(2).to("exec://FILEPATH?args=");
Where I am stuck is the header("CamelFileName"), and more specifically how the aggregator works.
With the completionSize set to 2 does it just suck up all the messages and store them in some data structure until a second message that matches the first comes through? Also, does the header() expect a specific value? I have multiple clients so I was thinking of having the client ID and the file name in the header, but then again I don't know if I have to give a specific value. I also don't know if I can use a regex or not.
Any ideas or tips would be super helpful.
Thanks
EDIT:
Here is some code I have now. Based on my description of the problem here and in comments on selected answer does it seem accurate (besides close brackets that I didn't copy over)?
public static void main(String args[]) throws Exception{
CamelContext c = new DefaultCamelContext();
c.addComponent("activemq", activeMQComponent("vm://localhost?broker.persistent=false"));
//ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("vm://localhost?broker.persistent=false");
//c.addComponent("jms", JmsComponent.jmsComponentAutoAcknowledge(connectionFactory));
c.addRoutes(new RouteBuilder() {
public void configure() {
from("activemq:queue:analytics.camelqueue").aggregate(new MyAggregationStrategy()).header("subject").completionSize(2).to("activemq:queue:analytics.success");
}
});
c.start();
while (true) {
System.out.println("Waiting on messages to come through for camel");
Thread.sleep(2 * 1000);
}
//c.stop();
}
private static class MyAggregationStrategy implements AggregationStrategy {
public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
if (oldExchange == null)
return newExchange;
// and here is where combo stuff goes
String oldBody = oldExchange.getIn().getBody(String.class);
String newBody = newExchange.getIn().getBody(String.class);
boolean oldSet = oldBody.contains("set");
boolean newSet = newBody.contains("set");
boolean oldFlow = oldBody.contains("flow");
boolean newFlow = newBody.contains("flow");
if ( (oldSet && newFlow) || (oldFlow && newSet) ) {
//they match so return new exchange with info so extractor can be started with exec
String combined = oldBody + "\n" + newBody + "\n";
newExchange.getIn().setBody(combined);
return newExchange;
}
else {
// no match so do something....
return null;
}
}
}
you must supply an AggregationStrategy to define how you want to combine Exchanges...
if you are only interested in the fileName and receiving exactly 2 Exchanges, then you can just use the UseLatestAggregationStrategy to just pass the newest Exchange through once 2 have been 'aggregated'...
that said, it sounds like you need to retain both Exchanges (one for each clientId) so you can pass that info on to the 'exec' step...if so, you can just combine the Exchanges into a GroupedExchange holder using the built-in aggregation strategy enabled via the groupExchanges option...or specificy a custom AggregationStrategy to combine them however you'd like. just need to keep in mind that your 'exec' step needs to handle whatever aggregated structure you decide to use...
see these unit tests for examples:
https://svn.apache.org/repos/asf/camel/trunk/camel-core/src/test/java/org/apache/camel/processor/aggregator/AggregatorTest.java
https://svn.apache.org/repos/asf/camel/trunk/camel-core/src/test/java/org/apache/camel/processor/aggregator/AggregateGroupedExchangeTest.java
I have a Play 2.1 controller in Java, and I need to call an external webservice to get some data. Then with this data result, I must call another web service with n calls, corresponding to the n results from the first webservice call.
For performance issues, I want to make the n calls in separated threads using promises.
So I would have a loop like this:
List<String> firstResults = WS.url("http://...") ///...blablabla
for(String keyword : firstResults){
Promise<ResultType> promise = play.libs.Akka.future(
new Callable<ResultType>() {
public Integer call() {
return //...
}
}
);}
How can I synchronize the n promises and then reduce the results in one response (a list of all results) using the Async API, and then return the http responses only when all calls are finished?
Not being able to know the number of calls make the problem more difficult ... (I can't declare promises as promise1, promise2 etc.)
Promise.waitAll is what you want:
List<String> firstResults = WS.url("http://...") ///...blablabla
List<Promise<? extends ResultType>> webServiceCalls = new ArrayList<>;
for(String keyword : firstResults){
Promise<ResultType> promise = WS.url("http://...?keyboard=" + keyword).get().map(
// function of Response to ResultType
);
webServiceCalls.add(promise);
}
// Don't be confused by the name here, it's not actually waiting
Promise<List<ResultType>> results = Promise.waitAll(webServiceCalls);
return async(results.map(new Function<List<ResultType, Result>>() {
public Result apply(List<ResultType> results) {
// Convert results to ResultType
}
});