I'm starting to integrate Spring WebFlux into our project and have recently run into an issue regarding Flux.flatMap(). The following code summarizes it:
#Component
#RequiredArgsConstructor
public class Client {
private final ParameterizedTypeReference<List<ResponseDto>> TYPE = new ParameterizedTypeReference<>() {
};
private final WebClient webClient;
public Mono<List<ResponseDto>> queryData(List<RequestDto> request) {
return webClient.post()
.uri("/api/query-data")
.bodyValue(request)
.retrieve()
.bodyToMono(TYPE);
}
}
#SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.DEFINED_PORT)
class DemoApplicationTest {
static final int COUNT = 1000_000;
#Autowired
private Client client;
#Test
void requestsAreSent() {
List<RequestDto> requests = getRequests();
List<List<RequestDto>> partition = Lists.partition(requests, 1000);
Flux.fromIterable(partition)
.flatMap(client::queryData)
.collectList()
.block();
}
#Test
void requestsAreNotSent() {
List<RequestDto> requests = getRequests();
List<List<RequestDto>> partition = Lists.partition(requests, 1000);
Flux.fromIterable(partition)
.map(client::queryData)
.collectList()
.block();
}
private static List<RequestDto> getRequests() {
return IntStream.range(0, COUNT)
.boxed()
.map(operand -> RequestDto.builder().build())
.collect(Collectors.toList());
}
}
If I use Flux.flatMap() I see that my test sends requests to webserver, but if instead I use Flux.map() no request are sent.
Could one explain this behavior? Is it something inherent or I misconfigured anything?
Reactive publishers like Mono and Flux are lazy. This means that they're not executed unless you subscribe to them. In both your requestsAreSent() and requestsAreNotSent() methods, you're only subscribing to the outer publisher by using the block() method.
The reason why your requests are sent within the requestsAreSent() method is because you use flatMap(). This operation flattens the inner publisher from client.queryData() into the outer publisher. This means that if you subscribe to the outer observable, you're also subscribed to the inner observables.
In your requestsAreNotSent() method on the other hand, the publishers aren't flattened, and because you only subscribe to the outer observable, the logic within client.queryData() isn't executed.
For example, if you subscribe to all those inner observables, the logic will be executed:
List<Mono<List<ResponseDto>>> list = Flux.fromIterable(partition)
.map(client::queryData)
.collectList()
.block();
list.forEach(Mono::block); // Subscribing to the inner observables
However, this is a bad practice and you should keep using flatMap().
Related
I'm new with Kafka and want to persist data from kafka topics to database tables (each topic flow to a specific table). I know Kafka connect exists and can be used to achieve this but there are reasons why this approach is preferred.
Unfortunately only one topic is writing the database. Kafka seems to not process() all processors concurrently. Either MyFirstData is writing to database or MySecondData but never but at the same time.
According the my readings the is the option overriding init() from of kafka stream Processor interface which offers context.forward() not sure if this will help and how to use it in my used case.
I use Spring Cloud Stream (but got the same behaviour with Kafka DSL and Processor API implementations)
My code snippet:
Configuring the consumers:
#Configuration
#RequiredArgsConstructor
public class DatabaseProcessorConfiguration {
private final MyFirstDao myFirstDao;
private final MySecondDao mySecondDao;
#Bean
public Consumer<KStream<GenericData.Record, GenericData.Record>> myFirstDbProcessor() {
return stream -> stream.process(() -> {
return new MyFirstDbProcessor(myFirstDao);
});
}
#Bean
public Consumer<KStream<GenericRecord, GenericRecord>> mySecondDbProcessor() {
return stream -> stream.process(() -> new MySecondDbProcessor(mySecondDao));
}
}
This MyFirstDbProcessor and MySecondDbProcessor is analog to this.
#Slf4j
#RequiredArgsConstructor
public class MyFirstDbProcessor implements Processor<GenericData.Record, GenericData.Record, Void, Void> {
private final MyFirstDao myFirstDao;
#Override
public void process(Record<GenericData.Record, GenericData.Record> record) {
CdcRecordAdapter adapter = new CdcRecordAdapter(record.key(), record.value());
MyFirstTopicKey myFirstTopicKey = adapter.getKeyAs(MyFirstTopicKey.class);
MyFirstTopicValue myFirstTopicValue = adapter.getValueAs(MyFirstTopicValue.class);
MyFirstData data = PersistenceMapper.map(myFirstTopicKey, myFirstTopicValue);
switch (myFirstTopicValue.getCrudOperation()) {
case UPDATE, INSERT -> myFirstDao.persist(data);
case DELETE -> myFirstDao.delete(data);
default -> System.err.println("unimplemented CDC operation streamed by kafka");
}
}
}
My Dao implementations: I try an implementation of MyFirstRepository with JPARepository and ReactiveCrudRepository but same behaviour. MySecondRepository is implemented analog to MyFirstRepository.
#Component
#RequiredArgsConstructor
public class MyFirstDaoImpl implements MyFirstDao {
private final MyFirstRepository myFirstRepository;
#Override
public MyFirstData persist(MyFirstData myFirstData) {
Optional<MyFirstData> dataOptional = MyFirstRepository.findById(myFirstData.getId());
if (dataOptional.isPresent()){
var data = dataOptional.get();
myFirstData.setCreatedDate(data.getCreatedDate());
}
return myFirstRepository.save(myFirstData);
}
#Override
public void delete(MyFirstData myFirstData) {
System.out.println("delete() from transaction detail dao called");
MyFirstRepository.delete(myFirstData);
}
}
I currently have implemented in a Spring Boot project running on Fargate an SQS listener.
It's possible that under the hood, the SqsAsyncClient which appears to be a listener, is actually polling though.
Separately, as a PoC, on I implemented a Lambda function trigger on a different queue. This would be invoked when there are items in the queue and would post to my service. This seems unnecessarily complex to me but removes a single point of failure if I were to only have one instance of the service.
I guess my major point of confusion is whether I am needlessly worrying about polling vs listening on a SQS queue and whether it matters.
Code for example purposes:
#Component
#Slf4j
#RequiredArgsConstructor
public class SqsListener {
private final SqsAsyncClient sqsAsyncClient;
private final Environment environment;
private final SmsMessagingServiceImpl smsMessagingService;
#PostConstruct
public void continuousListener() {
String queueUrl = environment.getProperty("aws.sqs.sms.queueUrl");
Mono<ReceiveMessageResponse> responseMono = receiveMessage(queueUrl);
Flux<Message> messages = getItems(responseMono);
messages.subscribe(message -> disposeOfFlux(message, queueUrl));
}
protected Flux<Message> getItems(Mono<ReceiveMessageResponse> responseMono) {
return responseMono.repeat().retry()
.map(ReceiveMessageResponse::messages)
.map(Flux::fromIterable)
.flatMap(messageFlux -> messageFlux);
}
protected void disposeOfFlux(Message message, String queueUrl) {
log.info("Inbound SMS Received from SQS with MessageId: {}", message.messageId());
if (someConditionIsMet())
deleteMessage(queueUrl, message);
}
protected Mono<ReceiveMessageResponse> receiveMessage(String queueUrl) {
return Mono.fromFuture(() -> sqsAsyncClient.receiveMessage(
ReceiveMessageRequest.builder()
.maxNumberOfMessages(5)
.messageAttributeNames("All")
.queueUrl(queueUrl)
.waitTimeSeconds(10)
.visibilityTimeout(30)
.build()));
}
protected void deleteMessage(String queueUrl, Message message) {
sqsAsyncClient.deleteMessage(DeleteMessageRequest.builder()
.queueUrl(queueUrl)
.receiptHandle(message.receiptHandle())
.build())
.thenAccept(deleteMessageResponse -> log.info("deleted message with handle {}", message.receiptHandle()));
}
}
I have spring integration flow that gets triggered once a every day, that pulls all parties from database and sends each party to an executorChannel.
The next flow would pull data for each party and then process them parallelly by sending in to a different executor channel.
Challenge i'm facing is how do i know when this entire process ends. Any ideas on how to acheve this .
Here's my pseudo code of executor channels and integration flows.
#Bean
public IntegrationFlow fileListener() {
return IntegrationFlows.from(Files.inboundAdapter(new
File("pathtofile"))).channel("mychannel").get();
}
#Bean
public IntegrationFlow flowOne() throws ParserConfigurationException {
return IntegrationFlows.from("mychannel").handle("serviceHandlerOne",
"handle").nullChannel();
}
#Bean
public IntegrationFlow parallelFlowOne() throws ParserConfigurationException {
return IntegrationFlows.from("executorChannelOne").handle("parallelServiceHandlerOne",
"handle").nullChannel();
}
#Bean
public IntegrationFlow parallelFlowTwo() throws ParserConfigurationException {
return IntegrationFlows.from("executorChannelTwo").handle("parallelServiceHandlerTwo",
"handle").nullChannel();
}
#Bean
public MessageChannel executorChannelOne() {
return new ExecutorChannel(
Executors.newFixedThreadPool(10));
}
#Bean
public MessageChannel executorChannelTwo;() {
return new ExecutorChannel(
Executors.newFixedThreadPool(10));
}
#Component
#Scope("prototype")
public class ServiceHandlerOne{
#Autowired
MessageChannel executorChannelOne;
#ServiceActivator
public Message<?> handle(Message<?> message) {
List<?> rowDatas = repository.findAll("parties");
rowDatas.stream().forEach(data -> {
Message<?> message = MessageBuilder.withPayload(data).build();
executorChannelOne.send(message);
});
return message;
}
}
#Component
#Scope("prototype")
public class ParallelServiceHandlerOne{
#Autowired
MessageChannel executorChannelTwo;;
#ServiceActivator
public Message<?> handle(Message<?> message) {
List<?> rowDatas = repository.findAll("party");
rowDatas.stream().forEach(data -> {
Message<?> message = MessageBuilder.withPayload(data).build();
executorChannelTwo;.send(message);
});
return message;
}
}
First of all no reason to make your services as #Scope("prototype"): I don't see any state holding in your services, so they are stateless, therefore can simply be as singleton. Second: since you make your flows ending with the nullChannel(), therefore point in returning anything from your service methods. Therefore just void and flow is going to end over there naturally.
Another observation: you use executorChannelOne.send(message) directly in the code of your service method. The same would be simply achieved if you just return that new message from your service method and have that executorChannelOne as the next .channel() in your flow definition after that handle("parallelServiceHandlerOne", "handle").
Since it looks like you do that in the loop, you might consider to add a .split() in between: the handler return your List<?> rowDatas and splitter will take care for iterating over that data and replies each item to that executorChannelOne.
Now about your original question.
There is really no easy to say that your executors are not busy any more. They might not be at the moment of request just because the message for task has not reached an executor channel yet.
Typically we recommend to use some async synchronizer for your data. The aggregator is a good way to correlate several messages in-the-flight. This way the aggregator collects a group and does not emit reply until that group is completed.
The splitter I've mentioned above adds a sequence details headers by default, so subsequent aggregator can track a message group easily.
Since you have layers in your flow, it looks like you would need a several aggregators: two for your executor channels after splitting, and one top level for the file. Those two would reply to the top-level for the final, per-file grouping.
You also may think about making those parties and party calls in parallel using a PublishSubscribeChannel, which also can be configured with a applySequence=true. This info then will be used by the top-level aggregator for info per file.
See more in docs:
https://docs.spring.io/spring-integration/docs/current/reference/html/core.html#channel-implementations-publishsubscribechannel
https://docs.spring.io/spring-integration/docs/current/reference/html/message-routing.html#splitter
https://docs.spring.io/spring-integration/docs/current/reference/html/message-routing.html#aggregator
I'm actively using ApplicationEventPublisher in my app and the main result of some methods executions is publishing event with ApplicationEventPublisher.
I am using a simple trap for events in the test environment in order to collect events and verify them:
#Singleton
public class MessageListenerTestHelper {
private ConcurrentLinkedQueue queue = new ConcurrentLinkedQueue<>();
#Async
#EventListener
public void onEvent(Object event) {
queue.add(event);
}
public Queue getQueue() {
return queue;
}
public <T> Future<T> getEventFromQueue(Class<T> eventClass) {
CompletableFuture<T> future = new CompletableFuture<>();
Executors.newSingleThreadScheduledExecutor().scheduleAtFixedRate(() -> {
Optional eventOpt = queue.stream()
.filter(eventClass::isInstance)
.findAny();
if (eventOpt.isPresent()) {
future.complete((T) eventOpt.get());
}
}, 100, 100, TimeUnit.MILLISECONDS);
return future;
}
}
But my tests are flaky - its usually fails in github actions, but works at my computer. So I want to fix it by mock ApplicationEventPublisher. But #Replaces annotation doesn't work. I tried it in the test and in factory available only in test environment, but neither of this is worked.
I am going to refuse to use #MicronautTest annotation, and inject mocks manually. But maybe there is another choise?
I'm playing around with reactive patterns in a Java (8) Spring Boot (1.5.2.RELEASE) application with Akka (2.5.1). It's coming along nicely but now I'm stuck trying to run a CompletableFuture from an actor. To simulate this I have created a very simple service that returns a CompletableFuture. However, when I then try to return the result to the calling controller I get errors about dead-letters and no response is returned.
The error I am getting is:
[INFO] [05/05/2017 13:12:25.650] [akka-spring-demo-akka.actor.default-dispatcher-5] [akka://akka-spring-demo/deadLetters] Message [java.lang.String] from Actor[akka://akka-spring-demo/user/$a#-1561144664] to Actor[akka://akka-spring-demo/deadLetters] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
Here is my code. This is the controller calling the actor:
#Component
#Produces(MediaType.TEXT_PLAIN)
#Path("/")
public class AsyncController {
#Autowired
private ActorSystem system;
private ActorRef getGreetingActorRef() {
ActorRef greeter = system.actorOf(SPRING_EXTENSION_PROVIDER.get(system)
.props("greetingActor"));
return greeter;
}
#GET
#Path("/foo")
public void test(#Suspended AsyncResponse asyncResponse, #QueryParam("echo") String echo) {
ask(getGreetingActorRef(), new Greet(echo), 1000)
.thenApply((greet) -> asyncResponse.resume(Response.ok(greet).build()));
}
}
Here is the service:
#Component
public class GreetingService {
public CompletableFuture<String> greetAsync(String name) {
return CompletableFuture.supplyAsync(() -> "Hello, " + name);
}
}
Then here is the actor receiving the call. At first I had this:
#Component
#Scope(ConfigurableBeanFactory.SCOPE_PROTOTYPE)
public class GreetingActor extends AbstractActor {
#Autowired
private GreetingService greetingService;
#Autowired
private ActorSystem system;
#Override
public Receive createReceive() {
return receiveBuilder()
.match(Greet.class, this::onGreet)
.build();
}
private void onGreet(Greet greet) {
greetingService.greetAsync(greet.getMessage())
.thenAccept((greetingResponse) -> getSender().tell(greetingResponse, getSelf()));
}
}
This resulted in 2 calls being handled correctly but after that I would get dead-letter errors. Then I read here what was probably causing my problems:
http://doc.akka.io/docs/akka/2.5.1/java/actors.html
Warning
When using future callbacks, inside actors you need to carefully avoid closing over the containing actor’s reference, i.e. do not call methods or access mutable state on the enclosing actor from within the callback. This would break the actor encapsulation and may introduce synchronization bugs and race conditions because the callback will be scheduled concurrently to the enclosing actor. Unfortunately there is not yet a way to detect these illegal accesses at compile time. See also: Actors and shared mutable state
So I figured the idea is that you pipe the result to self() after which you can do getSender().tell(response, getSelf()).
So I altered my code to this:
#Component
#Scope(ConfigurableBeanFactory.SCOPE_PROTOTYPE)
public class GreetingActor extends AbstractActor {
#Autowired
private GreetingService greetingService;
#Autowired
private ActorSystem system;
#Override
public Receive createReceive() {
return receiveBuilder()
.match(Greet.class, this::onGreet)
.match(String.class, this::onGreetingCompleted)
.build();
}
private void onGreet(Greet greet) {
pipe(greetingService.greetAsync(greet.getMessage()), system.dispatcher()).to(getSelf());
}
private void onGreetingCompleted(String greetingResponse) {
getSender().tell(greetingResponse, getSelf());
}
}
The onGreetingCompleted method is being called with the response from the GreetingService but at that time I again get the dead-letters error so for some reason it can't send the response back to the calling controller.
Note that if I change the service to this:
#Component
public class GreetingService {
public String greet(String name) {
return "Hello, " + name;
}
}
And the onGreet in the actor to:
private void onGreet(Greet greet) {
getSender().tell(greetingService.greet(greet.getMessage()), getSelf());
}
Then everything works fine. So it would appear that I have my basic Java/Spring/Akka set up correctly, it's just when trying to call a CompletableFuture from my actor that the problems start.
Any help would be much appreciated, thanks!
The getSender method is only reliably returning the ref of the sender during the synchronous processing of the message.
In your first case, you have:
greetingService.greetAsync(greet.getMessage())
.thenAccept((greetingResponse) -> getSender().tell(greetingResponse, getSelf()));
Which means that getSender() is invoked async once the future completes. Not reliable anymore. You can change that to:
ActorRef sender = getSender();
greetingService.greetAsync(greet.getMessage())
.thenAccept((greetingResponse) -> sender.tell(greetingResponse, getSelf()));
In your second example, you have
pipe(greetingService.greetAsync(greet.getMessage()), system.dispatcher()).to(getSelf());
You are piping the response to "getSelf()", i.e. your worker actor. The original sender will never get anything (thus the ask expires). You can fix that into:
pipe(greetingService.greetAsync(greet.getMessage()), system.dispatcher()).to(getSender());
In the third case, you have getSender() being executed synchronously during the processing of the message, thus it works.