We Keep Coding

Message duplication with PubSubInboundChannelAdapter in Spring Boot with Google PubSub

What I'm trying to do:
I am developing a Spring Boot application that should act as a subscriber for Google PubSub with "Exactly-once delivery" requirement. I have followed all the guidelines mentioned in the Google Cloud documentation for this:
https://cloud.google.com/pubsub/docs/exactly-once-delivery?_ga=2.207375771.-964146300.1676062320#pubsub_subscriber_exactly_once-java
However, instead of using the PubSub client library directly, I am using Spring Integration.
Problem:
The problem is that when I use the PubSubInboundChannelAdapter in my application, it is duplicating the same message multiple times with the same Id. My configuration class for PubSubInboundChannelAdapter is shown below:
#Slf4j
#Configuration
public class PubSubConfig {
#Value("${values.gcp.pubsub.subscription.name}")
private String subscriptionName;
/**
* This bean enables serialization/deserialization of Java objects to JSON allowing you
* utilize JSON message payloads in Cloud Pub/Sub.
*
* #param objectMapper the object mapper to use
* #return a Jackson message converter
*/
#Bean
public JacksonPubSubMessageConverter jacksonPubSubMessageConverter(ObjectMapper objectMapper) {
return new JacksonPubSubMessageConverter(objectMapper);
}
#Bean
public MessageChannel pubsubInputChannel() {
return new DirectChannel();
}
#Bean
public PubSubInboundChannelAdapter messageChannelAdapter(
#Qualifier("pubsubInputChannel") MessageChannel inputChannel,
PubSubTemplate pubSubTemplate) {
PubSubInboundChannelAdapter adapter =
new PubSubInboundChannelAdapter(pubSubTemplate, subscriptionName);
adapter.setOutputChannel(inputChannel);
adapter.setPayloadType(MyObjectThatNeedBeUnique.class);
adapter.setAckMode(AckMode.AUTO_ACK);
return adapter;
}
}
And, the listener for this is:
#Slf4j
#RequiredArgsConstructor(onConstructor = #__(#Autowired))
#Component
public class CreateVMListener {
#ServiceActivator(inputChannel = "pubsubInputChannel")
public void createVMListener(#Payload MyObjectThatNeedBeUnique payload,
#Header(GcpPubSubHeaders.ORIGINAL_MESSAGE) BasicAcknowledgeablePubsubMessage message)
throws IOException, ExecutionException, InterruptedException, TimeoutException {
log.info("Message arrived! Payload: " + payload.toString() + " | MessageId: " + message.getPubsubMessage().getMessageId() );
// Do some processing that takes 5 min to proccess
}
}
In the application.yml, I have configured the max-ack-extension-period to be 600 seconds and did the same in the Google Cloud PubSub dashboard:
Screenshot of the google cloud pubsub dashboard showing the Exactly once delivery configuration and the Acknowledgement deadline in 600 seconds
Log showing the duplication issue:
2023-02-05 23:22:20.055 [thread1] INFO - Message arrived! Payload: MyObjectThatNeedBeUnique (userId=432) | MessageID: 6846773022764035
2023-02-05 23:22:31.969 [thread2] INFO - Message arrived! Payload: MyObjectThatNeedBeUnique (userId=432) | MessageID: 6846773022764035
2023-02-05 23:23:33.028 [thread3] INFO - Message arrived! Payload: MyObjectThatNeedBeUnique (userId=432) | MessageID: 6846773022764035
2023-02-05 23:24:34.055 [thread4] INFO - Message arrived! Payload: MyObjectThatNeedBeUnique (userId=432) | MessageID: 6846773022764035
For example, in this log, the same message was repeated 4 times in a short period of time, indicating that 4 threads were simultaneously processing the same information.
Questions:
Why is this duplication happening?
What can I do to prevent duplications while retaining the use of the PubSubInboundChannelAdapter (which offers more efficient streaming pull)?
Additional Information: Identifying the Issue
To find the source of the problem, I tried multiple approaches and eventually discovered that the issue was with the PubSubInboundChannelAdapter. I switched to a synchronous option, the PubSubMessageSource (as the code below), which fixed the duplication of messages. However, this solution has a disadvantage as it is synchronous and not as performant as the PubSubInboundChannelAdapter. Because of this I want to know if there is a way to use PubSubInboundChannelAdapter without any duplication problem.
#Bean
#InboundChannelAdapter(channel = "pubsubInputChannel")
public MessageSource<Object> pubsubAdapter(PubSubTemplate pubSubTemplate) {
PubSubMessageSource messageSource = new PubSubMessageSource(pubSubTemplate, createVMSubscriptionName);
messageSource.setAckMode(AckMode.AUTO_ACK);
messageSource.setPayloadType(CreateOrStartVM.class);
messageSource.setBlockOnPull(true);
return messageSource;
}

After consulting with some peers who were not experiencing the issue on their machines, I realized the problem was in the IntelliJ debug configuration.
The reason why the same message is being duplicated multiple times with the same ID is because the PubSubInboundChannelAdapter is a Stream, and in order to make the Reactive Stream Debugger work properly without bugs, you need to enable Hooks.onOperatorDebug() in IntelliJ:
Image showing the Reactive Stream configuration in IntelliJ
After some extra search, I found that the PubSubInboundChannelAdapter uses Reactive Stream in the background, and to debug reactive streams, you need a tool that can visualize and inspect the events flowing through it.

Can Spring Integration be used in Mutli Server Environment

Can we use Spring Integration to configure directory polling for files such that -
With 2 servers configured, polling occurs on 1 server and corresponding processing get distributed b/w both the servers.
Also, can we switch the polling on either of the servers on runtime ?
Edit -
Tried configuring JBDC MetaStore and run the two instances separately, able to poll and process but getting intermittently DeadLockLoserDataAccessException
Configuration below
#Bean
public MessageChannel fileInputChannel(){
return new DirectChannel();
}
#Bean(PollerMetadata.DEFAULT_POLLER)
public PollerMetadata defaultPoller(){
PollerMetadata pollermetadata = new PollerMetadata();
pollermetadata.setMaxMessagesPerPoll(-1);
pollermetadata.setTrigger(new PeriodicTrigger(1000));
return pollermetadata;
}
#Bean
#InBoundChannelAdapter(value = "fileInputChannel"){
FileReadingMessageSource source = new FileReadingMessageSource();
source.setDirectory("Mylocalpath");
FileSystemPersistentAcceptOnceFileListFilter acceptOnce = new FileSystemPersistentAcceptOnceFileListFilter();
ChainFileListFilter<File> chainFilter = ChainFileListFilter(".*\\.txt"));
chainFilter.addFilter(acceptOnce);
source.setFilter(chainFilter);
source.setUseWatchService(true);
source.setWatchEvents(FileReadingMessageSource.WatchEventType.CREATE,FileReadingMessageSource.WatchEventType.MODIFY);
return source;
}
#Bean
public IntegrationFlow processFileFlow(){
return IntegrationFlows.from("fileInputChannel")
.handle(service).get();
}

It is really one of the features of Spring Integration to easy implement a distributed solution. You just need add a messaging middle ware into your cluster infrastructure and have all the nodes connected to some destination for sending and receiving. A good example could be a SubscribableJmsChannel which you just simple can declare in your application context and all the nodes of your cluster are going to subscribe to this channel for round-robin consumption from JMS queue. It already wouldn't matter which node produces to this channel.
See more in docs: https://docs.spring.io/spring-integration/docs/current/reference/html/jms.html#jms-channel.
Another sample of similar distributed channels are: AMQP, Kafka, Redis, ZeroMQ.
You also can have a shared message store and use it in the QueueChannel definition: https://docs.spring.io/spring-integration/docs/current/reference/html/system-management.html#message-store
It is not clear what you mean about "poller on runtime", so I would suggest you to start a new SO thread with much more info.
See rules as a guidance: https://stackoverflow.com/help/how-to-ask

Spring Cloud #StreamListener condition deprecated what is the alternative

We have multiple applications consumer listening to the same kafka topic and a producer sets the message header when sending message to the topic so specific instance can evaluate the header and process the message. eg
#StreamListener(target=ITestSink.CHANNEL_NAME,condition="headers['franchiseName'] == 'sydney'")
public void fullfillOrder(#Payload TestObj message) {
log.info("sydney order request received message is {}",message.getName());
}
In Spring Cloud Stream 3.0.0 the #StreamListener is deprecated and I could not find the equivalent of the condition property in Function.
Any suggestion?

Though I was not able to find the equivalent for the functional approach either, I do have a suggestion.
The #StreamListener annotations condition does not stop the fact that the application must consume the message, read its header, and filter out specific records before passing it to the listener (fullfillOrder()). So it's safe to assume you're consuming every message that hits the topic regardless (by the event receiver that Spring Cloud has implemented for us under the hood), but the listener only gets executed when header == sydney.
If there was a way to configure the event receiver that Spring Cloud uses (to discard message before hitting listener), I would suggest looking into that. If not, would resort to filtering out any messages (non-sydney) before doing any processing. If you're familiar with Spring Cloud's functional approach, would look something like this:
#Bean
public Consumer<Message<TestObj>> fulfillOrder() {
return msg -> {
// to get header - msg.getHeaders().get(key, valueType);
// filter out bad messages
}
}
or
#Bean
public Consumer<ConsumerRecord<?, TestObj>> fulfillOrder() {
return msg -> {
// msg.headers().lastHeader("franchiseName").value() -> filter em out
}
}
Other:
^ my code assumes you're integrating the kafka-client API with Spring cloud stream via spring-cloud-stream-binder-kafka. based on tags listed, i will note Spring Cloud Stream has two versions of binders for Kafka - one for the kafka client library, and one for kafka streams library.
Without considering Spring Cloud / Frameworks, the high-lvl DSL in kafka streams doesn't give you access to headers, but the low-level Processor API does. From the example, it seems like you're leveraging the client binder and not spring-cloud-stream-binder-kafka-streams / kafka streams binder. I haven't seen an implementation of spring cloud stream + kafka streams binder using the low-level processor API, so i can't tell if that was the aim.

Spingboot Websocket Stomp

Could anyone tell me if the server-side implementation is using stomp WebSocket, is the client also expected to implement stomp?
I am trying to implement a spring boot application and I am confused if I should go with or without stomp implementation. From my research, I understand, if you want to scale the application, it is better to use stomp and embedded broker( RabbitMQ for eg.) as it will handle the sessions, heartbeat etc. instead of an in-memory broker.
The examples available online just shows implementations with and without stomp.
I am basically trying to get different datasets from the table upon client request and write to a WebSocket continuously.
Could anyone please confirm if my understanding so far is correct?
What are the essential things I will have to take care of if I go with stomp + websocket?
Updating the usecase below:
The mobile client would be displaying charts upon user login. There would be links in the left panel for eg. Sales, Discounts etc. which upon clicking, the request will reach server through websocket channel. Server will check the datatype in the request, generate the model using data from DB and write the data to the websocket.
Updating code - v1
MyWebSocketHandler:
#Component
public class MyWebSocketHandler extends TextWebSocketHandler {
Logger logger = LoggerFactory.getLogger(getClass());
#Autowired
DashboardUtil dashboardutil;
#Resource(name = "socketSessionsMap")
private Map<String, WebSocketSession> socketSessionsMap;
#Override
public void handleTextMessage(WebSocketSession session, TextMessage message)
throws InterruptedException, IOException {
try {
//Gets the socket session from map and writes a json to that socket - did for testing purpose.
socketSessionsMap.put("session", session);
//String payload = message.getPayload();
String jsonString = dashboardutil.getDataInJSON(); // gets hardcoded json model
session.sendMessage(new TextMessage(jsonString));
} catch (Exception e) {
e.printStackTrace();
}
}
#Override
public void afterConnectionEstablished(WebSocketSession session) throws Exception {
}
#Override
public void afterConnectionClosed(WebSocketSession session, CloseStatus status) throws Exception {
}
}
WebSecurityConfig:
#Configuration
#EnableWebSocket
public class WebSocketConfig implements WebSocketConfigurer {
#Autowired
private MyWebSocketHandler myWebSocketHandler;
#Override
public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
registry.addHandler(myWebSocketHandler, "/socketHandler").setAllowedOrigins("*").withSockJS();
}
}

Could anyone tell me if the server-side implementation is using stomp
WebSocket, is the client also expected to implement stomp?
You can register multiple handlers in your web socket configuration. So in theory you can provide a handler for STOMP and another one for plain web socket. If you only provide a STOMP handler then the handshake from a standard web socket client will fail.
From my research, I understand, if you want to scale the application,
it is better to use stomp and embedded broker( RabbitMQ for eg.) as it
will handle the sessions, heartbeat etc. instead of an in-memory
broker.
That's correct. STOMP also offers a few more nice features especially the subscription to certain endpoints.
I am basically trying to get different datasets from the table upon
client request and write to a WebSocket continuously.
That's a really simple description ...
You should think about if you need to share sessions across multiple instances or if you need to send messages between web socket sessions.
From your description it sounds like you just accept a web socket connection and continuously push data to the client. If you want to scale this application you can just put a load balancer in front of your instances and you are good to go.

Using Spring Cloud Stream Source to send method results to stream

I'm trying to create a Spring Cloud Stream Source Bean inside a Spring Boot Application that simply sends the results of a method to a stream (underlying Kafka topic is bound to the stream).
Most of the Stream samples I've seen use #InboundChannelAdapter annotation to send data to the stream using a poller. But I don't want to use a poller. I've tried setting the poller to an empty array but the other problem is that when using #InboundChannelAdapter you are unable to have any method parameters.
The overall concept of what I am trying to do is read from an inbound stream. Do some async processing, then post the result to an outbound stream. So using a processor doesn't seem to be an option either. I am using #StreamListener with a Sink channel to read the inbound stream and that works.
Here is some code i've been trying but this doesn't work at all. I was hoping it would be this simple because my Sink was but maybe it isn't. Looking for someone to point me to an example of a source that isn't a Processor (i.e. doesn't require listening on an inbound channel) and doesn't use #InboundChannelAdapter or to give me some design tips to accomplish what I need to do in a different way. Thanks!
#EnableBinding(Source.class)
public class JobForwarder {
#ServiceActivator(outputChannel = Source.OUTPUT)
#SendTo(Source.OUTPUT)
public String forwardJob(String message) {
log.info(String.format("Forwarding a job message [%s] to queue [%s]", message, Source.OUTPUT));
return message;
}
}

Your orginal requirement can be achieved through the below steps.
Create your custom Bound Interface (you can use the default #EnableBinding(Source.class) as well)
public interface CustomSource {
String OUTPUT = "customoutput";
#Output(CustomSource.OUTPUT)
MessageChannel output();
}
Inject your bound channel
#Component
#EnableBinding(CustomSource.class)
public class CustomOutputEventSource {
#Autowired
private CustomSource customSource;
public void sendMessage(String message) {
customSource.output().send(MessageBuilder.withPayload(message).build());
}
}
Test it
#RunWith(SpringRunner.class)
#SpringBootTest
public class CustomOutputEventSourceTest {
#Autowired
CustomOutputEventSource output;
#Test
public void sendMessage() {
output.sendMessage("Test message from JUnit test");
}
}

So if you don't want to use a Poller, what causes the forwardJob() method to be called?
You can't just call the method and expect the result to go to the output channel.
With your current configuration, you need an inputChannel on the service containing your inbound message (and something to send a message to that channel). It doesn't have to be bound to a transport; it can be a simple MessageChannel #Bean.
Or, you could use a #Publisher to publish the result of the method invocation (as well as being returned to the caller) - docs here.
#Publisher(channel = Source.OUTPUT)

Thanks for the input. It took me a while to get back to the problem. I did try reading the documentation for #Publisher. It looked to be exactly what I needed but I just couldn't get the proper beans initialized to get it wired properly.
To answer your question the forwardJob() method is called after some async processing of the input.
Eventually I just implemented using spring-kafka library directly and that was much more explicit and felt easier to get going. I think we are going to stick to kafka as the only channel binding so I think we'll stick with that library.
However, we did eventually get the spring-cloud-stream library working quite simply. Here was the code for a single source without a poller.
#Component
#EnableBinding(Source.class)
public class JobForwarder {
private Source source;
#Autowired
public ScheduledJobForwarder(Source source) {
this.source = source;
}
public void forwardScheduledJob(String message) {
log.info(String.format("Forwarding a job message [%s] to queue [%s]", message, Source.OUTPUT));
source.output().send(MessageBuilder.withPayload(message).build());
}
}