I have started using lagom recently. Trying out a microservice where I receive a kafka message and after some processing publish another message to a different kafka topic. Based on this link my understanding is, a message should be published on the constructed topic - especially this part of the sample code I am referring to.
final PubSubRef<Temperature> topic = pubSub.refFor(TopicId.of(Temperature.class, id));
topic.publish(temperature);
I couldn’t build Temperature DTO to POST from rest client. So I created my on DTO which is exactly similar to HelloEvent - in my case its KafkaEvent.
I tried to use the code from here
However I did not see the topic created after performing POST operation. I did add print statements and they do appear in console.
System.out.println("Received id:" + id);
final PubSubRef<KafkaEvent> topic = pubSub.refFor(TopicId.of(KafkaEvent.class, id));
topic.publish(temperature);
System.out.println("Sent to:" + topic.toString());
I am not seeing any error in kafka server log or in my project.
Is there any step I am missing? or my understanding is wrong in usage of PubSubRegistry?
Please do let me know if further details are required.
Thanks in advance
Naveena
If you want to use Kafka, you are using incorrect approach. This post that you described does not use Kafka. It just broadcast messages to all subscribers. If you want to use Kafka you need to use message broker support, it will create what you want. Please read section limitations, it will give you mire information.
Related
I have a Spring boot application (v2.2.10.RELEASE) that subscribes to multiple topics in pubSub and pulls async data and sends it to somewhere else. I am not using SpringGCP, just native google libraries
this is my subscriber setting:
// Instantiate an asynchronous message receiver.
MessageReceiver receiver =
(PubsubMessage message, AckReplyConsumer consumer) -> {
messages.add(message);
consumer.ack();
};
Subscriber subscriber = Subscriber.newBuilder(subscriptionName, receiver)
.setParallelPullCount(2)
.setFlowControlSettings(flowControlSettings)
.setCredentialsProvider(credentialsProvider)
.setExecutorProvider(executorProvider)
//.setChannelProvider()
.build();
With high traffic and big messages (2 - 4 kb) I encounter this info message:
[grpc-default-worker-ELG-1-1] INFO i.grpc.internal.AbstractClientStream - Received data on closed stream
first of all, I don't fully understand what that means? all that I noticed was that when this happens the delivered duplicated messages increase. so I assumed it meant that pubSub tried to reach the subscriber with some messages but the subscriber for some reason was not ready so pubSub will try to deliver the messages again. and hence more duplicates, is that right?
would this problem be solved using the TransportChannelProvider in subscribers? my understanding of the poorly written documentation, that this will create a new channel for delivery when the current in-use channel is closed, hence get rid of the previous log message.
if yes, how do I define the channel target string? and where can I find A NameResolver-compliant URI for the mangagedChannel. the snippet I mean is this:
private TransportChannelProvider getChannelProvider() {
ManagedChannel channel = ManagedChannelBuilder.forTarget(target).usePlaintext(true).build();
return FixedTransportChannelProvider.create(GrpcTransportChannel.create(channel));
}
I am pretty new to GCP so sorry if my question is not coherent enough
Using a custom TransportChannelProvider won't solve this type of issue. This is more likely an issue deeper down in the stack, e.g., at the gRPC level. There have been some open issues for this type of error [1, 2].
With regard to why it is causing duplicates, it is possible that the messages are getting delivered via a stream that is already closed (which aligns with the error message) because they were trapped in a lower-level buffer at the gRPC layer and therefore ended up being duplicates of messages that were subsequently delivered and processed via another stream. This could be a version of the issue discussed in the documentation around large backlogs of small messages. There was a fix for this issue in v1.109.0 of the Java client library, so if you are using a version older than that, it is worth updating.
If duplicates continue to be an issue, it would be best to reach out to support with the name of your subscription and the message IDs of some of the duplicate messages so that they can look at the delivery patterns for those messages and further diagnose if these redeliveries are unexpected.
We have a streams application that consumes messages from a source topic, does some processing and forward the results to a destination topic.
The structure of the messages are controlled by some avro schemas.
When starting consuming messages if the schema is not cached yet the application will try to retrieve it from schema registry. If for whichever reason the schema registry is not available (say a network glitch) then the currently being processed message is lost because the default handler is something called LogAndContinueExceptionHandler.
o.a.k.s.e.LogAndContinueExceptionHandler : Exception caught during Deserialization, taskId: 1_5, topic: my.topic.v1, partition: 5, offset: 142768
org.apache.kafka.common.errors.SerializationException: Error retrieving Avro schema for id 62
Caused by: java.net.SocketTimeoutException: connect timed out
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:na]
...
o.a.k.s.p.internals.RecordDeserializer : stream-thread [my-app-StreamThread-3] task [1_5] Skipping record due to deserialization error. topic=[my.topic.v1] partition=[5] offset=[142768]
...
So my question is what would be the proper way of dealing with situations like described above and make sure you don't lose messages no matter what. Is there an out of the box LogAndRollbackExceptionHandler error handler or a way of implementing your own?
Thank you in advance for your inputs.
I've not worked a lot on Kafka, but when i did, i remember having issues such as the one you are describing in our system.
Let me tell you how we took care of our scenarios, maybe it would help you out too:
Scenario 1: If your messages are being lost at the publishing side (publisher --> kafka), you can configure Kafka acknowledgement setting according to your need, if you use spring cloud stream with kafka, the property is spring.cloud.stream.kafka.binder.required-acks
Possible values:
At most once (Ack=0)
Publisher does not care if Kafka acknowledges or not.
Send and forget
Data loss is possible
At least once (Ack=1)
If Kafka does not acknowledge, publisher resends message.
Possible duplication.
Acknowledgment is sent before message is copied to replicas.
Exactly once (Ack=all)
If Kafka does not acknowledge, publisher resends message.
However, if a message gets sent more than once to Kafka, there is no duplication.
Internal sequence number, used to decide if message has already been written on topic or not.
Min.insync.replicas property needs to be set to ensure what is the minimum number of replices that need to be synced before kafka acknowledges to the producer.
Scenario 2: If your data is being lost at the consumer side (kafka --> consumer), you can change the Auto Commit feature of Kafka according to your usage. This is the property if you are using Spring cloud stream spring.cloud.stream.kafka.bindings.input.consumer.AutoCommitOffset.
By default, AutoCommitOffset is true in kafka, and every message that is sent to the consumer is "committed" at Kafka's end, meaning it wont be sent again. However if you change AutoCommitOffset to false, you will have the power to poll the message from kafka in your code, and once you are done with your work, explicitly set commit to true to let kafka know that now you are done with the message.
If a message is not committed, kafka will keep resending it until it is.
Hope this helps you out, or atleast points you in the right direction.
The idea is to send messages to topic and consume them in stripes via pub/sub only (without queues) and using PERSISTENT delivery mode. For simplicity, lets say producer(s) publish messages to specific topics having the following hierarchy: bus/<componentId/<transactionId>.
Consumers want to receive topic "stripes", for simplicity lets say there are 10 consumers and they want to stripe traffic among themselves by transactionId:
bus/*/0*
bus/*/1*
...
bus/*/9*
When I try to subscribe to topic endpoint using wildcards, like this:
DurableTopicEndpoint topicEndpoint = JCSMPFactory.onlyInstance().createDurableTopicEndpoint("bus/*/1*");
ConsumerFlowProperties propsFlow = new ConsumerFlowProperties();
propsFlow.setEndpoint(topicEndpoint);
I get the following exception:
Exception in thread "main" java.lang.IllegalArgumentException: Topic Endpoint name "bus/*/1*" contains illegal character [*]
at com.solacesystems.common.util.DestinationUtil.isValidEndpointName(DestinationUtil.java:234)
at com.solacesystems.common.util.DestinationUtil.isValidTopicEndpointPhysicalName(DestinationUtil.java:209)
at com.solacesystems.common.util.DestinationUtil.isValidDTEPhysicalName(DestinationUtil.java:213)
at com.solacesystems.jcsmp.impl.SessionModeSupport.createFlow(SessionModeSupport.java:247)
at com.solacesystems.jcsmp.impl.SessionModeSupport.createFlow(SessionModeSupport.java:170)
at com.solacesystems.jcsmp.impl.JCSMPBasicSession.createFlow(JCSMPBasicSession.java:953)
In the light of this article's section "Adding Subscriptions to Topic Endpoints" - is it at all possible with Solace Java API?
There are two problems here.
You are trying to create an TopicEndpoint named bus/*/1*. Note that this is the name of the TopicEndpoint and not the topic that it is subscribing to. * is not a valid character for the name of a TopicEndpoint.
TopicEndpoints are only allowed to have one subscription. This means that you can only subscribe to bus/*/0*. If you want to subscribe to bus/*/0* all the way to bus/*/9* you will need to make use of a Queue instead of a TopicEndpoint.
I am implementing spring-integration based application where i need to send messages (POJO) to ActiveMQ queues dynamically (based on message params). What is the best way to implement this using spring integration concepts ?
E.g., The request message (id:123, processor:ABC) should be sent to ABC.REQUEST queue and response message should be received from ABC.RESPONSE queue. Similarly the request message (id:456, processor:XYZ) should be sent to XYZ.REQUEST queue and response message should be received from XYZ.RESPONSE queue.
You don't need to programmatically create them programmatically, you can use dynamic destinations.
See request-destination-expression and reply-destination-expression in the documentation.
Something like...
request-destination-expression="payload.processor + '.REQUEST'"
reply-destination-expression="payload.processor + '.RESPONSE'"
...should do it.
I am building a small api around the JMS API for a project of mine. Essentially, we are building code that will handle the connection logic, and will simplify publishing messages by providing a method like Client.send(String message).
One of the ideas being discussed right now is that we provide a means for the users to attach interceptors to this client. We will apply the interceptors after preparing the JMS message and before publishing it.
For example, if we want to timestamp a message and wrote an interceptor for that, then this is how we would apply that
...some code ...
Message message = session.createMessage()
..do all the current processing on the message and set the body
for(interceptor:listOfInterceptors){
interceptor.apply(message)
}
One of the intrerceptors we though of was to compress the message body. But when we try to read the body of the message in the interceptor, we are getting a MessageNotReadableException. In the past, I normally compressed the content before setting it as the body of the message - so never had to worry about this exception.
Is there any way of getting around this exception?
It looks like your JMS client attempts to read a write-only message. Your interceptor cannot work this way, please elaborate how you were compressing message earlier.