We Keep Coding

Message transfer in between two topics in google cloud pub sub

We have a use case where on any action from UI we need to read messages from google pub/sub Topic A synchronously and move those messages to Topic B.
Below is the code that has been written to handle this behavior and this is from Google Pub Sub docs to access a Topic synchronusly.
public static int subscribeSync(String projectId, String subscriptionId, Integer numOfMessages, int count, String acknowledgementTopic) throws IOException {
SubscriberStubSettings subscriberStubSettings =
SubscriberStubSettings.newBuilder()
.setTransportChannelProvider(
SubscriberStubSettings.defaultGrpcTransportProviderBuilder()
.setMaxInboundMessageSize(20 * 1024 * 1024) // 20MB (maximum message size).
.build())
.build();
try (SubscriberStub subscriber = GrpcSubscriberStub.create(subscriberStubSettings)) {
String subscriptionName = ProjectSubscriptionName.format(projectId, subscriptionId);
PullRequest pullRequest =
PullRequest.newBuilder()
.setMaxMessages(numOfMessages)
.setSubscription(subscriptionName)
.build();
// Use pullCallable().futureCall to asynchronously perform this operation.
PullResponse pullResponse = subscriber.pullCallable().call(pullRequest);
List<String> ackIds = new ArrayList<>();
for (ReceivedMessage message : pullResponse.getReceivedMessagesList()) {
// START - CODE TO PUBLISH MESSAGE TO TOPIC B
**publishMessage(message.getMessage(),acknowledgementTopic,projectId);**
// END - CODE TO PUBLISH MESSAGE TO TOPIC B
ackIds.add(message.getAckId());
}
// Acknowledge received messages.
AcknowledgeRequest acknowledgeRequest =
AcknowledgeRequest.newBuilder()
.setSubscription(subscriptionName)
.addAllAckIds(ackIds)
.build();
// Use acknowledgeCallable().futureCall to asynchronously perform this operation.
subscriber.acknowledgeCallable().call(acknowledgeRequest);
count=pullResponse.getReceivedMessagesList().size();
}catch(Exception e) {
log.error(e.getMessage());
}
return count;
}
Below is the sample code to publish messages to Topic B
public static void publishMessage(PubsubMessage pubsubMessage,String Topic,String projectId) {
Publisher publisher = null;
ProjectTopicName topicName =ProjectTopicName.newBuilder().setProject(projectId).setTopic(Topic).build();
try {
// Publish the messages to normal topic.
publisher = Publisher.newBuilder(topicName).build();
} catch (IOException e) {
log.error(e.getMessage());
}
publisher.publish(pubsubMessage);
}
Is this the right way of handling this use case or this can be handled in someother way. We do not want to use Cloud Dataflow. Can someone let us know if this is fine or there is an issue.
The code works but sometimes messages stay on Topic A even after hey are consumed synchronously.
Thanks'

There are some issues with the code as presented.
You should really only use synchronous pull if there are specific reasons why you need to do so. In general, it is much better to use asynchronous pull via the client libraries. It will be more efficient and reduce the latency of moving messages from one topic to the other. You do not show how you call subscribeSync, but in order to process messages efficiently and ensure that you actually process all messages, you'd need to be calling it many times in parallel continuously. If you are going to stick with synchronous pull, then you should reuse the SubscriberStub object as recreating it for every call will be inefficient.
You don't reuse your Publisher object. As a result, you are not able to take advantage of the batching that the publisher client can do. You should create the Publisher once and reuse it across your calls for publishes to the same topic. If the passed-in topic can differ across messages, then keep a map from topic to publisher and retrieve the right one from the map.
You don't wait for the result of the call to publish. It is possible that this call fails, but you do not handle that failure. As a result, you could acknowledge the message on the first topic without it having actually been published, resulting in message loss.
With regard to your question about duplicates, Pub/Sub offers at-least-once delivery guarantees, so even with proper acking, it is still possible to receive messages again (typical duplicate rates are around 0.1%). There can be many different reasons for duplicates. In your case, since you are processing messages sequentially and recreating a publisher for every call, it could be that later messages are not acked before the ack deadline expires, which results in redelivery.

JMS 2.0 - MQ 9 - Topic Shared subscription doesn't work

I'm facing problem when developing an application that subscribe a MQ Topic (MQ version 9).
I need to do a shared topic connection because the application will be ran in multiple instances (cluster).
The specs and the documentation says :
"A non-durable shared subscription is used by a client which needs to be able to share the work of receiving messages from a topic subscription amongst multiple consumers. A non-durable shared subscription may therefore have more than one consumer. Each message from the subscription will be delivered to only one of the consumers on that subscription."
For me, all the clients using the same subscription name are in the same "cluster", only one client will receive a message at one time.
In my code, inspired by this article, I've got an exception when the second client try to create the shared subscription. I really don't understand if this is a bug in MQ client libraries implementation or in my code.
Here my sample code :
import javax.jms.Connection;
import javax.jms.Message;
import javax.jms.MessageConsumer;
import javax.jms.Session;
import javax.jms.Topic;
import com.ibm.mq.jms.MQTopicConnectionFactory;
import com.ibm.msg.client.wmq.WMQConstants;
public class TestGB2 {
public static void main(final String[] args) throws Exception {
for (int i = 0; i < 10; i++) {
new Thread(new MyThread("THREAD" + i, "TESTSUB/#", "myClient", "SUBTEST")).start();
}
}
public static class MyThread implements Runnable {
private final String topicString;
private final String clientId;
private final String subscriptionName;
public MyThread(final String threadName, final String topicString, final String clientId, final String subscriptionName) {
Thread.currentThread().setName(threadName);
this.topicString = topicString;
this.clientId = clientId;
this.subscriptionName = subscriptionName;
}
#Override
public void run() {
try {
System.out.println(String.format("%s : Connecting...", Thread.currentThread().getName()));
MQTopicConnectionFactory cf = new MQTopicConnectionFactory();
cf.setHostName("xxxx");
cf.setPort(1416);
cf.setQueueManager("xxxx");
cf.setTransportType(WMQConstants.WMQ_CM_CLIENT);
cf.setChannel("xxx");
cf.setClientID(clientId);
Connection con = cf.createConnection();
Session session = con.createSession(false, Session.AUTO_ACKNOWLEDGE);
con.start();
Topic topic = session.createTopic(topicString);
MessageConsumer messageConsumer = session.createSharedConsumer(topic, subscriptionName); // fail here
System.out.println(String.format("%s : Waiting for a message...", Thread.currentThread().getName()));
Message msg = messageConsumer.receive();
System.out.println(String.format("%s : Received :\n%s", Thread.currentThread().getName(), msg));
}
catch (Exception ex) {
System.out.println(String.format("%s : FAILED", Thread.currentThread().getName()));
ex.printStackTrace();
}
}
}
}
The code below tries to create 10 threads consuming messages on the same topic. Only the first thread is able to connect, all the others fail with following exception :
com.ibm.msg.client.jms.DetailedIllegalStateException: JMSWMQ0026: Failed to subscribe to topic 'TESTSUB' with selector 'none' using MQSUB.
There may have been a problem creating the subscription due to it being used by another message consumer.
Make sure any message consumers using this subscription are closed before trying to create a new subscription under the same name. Please see the linked exception for more information.
at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:472)
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:214)
at com.ibm.msg.client.wmq.internal.WMQMessageConsumer.checkJmqiCallSuccess(WMQMessageConsumer.java:212)
at com.ibm.msg.client.wmq.internal.WMQMessageConsumer.checkJmqiCallSuccess(WMQMessageConsumer.java:112)
at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.initialize(WMQConsumerShadow.java:1038)
at com.ibm.msg.client.wmq.internal.WMQSyncConsumerShadow.initialize(WMQSyncConsumerShadow.java:134)
at com.ibm.msg.client.wmq.internal.WMQMessageConsumer.<init>(WMQMessageConsumer.java:470)
at com.ibm.msg.client.wmq.internal.WMQSession.createSharedConsumer(WMQSession.java:938)
at com.ibm.msg.client.jms.internal.JmsSessionImpl.createSharedConsumer(JmsSessionImpl.java:4228)
at com.ibm.msg.client.jms.internal.JmsSessionImpl.createSharedConsumer(JmsSessionImpl.java:4125)
at com.ibm.mq.jms.MQSession.createSharedConsumer(MQSession.java:1319)
at TestGB.lambda$0(TestGB.java:33)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2042' ('MQRC_OBJECT_IN_USE').
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:202)
... 11 more
Tried with the last lib :
<dependency>
<groupId>com.ibm.mq</groupId>
<artifactId>com.ibm.mq.allclient</artifactId>
<version>9.1.1.0</version>
</dependency>

Summary of the issue
The issue is not with your program, the issue is with the model queue associated to the topic you are subscribing to.
On the queue manager if you look at the topic object that your subscription will match, it will have a parameter MNDURMDL that points to a model queue.
If you look at the model queue you will note two parameters where either or both can cause the error you are receiving:
[ DEFSOPT( EXCL | SHARED ) ]
[ SHARE | NOSHARE ]
These must be set to DEFSOPT(SHARED) and SHARE. If either one is set to the other value you will only be able to have one subscriber on the shared subscription.
Additional details of the cause of the issue
With IBM MQ Pub/Sub, when you create a JMS subscription MQ treats this as a managed subscription, in the background IBM MQ will create a temporary queue to subscribe to the topic string. If it is a non-durable subscription the queue is a temporary dynamic queue.
The reason for the failure is that the first thread will open the temporary dynamic queue in an exclusive mode, any other threads then cannot open the temporary dynamic queue and you receive the MQRC_OBJECT_IN_USE error.
Possible cause where an application specific MNDURMDL model queue was created
I suspect the cause of this is that IBM comes with a few different default model queues.
The default for a non-durable subscriber has these settings:
QUEUE(SYSTEM.NDURABLE.MODEL.QUEUE) TYPE(QMODEL)
DEFSOPT(SHARED) SHARE
There is another default queue that is not pub/sub specific that has these settings:
QUEUE(SYSTEM.DEFAULT.MODEL.QUEUE) TYPE(QMODEL)
DEFSOPT(EXCL) NOSHARE
It is likely that the model queue created for use by your topic object was created with a command like the following that will default to use the setting of the SYSTEM.DEFAULT.MODEL.QUEUE.:
DEFINE QMODEL(xxx)
In the future you could either specifically set those two parameters, or define it with the LIKE keyword to force it to use a different queue to model settings from, both commands are below:
DEFINE QMODEL(xxx) DEFSOPT(SHARED) SHARE
DEFINE QMODEL(xxx) LIKE(SYSTEM.NDURABLE.MODEL.QUEUE)
Additional details on creation and usage of application specific TOPIC objects and MODEL queues
By default the root node of the tree is represented by the standard TOPIC object named SYSTEM.BASE.TOPIC, the default model queues associated to this TOPIC are shown below:
TOPIC(SYSTEM.BASE.TOPIC) TYPE(LOCAL)
TOPICSTR() MDURMDL(SYSTEM.DURABLE.MODEL.QUEUE)
MNDURMDL(SYSTEM.NDURABLE.MODEL.QUEUE)
If you do not define any further administrative TOPIC objects, then all topics match against SYSTEM.BASE.TOPIC. Additionally if you do not define any further administrative TOPIC objects and you want to give an application permission to a specific subset of the topic tree (for example topic strings beginning with TESTSUB) you must grant the permissions via SYSTEM.BASE.TOPIC, this in turn grants the application access for any arbitrary topic string with no restrictions.
Best practice would be to create a TOPIC object with a topic string that matches the portion of the topic tree an an application should have access to. Specific to your example of TESTSUB/# if your admin defined a new TOPIC object and specified the TOPICSTR(TESTSUB), the defaults would create it like this:
TOPIC(TESTSUB.TOPIC) TYPE(LOCAL)
TOPICSTR(TESTSUB) MDURMDL( )
MNDURMDL( )
the blank MDURMDL and MNDURMDL values tell MQ to use the value from the next closest higher topic object in the tree, if nothing else is defined this would be the SYSTEM.BASE.TOPIC and the model queues would still default to using the SYSTEM.DURABLE.MODEL.QUEUE and SYSTEM.NDURABLE.MODEL.QUEUE model queues.
The admin can instead create the TOPIC object and specify different model queues for example:
TOPIC(TESTSUB.TOPIC) TYPE(LOCAL)
TOPICSTR(TESTSUB) MDURMDL(TESTSUB.DURABLE.MODEL.QUEUE)
MNDURMDL(TESTSUB.NDURABLE.MODEL.QUEUE)
By doing this they can define application specific model queues that have the settings required for shared subscriptions and not impact the SYSTEM model queues. The other benefit is they can provide the application permissions for only topic strings that start with TESTSUB, for example TESTSUB/A or TESTSUB/B or TESTSUB/X/Y/Z.

Spring boot create a topic and publish so APPS can receive it

I'm creating an app that there's a button that says "Join the room" and when you click on it you join an imaginary room where you can see more users entering.
My idea is to create a topic for instance nameRoom and every time a user joins the room it automatically subscribe to them, so if there's another update he/she'll receive the update for instance one joined or one left.
The thing where I'm stuck is:
The "Administrator" can create a "Room" so every time the Administrator creates a Room should it be a new topic, right? So, my question is once I'm inside a room, I'd like to create like a countdown let's say 30 seconds, and when those 30 seconds are done, it starts to ask me questions and everyone can answer the question and I need to see how many users have answered and how many does not, this is another topic?
The flow is :
Administrator creates a room --> Room1
User1 joins the Room1 and sees only you are in this room
User2 joins the Room and sees there are 2 guys in this room (And so on until user 5)
Then the timer goes down 30 to 0
Then as a User1 I see "How old is Michael Jordan" and 4 checkbox and everyone can answer
Also there's a field saying how many answers already have been posted so if the time of the question is 30 seconds and they are 5 users and they answer in less than 30 seconds the question has to be moved to the next screen that is
Top 3 who answered faster and score
To be clear :
I'd like to know how to create topic and then publish to them using Spring. For example to create the room is not necessary MQTT but to check who joins and this stuff it is, so I'm asking this, how could I create this with MQTT?
Also, MQTT would be responsible to say all of the info? I mean every room has some questions so it's necessary to via MQTT know the ranking etc?

1) You need to create database that suits your application needs
Database name : ChatRoom
Tables:
Topics(To store all mqtt room topic names)
Room (Each room is associated with a topic)
User (Each user is associated with a room)
2) Setup an mqtt server which allows connection both on mqtt and websockets (To support javascript application)
3) Now create a spring boot application with following api and web pages
Web pages:
chatroom.html
chatroom.js
Api:
create chatroom (for admin)
list chatroom
join chatroom (for new users)
Steps:
The admin uses the chatroom.html web page to create new chatroom.On creation it calls create chatroom api to create new chatroom. In the api it will subscribe to a new topic for new chatroom.
When normal users visit chatroom.html it will list available chatrooms (use list chatroom api). Once you click on chatroom it will call join chatroom api to update the details in database. The javascript from browser can connect to mqtt topic for the specific chatroom.
For managing questions you need to have your own logic in backend as well as frontend side. You can use Mqtt topic for transferring messages
Refer the following links
For javascript
http://www.steves-internet-guide.com/using-javascript-mqtt-client-websockets/
For java
https://www.eclipse.org/paho/clients/java/
public final class MessageQueueClient implements MqttCallback
{
private MqttClient mqttClient;
private MessageQueueClient()
{
}
public static MessageQueueClient getInstance()
{
return messageQueueClient;
}
#Override
public void connectionLost(Throwable cause)
{
}
#Override
public void messageArrived(String topic, MqttMessage message)
{
}
#Override
public void deliveryComplete(IMqttDeliveryToken token)
{
}
//Call this method on server startup to connect to mqtt server(spring boot app start)
public boolean connect(String hostname, String clientuniqueid)
{
try
{
if (mqttCredentialsDTO != null)
{
MqttConnectOptions options = new MqttConnectOptions();
options.setAutomaticReconnect(true);
options.setCleanSession(true);
mqttClient = new MqttClient(hostname, clientuniqueid);
mqttClient.connect(options);
return true;
}
}
catch (Exception e)
{
e.printStacktrace();
}
return false;
}
//Call this method on server shutdown to disconnect from mqtt server
public boolean disconnect()
{
try
{
if (mqttClient != null)
{
mqttClient.disconnect();
mqttClient.close();
return true;
}
}
catch (MqttException e)
{
e.printStacktrace();
}
return false;
}
//call this method after mqtt connection established to subscribe to any topic
public boolean subscribe(String topicName, int qos)
{
try
{
if (topicName != null)
{
mqttClient.subscribe(topicName, qos);
return true;
}
}
catch (MqttException e)
{
e.printStacktrace();
}
return false;
}
//call this method after mqtt connection established to publish to any topic
public boolean publish(String topicName, String message, int qos)
{
try
{
if (topicName != null)
{
MqttMessage mqttMessage = new MqttMessage();
mqttMessage.setPayload(message.getBytes());
mqttMessage.setQos(qos);
mqttClient.publish(topicName, mqttMessage);
return true;
}
}
catch (MqttException e)
{
e.printStacktrace();
}
return false;
}
}

In my old project I created something similar to what you need.
I'm still sure Google (and Apple) notification systems are better. In any case here what you need.
You can use Eclipse Paho in order to produce and consume MQTT messages.
In my android app build.gradle file I added:
compile 'org.eclipse.paho:org.eclipse.paho.client.mqttv3:1.0.2'
Now there is a newer version of the library
This library offers to you all the needed API in order to consume and produce MQTT messages in and from an Android device.
In the documentation section you can find a sample application. You can start from there
On server side I used Apache ActiveMQ as broker. It offers an embedded implementation of MQTT handler and you can create topics and queues in order to handle MQTT messages.
I hope it's useful
Angelo
EDIT SECTION
Let's suppose you want to use ActiveMQ on server side.
You must download and install activemq. In the activemq.xml file inside the directory ${activemq_home}/conf you'll find the mqtt configuration. It's this line:
<transportConnector name="mqtt" uri="mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
This means that activemq handles mqtt protocol messages on port 1883 (the mqtt default TCP/IP port).
On the admin console of activemq you can create topic or queue you want to use for your messages. In the app you must connect the paho service to the created topic or queue.
Please note that by default activemq uses in memory DB. I suggest to yuo to configure it in order to use normal RDBMS or even NoSQL DB. The most important thing is that you configure it in order to store all messages not in memory otherwise you can risk messages will be lost.
Moreover if you expose on internet activemq I strongly suggest to you to protect it by secure credentials or by using SSL certificates.

Firestore admin "listens" to all documents again on reboot

TL;DR
Every time my Fiestore admin server reboots my document listener is triggered for all documents even if I have already listened to the document and processed it. How do I get around this?
End TL;DR
I'm working on building a backend for my Firestore chat application. The basic idea is that whenever a users enters a chat message through a client app the backend server listens for new messages and processes them.
The problem I'm running into is that whenever I reboot my app server the listener is triggered for all of the existing already processed chats. So, it will respond to each chat even though it has already responded previously. I would like the app server to only respond to new chats that it hasn't already responded to.
One idea I have for a work around is to put a boolean flag on each chat document. When the backend processes the chat document it will set the flag. The listener will then only reply to chats that don't have the flag set.
Is this a sound approach or is there a better method? One concern I have is that every time I reboot my app server I will be charged heavily to re-query all of the previous chats. Another concern I have is that listening seems memory bound? If my app scales massively will I have to store all chat documents in memory? That doesn't seem like it will scale well...
//Example listener that processes chats based on whether or not the "hasBeenRepliedTo" flag is set
public void startFirestoreListener() {
CollectionReference docRef = db.collection("chats");
docRef.addSnapshotListener(new EventListener<QuerySnapshot>() {
#Override
public void onEvent(#javax.annotation.Nullable QuerySnapshot queryDocumentSnapshots, #javax.annotation.Nullable FirestoreException e) {
if(e != null) {
logger.error("There was an error listening to changes in the firestore chats collection. E: "+e.getLocalizedMessage());
e.printStackTrace();
}
else if(queryDocumentSnapshots != null && !queryDocumentSnapshots.isEmpty()) {
for(ChatDocument chatDoc : queryDocumentSnapshots.toObjects(ChatDocument.class)) {
if(!chatDoc.getHasBeenRepliedTo() {
//Do some processing
chatDoc.setHasBeenRepliedTo(true); //Set replied to flag
}
else {
//No-op, we've already replied to this chat
}
}
}
}
});
}

Yes, to avoid getting each document all the time, you will have to construct a query that yields only the documents that you know have been processed.
No, you are not charged to query documents. You are charged only to read them, which will happen if your query yields documents.
Yes, you will have to be able to hold all the results of a query in memory.
Your problem will be much easier to solve if you use Cloud Functions to receive events for each new document in a collection. You won't have to worry about any of the above things, and instead just worry about writing a Firestore trigger that does what you want with each new document, and paying for those invocations.

Android/Firebase - Check to see if you are subscribed to topic

I am wondering if there is a way to test to see if you are subscribed to a topic on the android side of things.
Basically, I am HOPING that all devices will subscribe to a topic during their installation, when the token is first obtained by the device. However, there is always a chance that the device fails to subscribe. The FCM registration token should be installed on the device for a long time, and thus, the onTokenRefresh() method shouldn't be called again without clearing data, uninstall/reinstall, etc.
My idea was to test to see if the device is subscribed to a topic in my MainActivity, and if not, then try to subscribe again. If it fails to subscribe, then get a new token and try again, etc.
#Override
public void onTokenRefresh() {
// Get updated InstanceID token.
String refreshedToken = FirebaseInstanceId.getInstance().getToken();
Log.e(TAG, "Refreshed token: " + refreshedToken);
// Subscribe to a topic
Log.e(TAG, "Subscribing to topic");
FirebaseMessaging.getInstance().subscribeToTopic("test");
So, I can subscribe and unsubscribe, but how do I check if the device is subscribed to a topic? I did my fair share of googling, and couldn't find anything, unfortunately.
I would greatly appreciate any/all assistance. Thanks!

There is currently no way to check on the client side if they are subscribed to a topic.
The behavior for subscribeToTopic is it would immediately subscribe to the specified topic, if it fails, it would retry on it's own (unless your app was killed). See my answer here.
I think that forcing the onTokenRefresh call just to make sure that subscribeToTopic is too much. You could simply just call it in your initial activity if you want, that way, everytime the app starts, it sends the subscription request.

Actually this can be done by using this api: https://developers.google.com/instance-id/reference/server#get_information_about_app_instances
As IID_TOKEN you need the FCM token and in the header you have to pass Authentication: key=YOUR_SERVER_KEY. You can find the server key as described here: Firebase messaging, where to get Server Key?.
Don't forget to include details=true as query parameter in the url, otherwise the topics won't be included in the response.
I would recommend writing a Cloud Function to encapsulate it, so you don't deploy your server key to the client.