I am Using the Aws IVS for live streaming . when the stream ends I need to get the notification. I have configured the Event Bridge with source as IVS and destination as DEV, QA and PROD endpoints. when the streams ends I am getting the notification in all the endpoints.
But my requirement is, if streaming starts from the dev, only dev endpoint should receive the stream end notification. if streaming starts from qa, only qa endpoint should receive the stream end notification. how to achieve this ? Thanks in Advance.
I had a similar issue, and we end up creating and event-bridge for production, one for develop and another for staging.
Depending of the environment the producers sends to one event-bridge or another, same with the consumers.
the price remains the same
Related
The Java application posts async jobs to AWS and gets back a JobID. When the async job is finished, a message will appear in an SQS queue with that JobID. Each JobID is handled by a different thread. Each of those threads also polls SQS for messages until it finds the message which contains its JobID. Additionally, the application is distributed into multiple services so there can't be a single SQS processor.
I saw that SQS returns a maximum of 10 messages and after they are returned, a visibility timeout is applied so that they are not re-sent to other consumers. However, my consumers are the threads that want to consume only a single message and let the rest be consumed by other threads. Should I set the visibility timeout to 0? Will this make it so all consumers get the same set of 10 messages on every request? What's the best way for each consumer to sift through all the messages and find the one it wants?
TL;DR: SQS has 100 messages and there are 100 consumers, one for each message. How should I go about having each consumer find the message it wants (based on a JobID).
EDIT: I know that this is not an appropriate usage of SQS and I'd be very glad to not use it at all but our main integration is with Amazon Textract for which it is mandatory to use SQS for its asynchronous operations. Each Textract request is processed by a different thread which means that they each need to get back a specific SQS message, consumers are not universal. Not to mention the possibility of a clustered environment for which I'd like to avoid having to do any synchronization...
EDIT 2: This is for an on-premises, Setup.exe based, dev-hands-off application where we want to minimize the amount of unneeded AWS services used (both for cost and for customer setup/maintenance reasons) as well as the use of external components, again to minimize customer deployment/maintenance/servers. I understand that we are living in the world of microservices but there are still applications that want to benefit from intelligent services without being cloud-native themselves.
This is not an appropriate architecture for using Amazon SQS. Your processes should not be trying to find a specific message from an Amazon SQS queue.
You should find a different architecture for this message-passing task. Some ideas:
Create a message in Amazon S3 with an 'expected' Key. Have the each thread look for that object as a return message. (This is effectively using Amazon S3 as a Key-Value Store.)
Have a single Lambda function retrieve messages from SQS and update a database (or S3 as above). Then, have the threads consult the database instead of SQS.
I think you need to put something in between SQS and your threads. Like a DynamoDB table. You could have a Lambda function that processes all the SQS messages and just translates them into DynamoDB records. Then your different threads could easily check for the specific records they are interested in using a DynamoDB query.
Just because Textract mandates that you use SQS doesn't mean the final step in your architecture has to read those messages directly from SQS. In this case SQS is just a message bus that can integrate with other services in AWS, and those services are your building blocks you can use to create the architecture you need.
I am not able to find a way to send/broadcast a message to all application instances in Pivotal Cloud Foundry. How can we notify to all app instances of some events? If we use the HTTP request, PCF router will dispatch it to a single instance of the app. How can we solve this problem?
What #Florian said is probably the safer option, but if you want something quick and easy, you can send HTTP requests directly to an app instance by using the X-CF-APP-INSTANCE header. The format for the header is YOUR-APP-GUID:YOUR-INSTANCE-INDEX.
https://docs.cloudfoundry.org/concepts/http-routing.html#app-instance-routing
So given an app guid, you could iterate over the number of instances, say 0 to 5, and send an HTTP request to each one. Make sure to check the response to confirm that each one succeeded.
This also requires that you know the app guid for your app (i.e. cf app <name> --guid) and the number of instances of your app.
CF, out of the box, does not provide any event queue mechanism where apps can subscribe to.
What I would do (assuming you've two app instances A and B):
Provide an event endpoint in your application code, e.g. POST /api/event (alternatively, if the event should arise from another app (e.g. another microservice), this one could directly send messages onto the queue)
All app instances are listening on an internal event queue for new events
instance A receives the call from the CF router and processes it by issuing an event on an internal event queue, the instance will not react to the event, yet
When A publishes the event, A and B receives the event and processes it accordingly
Now, the internal event queue you can use highly depends on your deployment. On AWS you probably can use SQS or SNS or something similar. PCF, as I know, may also provide a messaging system which would suit here as well, rabbitmq. You could also use features of other services that would allow you to subscribe to events, such as redis (pub/sub commands) or similar.
If you provide more information about what you want to achieve more concretely, more detailed answer would be possible, though.
I have a service that sends data to SQS which is working perfectly (code same as seen from Amazon Java SDK) while writing the consumer to read these messages in another queue I am facing issues. The function is never called? Again, the consumer code is also the same as that from the SDK, do I need to provide something else? Or are some more configurations required which are not present in the SDK?enter image description here
I have also attached the code which I have seen from the SDK. I am doing long-polling as well.
I have an application that contains order data . I want to send this to all users of this app as notification.My target is to send Push Notifications to all of the users of my applications. How would I accomplish that?
To begin with your app mush already be using GCM and listening to the correct topic. Otherwise you have to roll out an update with the new GCM feature and hope that all users update it.
If you want to see some code for this take a look at this sample i crated some time ago. What you need to implement from the sample is the "SubscribeTopic" part. And to test if its working you can use this java program.
The good thing about topics is that you don’t need to save the users registration tokens and the message is sent to everyone listening for that exact topic.
Use Apache kafka
The original use case for Kafka was to be able to rebuild a user
activity tracking pipeline as a set of real-time publish-subscribe
feeds. This means site activity (page views, searches, or other
actions users may take) is published to central topics with one topic
per activity type. These feeds are available for subscription for a
range of use cases including real-time processing, real-time
monitoring, and loading into Hadoop or offline data warehousing
systems for offline processing and reporting.
To start with, note that a full GCM implementation requires both a client implementation and a server implementation. Before you can write client apps that use GCM, you must have an application server that meets the following criteria:
Able to communicate with your client.
Able to send properly formatted requests to the GCM connection
server.
Able to handle requests and resend them using exponential back-off.
Able to securely store the API key and client registration tokens.
Note: never include the API key in any client code.
For XMPP, the server must be able to generate message IDs to uniquely
identify each message it sends (GCM HTTP connection server generates
message IDs and returns them in the response). XMPP message IDs
should be unique per sender ID.
Complete documentation, how-to-guides for sending messages and links to examples can be found from Cloud Messaging - Messaging Concepts and Options.
You may also check ANDROID AND GCM – BROADCAST YOURSELF for the tutorial and demonstration on the use of GCM for the broadcast of messages to an Android client from a Tomcat server and use of sending broadcastintents from a service and receiving those broadcastintents from an app's activity.
sorry for maybe a noobish question and my english.. I want to create a personal aggregation of all messages (chat, group) and posts (from pubsub services) with my xmpp client (e.g. new private messages and posts from different pubsubs will be aggregated in one place (read and unread messages). Furthermore is it possible to receive this aggregated stream with posts on different resources (even if some of the messages have been read on one device but on which not all the messages have been read)?
Is that possible with xmpp? Do I have to create a dedicated personal (user) pubsub to which I will forward (publish) all the messages (or a kind of a webservice for this with an access to a table "inbox" to store the messages). So whatever client of mine which goes online first will collect the private messages and posts from different pubsubs and then will forward to the dedicated pubsub (or web service) from which other resources of mine will get the messages because all the clients are also subscribed to the dedicated pubsub. Is my thinking right? I hope it's not all trash what I'm writing here..
Or is there a XEP for this?
Please, please help ..
In order to be able to notify and monitor other clients on different devices and at the same time need which messages are marked as unread in different customers you will need to write quite a lot boilerplate code.
For sure you will need a centralized web service which will receive the post streams (either in parallel with your client/s or first it will receive them and then send to the client/s). Pub/sub is suitable for this application but you will also need to send some additional data to the service from your clients like the time stamp of the last read message (in order to mark all newer as unread).
I think the easiest way would be to use the webservice as a gateway where all streams will be directed initially and where you can also monitor what is delivered and to which client.
Hope it helped