I have a service that sends data to SQS which is working perfectly (code same as seen from Amazon Java SDK) while writing the consumer to read these messages in another queue I am facing issues. The function is never called? Again, the consumer code is also the same as that from the SDK, do I need to provide something else? Or are some more configurations required which are not present in the SDK?enter image description here
I have also attached the code which I have seen from the SDK. I am doing long-polling as well.
Related
I am Using the Aws IVS for live streaming . when the stream ends I need to get the notification. I have configured the Event Bridge with source as IVS and destination as DEV, QA and PROD endpoints. when the streams ends I am getting the notification in all the endpoints.
But my requirement is, if streaming starts from the dev, only dev endpoint should receive the stream end notification. if streaming starts from qa, only qa endpoint should receive the stream end notification. how to achieve this ? Thanks in Advance.
I had a similar issue, and we end up creating and event-bridge for production, one for develop and another for staging.
Depending of the environment the producers sends to one event-bridge or another, same with the consumers.
the price remains the same
I'm using AWS SDK for Java.
Imagine I create a RDS instance as described in the AWS documentation.
AmazonRDS client = AmazonRDSClientBuilder.standard().build();
CreateDBInstanceRequest request = new CreateDBInstanceRequest().withDBInstanceIdentifier("mymysqlinstance").withAllocatedStorage(5)
.withDBInstanceClass("db.t2.micro").withEngine("MySQL").withMasterUsername("MyUser").withMasterUserPassword("MyPassword");
DBInstance response = client.createDBInstance(request);
If I call instance.getEndpoint() right after making the request it will return null to me, because AWS is still creating the database. I need to know this endpoint when it becomes available, but I'm not figuring out how to do it.
Is there a way, using the AWS SDK, to be notified when the instance was finally created?
You can use the RDS SNS notifications:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Events.html#USER_Events.Messages
Subscribing to Amazon RDS Event Notification
You can create an Amazon
RDS event notification subscription so you can be notified when an
event occurs for a given DB instance, DB snapshot, DB security group,
or DB parameter group. The simplest way to create a subscription is
with the RDS console. If you choose to create event notification
subscriptions using the CLI or API, you must create an Amazon Simple
Notification Service topic and subscribe to that topic with the Amazon
SNS console or Amazon SNS API. You will also need to retain the Amazon
Resource Name (ARN) of the topic because it is used when submitting
CLI commands or API actions. For information on creating an SNS topic
and subscribing to it, see Getting Started with Amazon SNS.
Disclaimer: Opinionated Answer
IMO creating infrastructure at runtime in code like this is devil's work. Stacks are the way to go here, much more modular and you will get some of the following benefits:
If you start creating more than one table per customer you will be able to logically group them into a stack and clean then up easier as needed
If for some reason the creation of a resource fails you can see this very easily in the stack console
Management is much easier to search through stacks as you have a console already built for you
Updating a stack in AWS is much easier as well than updating tables individually
MOST IMPORTANT: If an error occurs the stack functionality already has rollback and redundancy functionality built in, which you control the behaviour of. If something happens in your code during your on boarding process it will be a mess to clean up, what if one table succeeded and the other not? You will have to troll through logs (if they exist) to find out what happened.
You can also combine this approach with using something like AWS Pipelines or even AWS Simple Workflow Service to add custom steps in your custom on-boarding process, eg run a lambda function, send a notification when completed, wait for some payment. This builds on my last point that if this pipeline does fail, you will be able to see which step failed, and why it failed. You will also be able to see if things timeout.
Lastly I want to advise caution in creating infrastructure per customer. It's much more work and adds allot more ways in which things can break. Make sure you put limits in AWS as well that you don't have a situation in which your bill sky-rockets because of some bug creating infrastructure.
I'm trying to implement a service which consumes a google pubsub subscription at its own pace. By that, I mean I need fine control on when I need to consume messages i.e get a batch of messages, pause for a while, do not get more than X messages...
Using google client libraries I did not find a way to do it as the MessageReceiver is running in its own thread and I don't have any control on what exactly happens.
Basically, being able to consume messages in a synchronous way should solve my issue.
Do you know how I can use the google client libs synchronously ? Or is there another way in the API I missed ?
You might try using setFlowControlSettings when you build your subscriber. In particular, you can use setMaxOutstandingElementCount or setMaxOutstandingRequestBytes to limit the messages sent to your MessageReceiver. When you have enough messages outstanding, i.e., messages for which you have not called Ack() or Nack(), to exceed these limits, then your MessageReceiver will not be called until messages have been acked or nacked.
I am currently working on a POC where I am draining messages from an azure queue - that is populated through a dynamics CRM plugin during DB CRUD operations.The queue drain operation using a java client fails with the message :
<Error>
<Code>500</Code>
<Detail>
The service was unable to process the request; please retry the operation. For more information on exception types and proper exception handling, please refer to http://go.microsoft.com/fwlink/?LinkId=761101. TrackingId:add85f1a-a249-4e69-b284-ad879cd29968_G27, SystemTracker:scsqueue1-ns:Queue:scsqueue1, Timestamp:7/6/2016 7:21:49 PM
</Detail>
</Error>
The java client can be found here :
https://github.com/sharpcodes/scs-bus-demo
However using a C# client works
I can't see the obvious issue in your code on GitHub. However, per my experience, I suggest that you can try to use the tool Fiddler for debugging the rest request from your code, because the Azure Service Bus SDK for Java is wrapped the REST APIs, such as Receive and Delete Message (Destructive Read) for draining messages.
Meanwhile, there are three ways below in Java for draining messages via the operation Receive and Delete Message on Azure.
Using Azure Service Bus SDK for Java, please refer to the section Receive messages from a queue of the tutorial "How to use Service Bus queues" to do with default mode ReceiveAndDelete.
Using Azure Service Bus REST API for Java with http client, please refer to the section Receive and delete a message from the queue of the tutorial "Service Bus brokered messaging REST tutorial".
Using JMS API with AMQP 1.0 in Java, please refer to the section Coding Java applications of the tutorial "How to use the Java Message Service (JMS) API with Service Bus and AMQP 1.0" to see the function SimpleSenderReceiver().
Thanks to peter for guiding on this..
Looks like there is an issue with the dynamics crm-azure bus plugin that prevents java/node-js clients from draining the queue using SDKs or RestFul calls.
Using JMS client seems to do the trick. Another way would be to have a C# middleware that could be wrapped by Java/Node.
Here is the reference to the issue : https://github.com/Azure/azure-sdk-for-php/issues/823
The issue is closed now as a fix had to be made by Azure Java SDK team to escape characters correctly..
If you create an SNS Topic and, in the Amazon Console, open the Delivery Status options under Other topic actions, you can see this:
As you can see, it's possible now to get SNS delivery status feedback by configuring success and failure IAM roles. This works fine and I can see all the logs in CloudWatch for all published messages to each subscriber.
What I can't do is to set these values with the Java AWS SDK, is there any way of doing this?
I'm using aws-java-sdk:1.10.23 (latest as of now)
As #david-murray pointed out in the documentation, this is the solution to configure the feedback for HTTP endpoints:
amazonSnsClient.setTopicAttributes(topicArn, "HTTPFailureFeedbackRoleArn", "arn:aws:iam::1234567890:role/SNSFailureFeedback");
The same idea can be used for Application, Lambda and SQS.
My mistake was trying to set all of them with a single call like the form in the screenshot does by using:
https://eu-west-1.console.aws.amazon.com/sns/v2/SetMultiTopicAttributes
Although this doesn't seem to be present in the SDK at the moment, 4 separate calls will have the same effect.
Thanks!