I'm using AWS SDK for Java.
Imagine I create a RDS instance as described in the AWS documentation.
AmazonRDS client = AmazonRDSClientBuilder.standard().build();
CreateDBInstanceRequest request = new CreateDBInstanceRequest().withDBInstanceIdentifier("mymysqlinstance").withAllocatedStorage(5)
.withDBInstanceClass("db.t2.micro").withEngine("MySQL").withMasterUsername("MyUser").withMasterUserPassword("MyPassword");
DBInstance response = client.createDBInstance(request);
If I call instance.getEndpoint() right after making the request it will return null to me, because AWS is still creating the database. I need to know this endpoint when it becomes available, but I'm not figuring out how to do it.
Is there a way, using the AWS SDK, to be notified when the instance was finally created?
You can use the RDS SNS notifications:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Events.html#USER_Events.Messages
Subscribing to Amazon RDS Event Notification
You can create an Amazon
RDS event notification subscription so you can be notified when an
event occurs for a given DB instance, DB snapshot, DB security group,
or DB parameter group. The simplest way to create a subscription is
with the RDS console. If you choose to create event notification
subscriptions using the CLI or API, you must create an Amazon Simple
Notification Service topic and subscribe to that topic with the Amazon
SNS console or Amazon SNS API. You will also need to retain the Amazon
Resource Name (ARN) of the topic because it is used when submitting
CLI commands or API actions. For information on creating an SNS topic
and subscribing to it, see Getting Started with Amazon SNS.
Disclaimer: Opinionated Answer
IMO creating infrastructure at runtime in code like this is devil's work. Stacks are the way to go here, much more modular and you will get some of the following benefits:
If you start creating more than one table per customer you will be able to logically group them into a stack and clean then up easier as needed
If for some reason the creation of a resource fails you can see this very easily in the stack console
Management is much easier to search through stacks as you have a console already built for you
Updating a stack in AWS is much easier as well than updating tables individually
MOST IMPORTANT: If an error occurs the stack functionality already has rollback and redundancy functionality built in, which you control the behaviour of. If something happens in your code during your on boarding process it will be a mess to clean up, what if one table succeeded and the other not? You will have to troll through logs (if they exist) to find out what happened.
You can also combine this approach with using something like AWS Pipelines or even AWS Simple Workflow Service to add custom steps in your custom on-boarding process, eg run a lambda function, send a notification when completed, wait for some payment. This builds on my last point that if this pipeline does fail, you will be able to see which step failed, and why it failed. You will also be able to see if things timeout.
Lastly I want to advise caution in creating infrastructure per customer. It's much more work and adds allot more ways in which things can break. Make sure you put limits in AWS as well that you don't have a situation in which your bill sky-rockets because of some bug creating infrastructure.
Related
I have a long running AWS Lambda function that I am executing from my webapp. Using the documentation [1], it works fine however my problem is this particular lambda function does not return anything back to the application its output is saved to S3 and it runs for a long time 20-30s. Is there a way to trigger the lambda and not wait for the return value since I don't want to wait/block my app while the lambda is running. Right now I am using an ExecutorService as a que to execute lambda requests since I have to wait for each invocation, when the app crashes or restarts I lose jobs that are waiting to be executed.
[1] https://aws.amazon.com/blogs/developer/invoking-aws-lambda-functions-from-java/
Tracking status is not necessarily a difficult issue. Use a simple S3 "file exists" call after each job execution to know if the lambda is done.
However, as you've pointed out, you might lose job information at some point. To remove this issue, you need some persistence layer outside your JVM. A KV store would work, store some (timestamp, jobId, status) fields in a database, and periodically check from your web server and only update from the lambda.
Alternatively, to reduce end-to-end time frame further, a queuing mechanism would be better (unless you also want the full history of jobs, but this can be constructed along with the queue). As mentioned in the comments, AWS offers many built in solutions that can directly be used with Lambda, or you need additional infrastructure like RabbitMQ / Redis to built a task event bus.
With that, lambda is now optional. You'd effectively periodically pull off events into a worker queue, which either can be very dumb passthroughs and invoke the lambda, or do the work themselves directly. Combine this with ECS/EKS/EC2 autoscaling and it might actually run faster than lambda since you can scale in/out based on queue size. Then you write the output events to a success/error notification "channel" after the S3 file is written
Back in the web server, you'll have to modify code to now be listening for messages asynchronously from that channel, and when you get a success message, you'll know that you should be able to access the S3 resources
I have a service that sends data to SQS which is working perfectly (code same as seen from Amazon Java SDK) while writing the consumer to read these messages in another queue I am facing issues. The function is never called? Again, the consumer code is also the same as that from the SDK, do I need to provide something else? Or are some more configurations required which are not present in the SDK?enter image description here
I have also attached the code which I have seen from the SDK. I am doing long-polling as well.
I built a FIX Initiator Application using the QuickFIX/J library to send orders to my broker. If you don't know what is a FIX Application, consider that my program is an application that sends message to a server through TCP connection.
To get and send the orders created by multiple algorithms I have a Directory Watcher (WatchService) that watches for modifications on a local directory that is synchronized with a S3 bucket using AWS Cli.
This approach works well, except for the fact that I have to wait about 6-8 seconds before the file is on my local directory, so I can parse it to fix orders and send to broker's FIX app. I really would like to decrease this delay between the order creation and the moment when it is send to the broker.
What are the possible solutions that I'd tought:
1) Reading directly from S3 bucket without using AWS CLI
2) Opening different FIX sessions for each different algorithm
3) Instead of reading from a bucket, peaking a database (MySQL) for new orders. The algos would generate table rows instead of files
4) Having an API between my FIX application and the algorithms, so the algos can connect directly with my application.
Solution (1) didn't improved the order receiving time because it takes about the same time to list S3 objects, get summary and filter the desired file.
Solution (2) I didn't tried, but I think it is not the best one. If I have, for example, 100 different strategies I would have to open 100 different connections and I am not sure if my broker app can handle. But I may be wrong.
Solution (3) I also didn't tried.
Solution (4) is what I believe that is ideal, but I don't know how to implement. I tried to create an REST API, but I don't know if it is conceptually correct. Supposing that my FIX application is currently connected to the broker's server, my idea was to (i) create a new webapp to create a REST API (ii) receive order info through a API, (iii) find the current alive session and (iv) send order to broker server using the current session. Unfortunately, I was not able to find the current session by ID using the following on a different of the class that is running the FIX application:
SessionID sessionID = new SessionID("FIX.4.4", "CLIENT1", "FixServer");
Session session = Session.lookupSession(sessionID);
What I would like to hear from you:
What do you think that is the best solution to send FIX orders created by multiple sources?
If I want to create an API to connect 2 different applications, what are the steps that I can follow?
I am sorry if I was a bit confuse. Let me know if you need further clarification.
Thank you
Q : What do you think that is the best solution to send FIX orders created by multiple sources?
Definitely the 4) -i.e.- consolidate your multiple sources of decisions and interface the Broker-side FIX Protocol Gateway from a single point.
Reasons:
- isolation of concerns in design/implementation/operations
- single point of authentication/latency-motivated colocation for FIX Protocol channel
- minimised costs of FIX Protocol Gateway acceptance-testing (without this Tier-1 market participants will not let you run business with, so expenses on FIX Protocol E2E-mutual-cooperation compliance-testing do matter - both costs-wise and time-wise )
Q : what are the steps that I can follow?
Follow you own use-case, that defines all the MVP-features that need to be ready for going into testing.
Do not try to generalise your needs into any "new-Next-Gen-API", your trading is all about latency+trading, so rather specialise on the MVP-definition and do not design/implement anything beyond an MVP with minimum latency (overhead) on a point to point basis. Using stable professional frameworks, like nanomsg or ZeroMQ, may avoid spending a bit of time on reinventing any already invented wheels for low-latency trading messaging/signaling tools. Using REST is rather an anti-pattern in the 3rd millenium low-latency motivated high performance distributed computing eco-system for trading.
I am creating an app that logs data. I am creating documents that have the data and sending those documents to a couchbase server. Or I am trying to anyways. One major concern I have is how do I confirm a document is stored on the server so that it can be immediately deleted on the device? I am hoping there is a quick and efficient way to do this. The end result is to have a thread constantly checking if there is a connection to couchbase, and if so start sending data up to clear it off the device. Most documentation seems to be regarding syncying the database, however I don't want to do this because I don't want to keep a copy of the data on the device. It would take up too much storage. Thanks for any help.
EDIT: For clarification, I currently have the app storing many data points in documents. I want to send these documents to a couchbase server. I don't want to "sync" the documents, but rather just insert them into the database then immediately delete them off the device. How would one go about doing this? Most examples I have seen typically sync documents such as profile information where changes can be made in various synced databases and all those changes would appear in every database. Instead I want a 1 way relationship with the database were information is sent, confirmed as received, then immediately deleted from the device.
There are at least a few possibilities.
If you are expecting a solid network connection, or are ok with handling errors yourself, you can achieve this with a direct REST call to Sync Gateway. You can, of course, always write your own REST server that talks directly to Couchbase Server, too.
The second way relies on an older version of Couchbase Lite. Couchbase Lite 2.x is a major rewrite of the product. As of the current shipping version (2.1), it does not support this approach, so you'll need to use the 1.x version (1.3 or later, IIRC). See further down for how to approach this with 2.1.
Set up a push only replication. After replication, cycle through the docs and purge all the ones that are not still pending. (This uses the isDocumentPending method on the Replication class. That's the key piece not available as of 2.1.) You can either run one shot replications and do this after the replication completes, or monitor the replication state of a continuous replication.
Purging a document from the local CB Lite database effectively makes it act as if it never existed on that device. By running a push only replication, you don't have to worry about the docs getting sent back to the device.
Using 2.1, you can't as easily determine if a document has been replicated. So you need to run a replication to completion while avoiding a race condition with writing something new.
One approach here is to pause writing documents, run a one shot replication, then purge the documents before starting up again. You could also work out something with alternating databases, or tracking documents yourself somehow, etc.
For completeness, if you were in a situation where you had a mixed use, that is, wanted only some documents pushed up off the device and forgotten, and some synced, you would control this through Sync Gateway channels.
I don't know Lite and Sync Gateway well enough, but from a Server perspective:
You could use the new Eventing service in Couchbase. When a document is created in bucket A, you could write an event to copy that document over to bucket B. Then, if the documents are deleted on the device, it wouldn't matter if they get deleted from bucket A.
I have a bucket "staging" and a bucket "final". I created a function called "moveIt" with "final" (I aliased as 'f').
The OnUpdate function could be as simple as:
function OnUpdate(doc, meta) {
f[meta.id] = doc;
}
My main concern would be the timing. I don't think there's an easy way for your mobile app to know that the event has finished copying a document before you decide to delete it in Lite and start a Sync. But it might be worth a try. Check out the docs to learn more about the Eventing service details.
In Couchbase Lite 2.5, you can use replicated events to detect when a document has synced (pushed to server or pulled from server). You can register a callback on the Couchbase Lite replicator to detect if the documents have been pushed to the sync gateway and then use the purge API to locally purge
We are developing a webapp that needs to send out emails written in Java/Groovy. Currently, we are persisting each email to a database before we call the Java Mail APIs to send the mail to our SMTP server.
I want to send out email asynchronously. I want to persist the email and then have another process pick up the email and send it (and send it only once). Ideally, this process is running outside of my webapp.
Are there any tools that do that?
Update: This solution needs to prevent duplicate emails and it needs to handle spikes in email. I was hoping someone already wrote an offline email processor. (I'd rather not implement this myself.)
The suggestions to use a cron job to read the database are quite workable.
Another good approach here is to use a Java Message Service (JMS) message queue. These are persistent (backed up by a database) and reliable. You can have one or more producer programs enqueue messages with the relevant data in them, and then one or more consumers process the messages and dequeue them. All of this is set up for very high reliability, and you gain the flexibility of asynchronously decoupling the operations, which means during email spikes the message queue can grow larger until the consumers catch up with the spike. Another benefit is that the email goes out as soon as a consumer gets to it instead of on a timer. Plus, if you require high availability, you can have multiple consumers in case one goes down.
Check out Apache's ActiveMQ for a good open source implementation of JMS.
If you're using Linux/Unix you could create a cron job to run every few minutes which calls a program to grab the email from the database and send it out. You could also have a field in the database to indicate whether the message has been sent. The downside of this approach is that there may be a delay of a few minutes from when your webapp persists the email and when the cron job is run.
Setup a cron job and use scripts to query the db and send out emails via sendmail.
On the off chance it's an Oracle DB, you can use the UTL_MAIL package to write PL/SQL to send the mail through your SMTP server. Then create a scheduled job to execute on your desired schedule.
http://download.oracle.com/docs/cd/B19306_01/appdev.102/b14258/u_mail.htm
Since you are already using groovy, this might be an interesting tool to solve your problem
http://gaq.sourceforge.net/
You could use Quartz, a scheduling library (similar to cron), to schedule a recurring task that reads the DB and sends the emails. If you're using Grails, there's a Quartz plugin that makes working with Quartz a bit more Groovy.