Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
We are building a new app that would require to store some data in S3 or DynamoDB. The number of users for this app would be relatively small (~ 500).
There are 2 ways of designing this.
We can either use the AWS SDK to persist information directly from the app
Build a back-end web service that talks to the data store and use HTTP to communicating with this service.
There are multiple pros and cons for each like Option #1 is way simpler and avoids a lot of overhead, Option #2 however is cleaner and provides better encapsulation. Also the back end web service could be used for any other task processing in the future.
Based on what aspects should once decide which option to choose ? Any help would be appreciated
I'm not convinced that the overhead of Option 2 is worth it, so I would go with Option 1.
Keep in mind that Amazon S3 and DynamoDB are Web services (as explained here for DynamoDB, same is true for S3). It's not like you're talking directly to a database through some specific port.
I would only go with Option 2 if I need to do server-side processing. If it's just for storing data, I would start with Option 1. You can always add the back-end service later.
Option 2 all the way forward, but give that back end service some role in your domain. Think SOA.
If you are talking only of "storing" in Dynamo, even option 1 will do. (Be aware that you need to keep the AWS secret and key in the app).
If you also need to "read" then option 2 is better, since you will get the flexibility of keeping a cache (like elastic cache) between your backend and dynamo to increase response times. Dynamo is "single region". Unless your read happens from the same region, then it is going to be very slow - latency issues. And this cache will help you reduce the cost of the dynamo, since you dont need to increase the throughput so high - Nothing is as fast as an in-memory cache !
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 11 months ago.
Improve this question
I want to create a catalog of java.util.functions so when I feed in A,B,C then the app will run those functions in the same order flowing the output to the next function?
The above statement is basic, which I would love to hear ideas. Eventually, the workflow will be fed an input as json for example to orchestrate this.
Also need to support reactor and where we can map, flatmap or filter.
For now all functions will be available in the project, and will be part of a Spring Boot app.
I am thinking, I can make the functions as beans, pull them and execute in order, although not sure how to orchestrate the functions in case there is a filter and then further if I want to use reactor where I will have map or flatmap.
Please share any ideas or any frameworks worth looking into.
Thanks
I think using a workflow orchestrator platform would be a great way to implement this use case.
You probably don't need that since all your functions are available in the same service, but using an orchestration platform will offer a lot flexibility as your service grows in complexity or when the need for orchestrating your use case across multiple services comes up.
An option for workflow orchestration is Netflix Conductor. Here is an article that talks about how to use Conductor to manage sequential tasks. Conductor platform has features to run flows in a distributed fashion as well, such as fork join, decisions etc. and this can be done across different services.
Ex: Service 1 hosting Function A, Service 2 hosting Function B and C. And you can build a flow running A -> B -> C while passing/referring to outputs from the previously executed functions without having to build any state management systems.
The above article was written as a response to this Stackoverflow question which is very similar to yours.
For clarity - I used to work at Netflix on the team that built and open sourced Conductor.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
I am making an API with Spring boot which will maybe have requests peaks.
So let's get into the worst scenario. Imagine that suddenly I have 2 million API requests.
What can I do to be able to handle this?
I read something about queues and workers but I don't know if that's the best way.
Is there some way to do it with something with AWS?
This is always a tricky question to answer. Firstly does your app really need to scale to 2 million API requests at peak? I ask because it is easy to over-engineer a solution 'to deal with future scale' which ends up a bit buggy and not even dealing with current scale very well as a result.
But assuming you really will have massive request peaks, the current microservice approach (or buzzword?) is quite a popular way of dealing with these periods of high demand. Basically you split your application up into smaller, self contained services ('microservices') which can be more-easily scaled as required.
Individual microservices can then be scaled up and down to match the load, using something like Kubernetes or Amazon ECS.
In terms of where Spring ties into this, Spring has a handy set of technologies called Spring Cloud - you'll notice Spring Cloud AWS there, too (although Spring Cloud in general can also work fine on bare metal servers, Docker, Kubernetes etc etc). A year ago I put together a simple Spring cloud/microservices demo on Github which shows how different Spring-powered microservices can fit together, which you may find useful.
The other nice thing with microservices is that you can swap out the language that a particular service is written in fairly easily, especially if the microservices 'speak' to each-other in a general format (such as JSON via REST requests). So if you had 10 different microservices, all powered by Spring Boot, but you find that a couple of them are better written in another language, you can just rewrite them: as long as they send and receive data in the same way, then the other parts of your system don't need to care.
Okay, that's a lot of buzzwords and new concepts. Feel free to ask questions if I can clarify anything, but microservices + kubernetes/AWS is a popular solution.
Other approaches which will probably work equally well, however, are:
Serverless - this is where you use a bunch of cloud-provider's tools and solutions to power your webapp, including things like lambdas, instead of hosting your application on a traditional server/VM. This medium tutorial gives a simple introduction to serverless.
Monoliths - this is a traditional web application which is a single, large, sprawling codebase, but this doesn't mean you can only have one instance of it running (i.e. you can still scale it). This very site is a successfully scalable monolith.
Your question is very broad, as there are lots of different solutions:
1) Use a load balancer and have multiple instances of your application
2) Use a containerization tool like docker and kubernetes to increase the amount of instances depending on current load. You can essentially scale on demand
3) We don't know what your app actually does: is it read heavy, is it write heavy? Will users be downloading content? The answers to this question can change whether or not a specific solution is feasible
4) You can maybe use a messenger queue like RabbitMQ to assist with distributing load across different services. You can have multiple services reading from this queue and performing actions at the same time...but again, this depends on what your app will actually be doing.
Check out AWS EC2 and Elastic Beanstalk. You can also get a simple load balancer up and running with nginx. Good luck
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I need to implement Session management for my company, we were using RDBMS till now but the latency is too high as the latency for session data should be very low.
So I thought of researching it out on web and found out these three options:
Ehcache + Terracota
In Memory DB (Timesten and others)
NoSQL (Aerospike, Couchbase )
Now I am starting to try these solutions out, but would like to get 2 cents from the experienced ones. The workload pattern would be 50/50 read/write and needs cluster for HA.
We need high throughput, low latency and eventual consistency.
As per what I have searched it seems like NoSQL is good fit for this scenario but would like to hear more on this.
Which of these options is best suited for such workloads?
Ad Business is one of the largest industry which uses session management. And I have seen almost all of them consider NoSQL based solution.
Since the prime requirement for Ad Business is display ads with least latency they usually tend to store data In-Memory for bringing down the latency. But that comes at cost.
Since you have already mentioned Aerospike, It is optimised for SSD and thus you can have sub millisecond level latency achieved by storing data on SSD as well. And this product is recently being open sourced.
There are other options like Redis, mentioned in above comment but Aerospike does provides benefits like Cross Data Center Replication for HA purpose and auto sharding. It is very easy to setup and get running with minimal configuration.
So I would vote for NoSQL looking at your use case.
Microsoft offers in ASP.NET and their PaaS, Azure, to use Redis as a session store. I suppose they know why they chose to support this specific NoSQL DB, so you might want to have a look into it.
Redis is in memory key value store and offers time bound inserts (data can be deleted automatically by redis after some time). Additionally you can also deploy it on multiple machines to reach a better performance or higher availability.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
I am planning to program a software (in java), which will be (hopefully) used very much. In the beginning I may run it on my own server, but if it becomes popular my server will most likly crash.
So my plan is to program it for a cloud service like Amazon EC2, Google App Engine, Lunacloud or others.
The application will not have a gui for now. It will concentrate on SOAP or JMS (or something like that) and should store a lot of data in a relational database (PostgreSQL would be nice).
Since I am new to the cloud services, I tried a little bit with GAE. The main use is easy, but as soon as I use JPA and ManyToMany-Relations GAE is shit. Also making a SOAP or JMS Server in GAE is not simple.
Since I lost my weekend with trying GAE, I thought it would be a good idea to ask the community.
Which cloud service will fit best for my requirements? What are the benefits and differences between these services? What else can you recommend?
This is question is too wide open to provide a good answer, but here is some tips that should help.
There is a difference between platform as a service (GAE, Jelastic, Heroku) and Infrastructure as a Service (EC2).
In the platform as a service category, you have more of an automated infrastructure, and often very little visibility of the underlying components. This can make things easier from a developer perspective, but it has its downsides. You are often locked into how a provider works and it may be difficult to switch. You may also have limitations as to what you can do with your application.
In the Infrastructure as a Service category, you get access to virtual machines that you can configure and automate yourself. You have more flexibility on this type of platform, but you are generally expected to handle more of the work yourself. EC2 does have its own version of platform as a service with elastic beanstalk.
i would recoomend also heroku because it does not have a traffic limit and you can run a basic instance for free. if you dont need nosql dbs and extra software it will be very cheap and the unlimited traffic is good for your webservices. Gae has is own filestructure so i can understand your problems with your db structure very good. heroku and ec2 does not restrict your plans but ec2 is generally expensive if you dont plan to scale up and down often. heroku gets also very expensive when you want to add extra software and scale up. i dont know if youre able to scale up as good as with ec2 if you want to use jelastic.
another but complex approach would be renting some normal root servers with unlimited traffic where one instance act as load balancer but you would have to do the configuration by yourseld
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I have not had much experience with Webservices or API's.
We have a website built on Oracle->Sun App Server->Java->Struts2 framework. We have a requirement to provide an API for our system. This API will be used by other systems outside of our system. API is just for a simple SP that we have on our database. The other system does not want to connect to our DB to gain access to the SP but instead and an API as a 'webservice'
Can the community please shed some light on how to go about this? Will the API be put on our webserver? is that how the other system will connect to it? And how to go about creating a public API?
Some things you'll need to think about are:
SOAP vs REST (Why would one use REST instead of SOAP based services?)
How will you handle authentication?
Does it need to scale?
You might want to take a look at https://jersey.dev.java.net/.
It would also be helpful to look at how another company does it, check http://www.flickr.com/services/api/ for some ideas.
If you are using the Sun App Server, it should be fairly trivial to make an EJB exposed as a web service with the #WebService tag, and then have that EJB call the Stored Proceedure and return the data. The app server gives you tools to publish a WSDL which is what they will use to know how to call you API.
That being said, what sounds easy at 50,000 feet is a real pain to deal with all the details. First, what about security? Second, are WebServices really required, or is there a better communication mechanism that is more obvious, such as (at a minimum) REST, if not some simple servlet communication. And the hardest part: In exactly what format will you return this result set?
Anyway, you could be dealing with a bit of a political football here ("what, you don't know how to do web services, everyone knows that, etc.") so it can be a bit hard to probe the requirements. The good news is that publishing a web service is pretty trivial in the latest Java EE (much easier than consuming one). The bad news is that the details will be a killer. I have seen experienced web service developers spend hours on namespace issues, for example.
Soap or Rest or .. is one side of the medal and depends on what the clients want.
The other (more) important thing is the api design itself. Shall it be stateless or stateful. Are clients co-located in the same VM (Appserver) or remote in the same LAN or even in a Wan.
As soon as the communication goes over the wire, it gets slow due to serialization. So you want API methods to obtain bigger (but not too big) chunks of data at a time.
Or in other words, your question can not really be answered without knowing a lot more about what you want and need to do.