I am new to amazon kinesis so might be this question is very basic question.
But i need help on this,
I am having one use case where i need to pull the data from amazon kinesis into my web application which has been created in JAva, i need to obtain the connection between the kinesis and java so that i can take the data from amazon kinesis and do some of the analytics on the data and if the data is modified then i need to put the data back to amazon kinesis from java application.
here my java application is not on the amazon clould, it is on my private cloud. so how do i do the above task.
Kindly help me.
First of all, Amazon Kinesis API endpoints are on public IP addresses, so you don't need to have EC2 instances within AWS environment to access Kinesis.
To read data from Kinesis, you can use Amazon's own Kinesis Client Library (KCL).
https://github.com/awslabs/amazon-kinesis-client
On AwsLabs GitHub, there are also sample applications written in Java.
https://github.com/awslabs/amazon-kinesis-connectors/tree/master/src/main/samples
About your architecture; if you want to process raw data and create meaningful extracts, I recommend you do some ETL tasks (aka: post processing) and write your results to another place (ie. RDBMS). On the view layer (your web app) you can display the resulting output in any format you like by reading from your database.
Related
we wanted to use amazon kinesis service to stream video from producer to consumer, so in the docs they had mentioned different producer libraries available. They have producer library for Java, android, c++, c. We wanted to do streaming from android device to kinesis stream so we had a look at android producer library but from the sample code we came to know that we require to use another amazon service Cognito to create user pool, identity pool and all.
So what we wanted was to stream from android device using only kinesis and no other amazon service. So since android support java, we wanted to know whether the Java producer library can be used in android since it doesn't use any other amazon service or even use C++ producer library for android?
Searched on web for this but there were no post related to this. So guys any help will be appreciated. Any reference will be really really helpful. Please help.
We have got a web based Java application which we are planning to migrate to cloud with an intention that multiple clients will be using it in a SaaS based environment. The current architecture of the application is quite asynchronous in nature. There are 4 different modules, each having a database of its own. When there is a need of data exchange between the modules we push the data using Pentaho and make use of a directory structure to store the interim data file, which is then picked up by the other module to populate its database. Given the nature of our application this asynchronous communication is very important for us.
Now we are facing a couple of challenges while migrating this application to cloud:
We are planning to use Multi Tenancy on our database server, but how do we ensure that the flat files we use for transferring the data between different modules are also channelized to their respective tenants in the DB.
Since we are planning to host this in cloud, would seek your views, if keeping a text file on a cloud server would be safe from a data security perspective.
File storage in cloud is safe and you can use control IAM roles setup to control the permissions of a file. Cloud providers like Google (Cloud storage), Amazon (AWS S3), etc provides a secure and scalable infrastructure to maintain files in the cloud.
In general setup, cloud storage provides you with buckets which are tagged with a global unique identification. For a multi-tenant setup you can create multiple buckets for individual tenants and store the necessary data feeds in it. Next, you can have jobs batch or streaming jobs using kettle (Pentaho) to push it to the right database based on the unique bucket definition.
Alternatively, you can also push (like other answers) to a streaming setup (like ActiveMQ, Kafka, etc) with user specific topics and have a streaming service (using java or pentaho) to ingest the data to respective database based on the topic.
Hope this helps :)
I cannot realistically give any specific advice without knowing more
about your system. However, based on my experience, I would
recommend switching to message queues, something like Kafka would
work nicely.
Yes, cloud providers offer enough security for static file storage. You can
limit access however you see fit, for example using AWS S3.
1- The multi tenancy may create a bit of issue while transferring the files. But from what information you have given the process of flat file movement across application will not be impacted. Still you can think of moving to MQ mode for passing the data across.
2-From data security view, AWS provides lot of features at access level, MFA, etc. If it needs to be highly secured i would recommend to get AWS Private cloud where nothing is shared with any one at any level.
I am working on a little android app that will be requesting json data 3 times a day, this json data will be populated base on data that is fetch from 3 different web sites.
My question is what cloud solution will allow me to used some sort of script/language (python,perl or java) to fetch the data within the cloud itself(make the fetching automatic) from the these site and make it available in a json friendly format so my app can fetch it .
One of the AWS services I have been looking at was API gateway but i don't see a way of me fetching the data from within AWS to later make it public as a json API.
thanks in advance for the help
Have you considered running scheduled events on AWS Lambda - https://docs.aws.amazon.com/lambda/latest/dg/invoking-lambda-function.html#supported-event-source-scheduled-events
At first, Sorry for my poor English.
I write a android app about movie information. In the GAE server,Parsing the movie data everyday. However, To reduce GAE computing flow effectively and access time from the client, I want to take the all parsing results daily to be a Json file. I think all of this reasonable...?
So, How to save a JSON file in my GAE?
If you are planning on using the Google toolstack you will need to persist the json file to one of persistent file storage services like Blobstore or Google Cloud Storage.
If it's a small json you could put it into Google's Datastore.
You could always connect to other cloud storage services like Amazon's S3, but since you're using Google's cloud products it might be easier to stick to them.
I have a Java desktop application which stores data into a SQLite db every 10 mins which is stored locally in the user's system. I have a cloud interface to visualize this data which uses PHP and MySQL.
I need to fetch data from the local db of the users. I think I will have to write a RESTful web service in Java so that the database which is locally stored is not exposed and the data is obtained by the web service.
I am a bit confused with this. Am I going in the right direction here?
You're on the right track. Your desktop application can connect to the cloud server and upload its information. You'll need to create an http request in your desktop application.
I think you should maybe approach this the other way round, rather than the web service getting data from your local system, your local system should upload its data to your cloud interface, perhaps using a RESTful web service on the server rather than the client.
Hope that helps.