I'm using Google App Engine and i need to store a big file (2-20Mb). It is a text file that i convert to a JSONArray. I need to be able to add JSONObjects to this array and to be able to read it.
I wanted to use Blobs but I noticed that blobs can't be updated(is it true??).
I don't want to enable billing thus, I can't use FileService(or...?).
Storing eache JSONOBject in the db explodes my reading quota.
With cache, the objects are sometimes removed.
Do you see a way to solve my problem?
Best regards!
This is what the blobstore is for.
https://developers.google.com/appengine/docs/java/blobstore/overview
The Blobstore API allows your application to serve data objects, called blobs, that are much larger than the size allowed for objects in the Datastore service. Blobs are useful for serving large files, such as video or image files, and for allowing users to upload large data files. Blobs are created by uploading a file through an HTTP request.
You get free quote here also.
No, you can't change them once you have uploaded them. If you want to do that then store your data as structured data in the datastore instead. But you can delete and replace blobs.
Related
I am wondering if there is a way to cache arbitrary data from web requests onto the disk with Android. The flow I am thinking of is as follows:
The data is stored as a key value pair where the key is some identifier and the value is the raw data. Before actually making my web request, I check to see if the key is in the cache, if so, I skip making the web request. If the key does not exist in the cache, then I make the web request and store the data on the disk. I would like the cached data to be accessible across multiple runs of the app so that I don't have to make the web request again every time I start the app.
I was considering using SharedPreferences for this. Would SharedPreferences be the best way to go about this? Is it okay to store 1 megabyte of data in a single key in SharedPreferences?
The best solution to storing cache files is to store them in a cache directory. Luckily, the Android API provides a solution to this problem: Context#getCacheDir. You are able to create files in the directory returned, you can use a map to store an identifier for each file in order to retrieve them.
Although, this solution has a few limitations:
The system will automatically delete files in this directory as disk space is needed elsewhere on the device.
Cache data should only be used for temporary storage of information.
I may be coming late, but a couple years ago I made a library just for this:
https://github.com/fcopardo/EasyRest
The idea is to allow the app to operate with unstable or no connection without having to implement a secondary data layer for persisting data, instead, it keeps the responses for as long as you want, and refresh them without forcing the user to wait. Take a look, you may get some ideas.
I need to sore attachments at server side. I can store them either under blob column of database or under file directory.
My question is which one is more reliable, scalable and maintainable?
EDIT:-
if we go for file system, we have to handle synchroniztion yourself. Is n't it ? For example if two users are trying to create/update the File under same directory how will we handle concurrency with filesystem?
Storing data in directory is more reliable due to indexing and data fetch and other operation. Just store the path of the file into DB and store that file into directory.
When there's lot's of data store request came on server it's very hard and complex to handle so much request.
So it's better to store data on directory so accessing of data becomes more faster and when the daily scale of DB storage increase then these become so important so when you start any system first of all study it well and then decide that what to do or which technique will be the best ?
When more data are there in DB then clustering and indexing become more important.
If you want to use it for small data storage then blob it good option but for large data I ll not recommend you because I have made online data store web application and faced this situation so at end I have used to store data in directory and just path in DB.
I am looking for a good possibility to store data online. I am programming with Java and Android. So there should be an interface to get access to these data. The most files are images. There is an increasing number of images. One image has a file size of nearly 200kb.
What is a common way to store these data? I need a good performance. So there should be a fast response and unlimited traffic. Maybe you can show me some options for secure data storage.
I have looked for webservers to store data. But many of these do not allow to store application data like images.
As I understand, you don't want to use DB for storing images. Ok, so the solution is to use file storage. You may want to take a look on the Amazon S3 (to my mind, great solution for storing static content) or Google Cloud Storage
I'm very new to App Engine, but familiar with Java. I need help understanding the Blobs API of Google App Engine.
I kind of went through the documentation online, but unfortunately did not understand much of it.
I also spent hours searching online for blogs that contain explanation/tutorial/walk-through about blobs on App engine to no avail.
I'm using the Data store API in the same project without troubles.
I need help with the following:
How do you upload a text file as a blob?
What is a blob key and how does it figure in creation of a blob?
Once I have the blob uploaded how do I open it and read it into a String (for modifying)?
How do I write the modified string back to the blob? [Not necessary to be append, one write would do]
Thank you!
An explanation or a link to a tutorial would be nice. :)
P.S If it means anything, I'm on eclipse.
Follow the example in Blobstore API, on how to upload file.
BlobKey is a long unique identifier (non-easily-guessable) generated by Blobstore when you store a blob into it.
Blobs in Blobstore are immutable, so you can not change their content once they are fully saved into blobstore. You must read blob data, modify it, save it as new blob (and potentially delete old blob). See Files API.
See 3.
Since you need to update blob data, then if your data size is smaller then 1Mb, you might be better off just storing blob into Datastore Entity. You can use a property of Blob or Text type.
i am using Google app engine for my development, my project involves around 60 PDfs to be available for users to download.
when i try to upload the project by clicking deploy button in eclipse i get the error app limit exceeded.
i just want to know if i try to use the paid account is there is a different in the application size in paid account or not?
as far as i know its 150 MB for now
You should use Blobstore service to store your PDF files and keep application only for files needed by your application logic and presentation, not data. Here is description of the Blobstore:
The Blobstore API allows your app to serve data objects, called blobs,
that are much larger than the size allowed for objects in the
Datastore service. Blobs are created by uploading a file through an
HTTP request. Typically, your apps will do this by presenting a form
with a file upload field to the user. When the form is submitted, the
Blobstore creates a blob from the file's contents and returns an
opaque reference to the blob, called a blob key, which you can later
use to serve the blob.
All good advice above, try to avoid putting content like that in your code. My app hit this issue and only has about 10MB of code/images/resources. What takes up a lot of space is the GWT compiling of 15 permutations of your app.
One thing that helped me, was changing my GWT javascript generation output style from Details to Obfuscated, resulting in much smaller code. You can also limit the number of permutations being created.
https://developers.google.com/web-toolkit/doc/1.6/FAQ_DebuggingAndCompiling#Can_I_speed_up_the_GWT_compiler?
According to http://code.google.com/intl/de/appengine/docs/quotas.html#Deployments the applications may not exceed 10 MB.
upto 10MB data u can upload to ur app engine
see following link
http://code.google.com/appengine/docs/quotas.html