I'm very new to App Engine, but familiar with Java. I need help understanding the Blobs API of Google App Engine.
I kind of went through the documentation online, but unfortunately did not understand much of it.
I also spent hours searching online for blogs that contain explanation/tutorial/walk-through about blobs on App engine to no avail.
I'm using the Data store API in the same project without troubles.
I need help with the following:
How do you upload a text file as a blob?
What is a blob key and how does it figure in creation of a blob?
Once I have the blob uploaded how do I open it and read it into a String (for modifying)?
How do I write the modified string back to the blob? [Not necessary to be append, one write would do]
Thank you!
An explanation or a link to a tutorial would be nice. :)
P.S If it means anything, I'm on eclipse.
Follow the example in Blobstore API, on how to upload file.
BlobKey is a long unique identifier (non-easily-guessable) generated by Blobstore when you store a blob into it.
Blobs in Blobstore are immutable, so you can not change their content once they are fully saved into blobstore. You must read blob data, modify it, save it as new blob (and potentially delete old blob). See Files API.
See 3.
Since you need to update blob data, then if your data size is smaller then 1Mb, you might be better off just storing blob into Datastore Entity. You can use a property of Blob or Text type.
Related
I'm a beginner and have never dealt with cloud-based solutions yet before, so apologies for the dumb question.
I have an Azure Blob Storage containing PDF files from which I want to extract data using PDFBox. Because PDFbox can't load blobs directly, I currently download these files locally first. However, eventually my project will need to become fully Cloud-based, preferably as an Azure Function.
The main hurdle therefore is figuring out how my Azure Function should access the files. When using the console inside my Azure Function I noticed it comes with a file storage. Can the Function download blobs and store them here before processing it? Does this file storage work the same as a local environment or are there differences to keep in mind?
I'm only looking to store files temporarily here, for only a few minutes at a time.
The main hurdle therefore is figuring out how my Azure Function should
access the files. When using the console inside my Azure Function I
noticed it comes with a file storage.
Yes, all of the information of your deployed azure function is stored in the file storage you set.(It is defined when you create the function app.)
Can the Function download blobs and store them here before processing
it? Does this file storage work the same as a local environment or are
there differences to keep in mind?
Yes, you can. And the root directory is D:/home/site/wwwroot. So if you don't specify, the file you create will be in this directory.
Remember to delete the files, because the storage space is limited. It is based on the plan you selected.
I'm only looking to store files temporarily here, for only a few
minutes at a time.
By the way, if you get a file from blob storage, at this time you have completely got its data. You can process the obtained data directly in the code without temporarily storing it in the current folder. (Of course, if you have special needs, please ignore this one.)
You can use a blob trigger or input binding to load a blob into memory of your function for processing by PDFBox.
With regards to the local file system, you can read about more about it here. From the description of your problem I think a blob trigger or input binding should be sufficient for you.
I'm using Google App Engine and i need to store a big file (2-20Mb). It is a text file that i convert to a JSONArray. I need to be able to add JSONObjects to this array and to be able to read it.
I wanted to use Blobs but I noticed that blobs can't be updated(is it true??).
I don't want to enable billing thus, I can't use FileService(or...?).
Storing eache JSONOBject in the db explodes my reading quota.
With cache, the objects are sometimes removed.
Do you see a way to solve my problem?
Best regards!
This is what the blobstore is for.
https://developers.google.com/appengine/docs/java/blobstore/overview
The Blobstore API allows your application to serve data objects, called blobs, that are much larger than the size allowed for objects in the Datastore service. Blobs are useful for serving large files, such as video or image files, and for allowing users to upload large data files. Blobs are created by uploading a file through an HTTP request.
You get free quote here also.
No, you can't change them once you have uploaded them. If you want to do that then store your data as structured data in the datastore instead. But you can delete and replace blobs.
I'm trying to work on an app which uses GTFS. This may seems like a stupid question but I couldn't find any answer to it.
The GTFS for Israel, a rather small country with not so many buses infrastructure, is around 120 MB zipped file.
Right now the only possible way I could think of for getting it working is to download the file, but downloading 120 MB using the phone could take quite a long time. Sure you can do this only once and save it in a database on the phone, but it still requires downloading 120 MB.
Since it is zipped, I can't unzip it over the server and than just get the txt files..
So basically I'm asking, How can I get the information to the phone, without downloading the zipped file?
I've seen and used apps which uses that same GTFS file, and they load up really fast, even on the first load..
I hope you understand my issue, not sure how to explain it better.
Thanks!
P.s I would make an iPhone app too, and it's the same issue, hence the iPhone tag
One approach might be to preprocess the GTFS data during your app development. You could load it into a SQLite database, and use Core Data to get the data you need out of the file at runtime. This also gives you an opportunity to include only the data that you actually need for your app - it doesn't make sense to ask users to download extra data that they won't need.
Use protocol binary format (pbf) formely google and now open source. It is compact and very fast searchable, so no need to decompress it on a device and load it into a database on that device because pbf acts as a database. Just include pbf library in your code to query it. Of course you have to compress it once before distributing the data online.
I use PHP to access my database and generate an XML file online. My android app then gets that XML file, parses it, and inserts the data into a SQLite database.
This works just fine but is INSANELY slow. We have an iOS app and an Android App both doing the same thing... the android app takes 7-10 seconds every time the user wants refreshed data, while the iOS app only takes 2-3 seconds at most.
There aren't a lot of records - 30-50 on average. There is a lot of content - some large articles, and each with 2-10 photos (I'm not downloading the photos - just importing their url, size...etc)
I followed an example on how to use Sax to import my XML (supposedly the fastest way).
TLDR:
Is there a better way I can format my data to make it MUCH quicker than how I'm doing it now? CSV? Use PHP to generate SQLite Insert statements? What is the "norm" and/or "best" for this?
Edit:
The more I read, the more it sounds like the difference between JSON and XML are miniscule, and can even be faster with XML if it's large data (like articles) instead of JSON. Not sure this is correct, just details from further reading.
You should try using JSON instead of XML i think it might be a lot faster to work with that. It is supported on Android and as far as I know iOS can handle it as well.
I used to create a SQLite db file and gzip it, then unzip it on device and use that directly. (Not a good way for sure)
For later data updates I used json to transfer data. JSON can surely handle large articles, but if you prefer you can just put urls to the articles in JSON and fetch them in subsequent transfers.
Instead of using XML or JSON, look into Google's Protobuf :
https://developers.google.com/protocol-buffers/docs/overview
http://code.google.com/p/protobuf/
since you are on PHP, you will need to find an implementation that works for you, here is a list :
http://code.google.com/p/protobuf/wiki/ThirdPartyAddOns
Going forward, this will be a very nice way to transfer and marshall data around. Please let us know if this works for you.
I am using GAE with Java to dynamically create a "text file" based on information provided in a form. With the form parameters, I successfully generated a String that should serve as the text in the txt file. However I am not sure exactly how to get that String in an easy to read format. Ultimately, I would like to have an iOS app be able to read the text. These are the two routes that I was thinking of using, but I've encountered problems with both.
Store the text in the database then showing that in another page. Unfortunately I exceed the 500 char limit for a String with GAE. I then tried using the Text class, but as it's not queryable, I'm not sure how to access it from a jsp page.
Dynamically generate a text file with the String of file contents. This option seems more of the "right way" by actually creating a new file. To be quite honest, I just don't know how to do that. I have looked extensively online, but have not found an answer. I've looked into Blob API, but what that allow me to create a brand new file that can be accessed at another URL location? Most of the documentation deals with images or other file types.
All responses are very much appreciated.
Thanks.
For 1, just remember the key of the entity, and fetch an entity with that key afterwards. For 2, you can serve text files as well. Now we're offering Google Cloud Storage integration, so it might be a better option.