There seems to be a 1MB limit on Android's Cursor Window size which limits the ability to read BLOBs from SQLite. I know you may say we should not store BLOBs in database but by definition, BLOB is considered a Binary Large Object and if there was no need to store them in database, there was no need to implement such object type in any database engines.
The 1 MB limit on the implementation of Cursor however, seems to be insufficient in almost all cases. I need to store my binary data for valid reasons in SQLite database and they are well over 1 MB. SQLite is capable of handling BLOBs perfectly since the C API is working perfectly fine in Xcode (iPhone platform) to retrieve large objects without any issues.
I'm wondering if we can possibly access the BLOB data in Android without using cursors. I am thinking of a lower level access to Sqlite in Java. Any suggestions?
As CL mentioned, using NDK is indeed a way to access Sqlite natively via C language in Java language. However I realized it could get really messy if I wanted to write a custom wrapper myself and try to access the functions in Java.
After searching around, I came across a brilliant open source project called Sqlite4java which is a tight wrapper around Sqlite, compiled to use on various platforms including Android. This library allows you to interact with Sqlite without using Android Cursor which removes the limitations.
I am able to retrieve 20 MB of Blob in 480 milliseconds. This is even faster than reading a small record from Sqlite via Cursors. I believe this can be used to enhance any query to Sqlite by skipping the use of Cursor. Here's the link to this great library: http://code.google.com/p/sqlite4java/
Android's Java API always has the 1 MB limit.
You should not store BLOBs of that size in the database; the file system is more efficient at handling them.
If you really want to use BLOBs, you have to go through the NDK to access the C API directly.
Related
I am working on a project where I have extracted images from sensor and saved them to the operating system directory. I have a Java API for uploading images to the server.
I need to upload these images and some other data typically float data type to the main server.
I need to decide an inter-mediator such as a database where I store those images and make connection through java to upload them or use HDFS.
Can some body please advise me, which option will be best for storing images? Database or HDFS?
Note: Images are up to 150 thousand can be more in future.
I think the best way to do that is to keep the floats you need and metadata of the images in the database. For easier searching and querying and easier interaction with the Java. The actual images are best stored on a file system to decrease the transformation from and to the database. I believe a simple file system would be good enough for that size of images. You probably won't use any of the fancy HDFS functions like map reduce and stuff like that. But that's up to you.
So in this case if a standard file system isn't good enough for you and you want something bigger then HDFS is the way to go. So the proper way would be a mixture of the two.
It totally depends on the usecase , you can choose
HDFS : when you wanna read them as a whole or transfer or process them to do any manipulation upon the images data and store or do someother action based on the processed results. In simple, if you wanna do Map-Reduce operation. And reading images in HDFS is sequentially , if you wanna perform to fetch particular image based on certain selection criteria, then it costly and performance impacted operations.
Database : It is better for query based operation where you wanna query or do DML operations upon images on certain criteria basis, In simple, WHERE conditions. But this is totally time consuming process, when you wanna process as a chunk. And the performance will be obviously very slow as you wanna store 150thousand of images
So My suggestion based on the requirement, you wanna store images as intermediate, it will be better to store in HDFS itself.
150.000 images is not considered a huge amount today. If an average of 10 MB is assumed for each image (uncompressed) the amount of data is 1.5 TB, which should be possible to store in an off-the-shelf database (with off-the-shelf hardware, i.e. a Linux box with some RAID disks) like postgreSQL. I'm no expert in HDFS even though I tried products in the same family as HDFS I find them easy to use, I guess you could try Hadoop then for processing of the images as well if you are looking for a way to parallelize the processing. Even though this product family is nice I would still use a standard database like postgreSQL if parallelisation is not really needed by nature (like you get in HDFS).
I use PHP to access my database and generate an XML file online. My android app then gets that XML file, parses it, and inserts the data into a SQLite database.
This works just fine but is INSANELY slow. We have an iOS app and an Android App both doing the same thing... the android app takes 7-10 seconds every time the user wants refreshed data, while the iOS app only takes 2-3 seconds at most.
There aren't a lot of records - 30-50 on average. There is a lot of content - some large articles, and each with 2-10 photos (I'm not downloading the photos - just importing their url, size...etc)
I followed an example on how to use Sax to import my XML (supposedly the fastest way).
TLDR:
Is there a better way I can format my data to make it MUCH quicker than how I'm doing it now? CSV? Use PHP to generate SQLite Insert statements? What is the "norm" and/or "best" for this?
Edit:
The more I read, the more it sounds like the difference between JSON and XML are miniscule, and can even be faster with XML if it's large data (like articles) instead of JSON. Not sure this is correct, just details from further reading.
You should try using JSON instead of XML i think it might be a lot faster to work with that. It is supported on Android and as far as I know iOS can handle it as well.
I used to create a SQLite db file and gzip it, then unzip it on device and use that directly. (Not a good way for sure)
For later data updates I used json to transfer data. JSON can surely handle large articles, but if you prefer you can just put urls to the articles in JSON and fetch them in subsequent transfers.
Instead of using XML or JSON, look into Google's Protobuf :
https://developers.google.com/protocol-buffers/docs/overview
http://code.google.com/p/protobuf/
since you are on PHP, you will need to find an implementation that works for you, here is a list :
http://code.google.com/p/protobuf/wiki/ThirdPartyAddOns
Going forward, this will be a very nice way to transfer and marshall data around. Please let us know if this works for you.
So I'm following a tutorial on how to create a simple Android app that stores a bunch of random quotes. I've noticed that they used a SQLite database and many other Android tutorials uses SQLite to store things. I'm not very used to SQLite. Can I achieve the same result just by storing the quotes in a hashtable?
What are the differences between a SQLite database and a hashtable in terms of performance?
What are the differences between a SQLite database and a hashtable in terms of performance?
An (in-memory) hash table is (you probably should use HashMap) is going to be faster. However, it does not address the problem of making your quotes "persist" when your app has shut down.
SQLite is a database, and the main point of a database is that the data persists.
(There are a whole bunch of other benefits in using a database of some kind. One that is potentially relevant to you is that you can store more stuff in a typical database than you can hold in memory. The database stores stuff on your device's hard drive / SSD / whatever which has far more capacity than main memory.)
http://www.brighthub.com/mobile/google-android/articles/25023.aspx
I suggest SQLite because u can access the database very easily and it is easy to understand and view the database tables also very easy through DDMS plugin for eclipse The above link helps you in DDMS
It's up to your intention. If you want to store data such as app preference, user score.. use SQLite. These data will be available for the next launch of application. Hashtable is in memory, of course it is faster than SQLite, all information stored in Hashtable will be lost when the app is deactived or terminated.
I've read this Android save app settings/data in Internal/External Storage but I don't think i've really got an answer.
I'm developing an app where I should store some "cache" data, like a dozen of images and some strings (json).
Initially I was storing all that on the sdcard, (external storage) but later i thought that this could be deprecated by the SDCard deprecation in most recent devices.
After reading a bit, I understood that external storage is not only sdcard, but "a removable storage media (such as an SD card) or an internal (non-removable) storage" so it should not be deprecated but...it's shared space, and there is not ownership over the files stored there, so the first problem was that I was unable to delete them when the app was deleted.
So I changed to the Internal Storage, to avoid having the files/images "public" and also having them removed after app deletion.
Is this the recommended approach?
On devices older with low internal storage but with a lot of space in the SDcard is this a good approach?
My application supports from 1.6 to 4.0 (so far) so I have a lot of legacy devices... and I must have the app working (well) on all.
Looking forward for some interesting answers!
It depends on the type of data you are wanting to store.
You mention it's cached data so myassumption is that it should not matter if for some reason it all disappears. This leads me to believe that you should be using the getCacheDir(). In this case the system will remove files if the cache becomes too big so devices with low internal storage shouldn't present a problem (although it is recommended to manage this your self anyway), it's relatively secure and it will be managed by the app so if there is an uninstall it will be removed.
getExternalCacheDir() was introduced in 2.2 so isn't any use to you unless you would like to detect the version and switch between the 2 caching directories getExternalCacheDir() doesn't provide security so data could be accessed by anyway with access to the SD card. The only reason I could think you might want to do this is because of size of cache you desire but from your description the data doesn't seem excessive.
UPDATED from comment:
although this is a specific case where it's cache...but I don't want
it to be deleted whenever the system wants. It's the kind of cache
that I need the to app decide when to purge. What is the main concern
of storing in on "normal" internal storage without being on the cache
dir?
If you get to the stage where the system is cleaning up internal cached data because storage is so low then you should probably leave it to clean up this sort of app data. By using standard internal data storage you are bypassing this safe guard which would probably create more unpleasant problems than having app data deleted.
If your data is of high importance then I would suggest trying to identify specific data that is more important and managing that separately. If this data you identify needs to be secure then internal storage using files or a db (depending on the data type) seems like your only real option but you would have to be wary of this data building up.
UPDATED from comment
What do you think about using SharedPreferences to save string data?
Is there a limit on SharedPreference string saved size? Is it a (good)
possibility?
I have used shared preferences to store relatively big json strings in the past with no problem, I find it's simpler than using the databases for primitive data types (and strings) where there are limited values to save. However when you have images or lot's of values, management becomes more complex. Also you will have the same problem as you would with standard internal storage in terms of storage space.
I would keep images on the external storage, probably in a "hidden" folder (adding a dot at the beggining of the folder's name: .folder) and also a the Media Scanner "avoider" (.nomedia), because, as you pointed, you want the application to work in old devices, and precisely those devices don't have to much internal memory.
You can also add an option in your application to remove that folder, so in case the user wants to uninstall, he can do that before.
I would use internal storage for cache. This will take away the chance of a user being able to access the files used in your app. I would go for internal
I need to make a data file to hold two empty tables with many fields
I have been successful at making a MySQL 5.5 Table with its DOS style IDE OK.
And MySQL registers with NetBeans very well.
Can I make a script to build this from with in my program
or can this be done directly in Java and get a Java file to read file parameters as its executed to create the data base file name.
I have no idea what direction to take to do this and what's possible.
I'm writing in Java and Delphi and Delphi has no MySQL support.
Has any one done anything similar before and how did they do it
The database is to hold 70 meduim size pictures, How slow will updates be when accessing pictures and should I use JPEG or BMP storage for wireless Java apps?
Can Java manage JPEG files and display them?
Lots of questions in one post. Let me try to address each point individually.
Yes, you can write Java code to create your MySQL database for you. On the other hand, tools such Liquibase can do this for you—you just need to learn its XML configuration syntax.
Lots of people have successfully written Java and Delphi code to access MySQL databases. If I recall correctly, at the very least Delphi supports MySQL access through ODBC, if not, through 3rd-party custom components.
Java can store & retrieve JPEG images to/from a database just like any other language, though, in practice, I wouldn't do it that way. Rather, I'd store the images themselves in the file system and just store their locations in the database. RDBMSes weren't really built with handling large binary BLOBs in mind.
How would you like to display the images? If on screen via a desktop client, then, yes, Java's Swing components can easily and readily display JPEG, even GIF & PNG images. If via a Web browser or remote client, then it's really just a matter of serving the images over HTTP to the browser/client app.