I'm trying to work on an app which uses GTFS. This may seems like a stupid question but I couldn't find any answer to it.
The GTFS for Israel, a rather small country with not so many buses infrastructure, is around 120 MB zipped file.
Right now the only possible way I could think of for getting it working is to download the file, but downloading 120 MB using the phone could take quite a long time. Sure you can do this only once and save it in a database on the phone, but it still requires downloading 120 MB.
Since it is zipped, I can't unzip it over the server and than just get the txt files..
So basically I'm asking, How can I get the information to the phone, without downloading the zipped file?
I've seen and used apps which uses that same GTFS file, and they load up really fast, even on the first load..
I hope you understand my issue, not sure how to explain it better.
Thanks!
P.s I would make an iPhone app too, and it's the same issue, hence the iPhone tag
One approach might be to preprocess the GTFS data during your app development. You could load it into a SQLite database, and use Core Data to get the data you need out of the file at runtime. This also gives you an opportunity to include only the data that you actually need for your app - it doesn't make sense to ask users to download extra data that they won't need.
Use protocol binary format (pbf) formely google and now open source. It is compact and very fast searchable, so no need to decompress it on a device and load it into a database on that device because pbf acts as a database. Just include pbf library in your code to query it. Of course you have to compress it once before distributing the data online.
Related
I'm working on a small side-project for our company that does the following:
PDF-based documents received through Office 365 Outlook are temporarily stored in OneDrive, using Power Automate
Text data is extracted from the PDFs using a few Java libraries
Based on extracted data an appropriate filename and filepath is created
The PDFs are permanently saved in OneDrive
The issue right now is that my Java program is locally-run, i.e. point 2,3,4 require code to run 24/7 on my PC. I'd like to transition to a Cloud-based solution.
What is the easiest way to accomplish this? The solution doesn't have to be free, but shouldn't cost more than $20/mo. Our company already has an Azure subscription, though I'm not familiar yet with Azure.
What you are looking for is a solution that uses a serverless computing execution model. Azure Functions seems to be a possible choice here. It does seem to have input bindings that respond to OneDrive files and an likewise output bindings.
The cost will depend on the number of documents, not the time the solution is available. I assume we are talking about a small number of documents a month so this will come out cheaper than other execution models.
I'm building a dictionary application and I have a problem right now. My application's is 16MB and when I install it on a phone, Database files copies to the data folder and in the manage apps section I see that my application size is 32MB (my app+data folder).
I don't cheat user, I want to say, my app is 16MB, but when user install it , it become 32MB. Why? this is a negative point and I want to solve it. I want my app uses only 16MB in users phone. just that
How I can fix this? I have to read and write in assets folder directly or there is other solution? this is a problem in low storage size phones. how I can fix this?
I am not sure how your database is structured in terms of whether it is a pre-loaded database wherein you just include you .db file with all the data OR is it something where in you push all your DB content with the app and then at the time of app installation you actually install all you data in the DB.
In case of the latter situation you double the size of your app because you already have data content (in files) which you want to use to populate your database (say 16 mb in this case). Then you use these files to actually create your DB file (which is 16mb again) and this doubles the size of the app.
So what you could do is pre-populate your DB content in a .db file and then just use this file directly as the Db file in your app (this will keep it to 16mb). Follow this tutorial :
http://www.reigndesign.com/blog/using-your-own-sqlite-database-in-android-applications/
Hope this helps.
Not sure I fully understand your situation.
Do you have a roughly 16MB dictionary, that is packaged inside your app as string constants in your code or some resource file or something (to make it 16MB) and then, when your app installs or first launches, you also write this dictionary into your app's database?
If so, then now you have 2 copies of your dictionary around to make it 32MB.
To solve this, either keep only one copy in your app, or download the dictionary from somewhere to get it into your database rather than storing it as a constant in your app.
I use PHP to access my database and generate an XML file online. My android app then gets that XML file, parses it, and inserts the data into a SQLite database.
This works just fine but is INSANELY slow. We have an iOS app and an Android App both doing the same thing... the android app takes 7-10 seconds every time the user wants refreshed data, while the iOS app only takes 2-3 seconds at most.
There aren't a lot of records - 30-50 on average. There is a lot of content - some large articles, and each with 2-10 photos (I'm not downloading the photos - just importing their url, size...etc)
I followed an example on how to use Sax to import my XML (supposedly the fastest way).
TLDR:
Is there a better way I can format my data to make it MUCH quicker than how I'm doing it now? CSV? Use PHP to generate SQLite Insert statements? What is the "norm" and/or "best" for this?
Edit:
The more I read, the more it sounds like the difference between JSON and XML are miniscule, and can even be faster with XML if it's large data (like articles) instead of JSON. Not sure this is correct, just details from further reading.
You should try using JSON instead of XML i think it might be a lot faster to work with that. It is supported on Android and as far as I know iOS can handle it as well.
I used to create a SQLite db file and gzip it, then unzip it on device and use that directly. (Not a good way for sure)
For later data updates I used json to transfer data. JSON can surely handle large articles, but if you prefer you can just put urls to the articles in JSON and fetch them in subsequent transfers.
Instead of using XML or JSON, look into Google's Protobuf :
https://developers.google.com/protocol-buffers/docs/overview
http://code.google.com/p/protobuf/
since you are on PHP, you will need to find an implementation that works for you, here is a list :
http://code.google.com/p/protobuf/wiki/ThirdPartyAddOns
Going forward, this will be a very nice way to transfer and marshall data around. Please let us know if this works for you.
The application I'm trying to build will have a lot of images displayed (in ImageViews), and I'm not fetching them from a server/online service as it will need to be used offline. I know I can just dump them in the res/drawable directories, but I was wondering if there's any way to optimize this. Is there a way to somehow compress these images (besides making them smaller, they're already as small as I need) or use some other sort of android tool to better store them locally on the device?
I could just be overlooking a well used feature, and if so, it'd be great if someone could point me to that.
Edit: If I were to compress the images somehow, I would need to decompress at runtime or something, and that would take another thread/loading time. I'm not sure how to do that either, so I'm just brainstorming various ways, and I thought someone here would've come across this at some point.
If you haven't already, this is a good read - http://developer.android.com/guide/practices/ui_guidelines/icon_design.html#design-tips
When saving image assets, remove unnecessary metadata
Although the Android SDK tools will automatically compress PNGs when
packaging application resources into the application binary, a good
practice is to remove unnecessary headers and metadata from your PNG
assets. Tools such as OptiPNG or Pngcrush can ensure that this
metadata is removed and that your image asset file sizes are
optimized.
Outside of all other compression logic the above would be the place to start. Also when you say "optimize" - do you mean optimize the way images/drawables are loaded in your app or just the amount of space (on disk) the app will consume?
I'm looking to create an Android (altho for iOS the problem will be the same) application which will function pretty much as a webshop.
It will contain a lot of products - which can be acces through any way we want since that still has to be build.
The problem is, we created a plain text file to test the size, and it turns out that even a selection of the products, with no structure (XML, JSON..) is already 300mb.
Once we add a structure, this will logically only cause more overhead and increase this size.
Like I said, pretty much anything is possible in matters of receiving the data.
They can build an API to be able to fetch products once at a time when needed, or 1 big file to parse in a background process...
However, one of the wishes is being (as much as possible) offline. This would normally mean saving all the data into a database on the phone, but if this will result in 300mb on your SD card, this is no good.
To sum it up what I exactly want to know;
Are there any other ways to handle big data like this, without having to keep a connection to internet constantly, or having to download 300mb on someone's phone.
Some kind of compression, special way to save it in the database... any ideas are welcome.