I was trying to develop an mobile app which have some similar idea just like Uber, which is real time update driver & customer location. So here I am seeking for suggestion for what I was thinking for the app structure.
For what I research, in order to provide fast real time update location, I may need to make use of real time database such as Firebase for the backend. So, I was thinking to combine 2 different type of database to achieve what I was thinking...
Firebase - For real time fast update user location
MySQL - For backend api business logic
However, I have no experience with firebase, so I hope you all can give some advise. I plan to only store the user location coordinate information in firebase database, then retrieve it from mobile app to update realtime.
My problem is I not sure should I forever persist those driver coordinate data in firebase database? Since the coordinate data keep changing update in firebase, so should I delete those coordinate data from firebase as soon as the driver have reach the destination. (No need to keep those data persist, only real time data keep change on firebase)
Thanks for reading such long question, I will also happy that if you all can remind to me any other concern if I use 2 different database for my application.
You'll typically keep a list of drivers and their locations in Firebase:
driverlocations
driver1id: location
driver2id: location
This means that you're not adding new data, but updating existing data. So you don't have to worry about the list constantly growing.
To ensure you don't have stale data for drivers that closed the app/stopped driving, you can use Firebase's onDisconnect() handlers to remove the data when they disconnect.
Now querying this data for nearby drivers is still tricky. You'll want to look at GeoFire for that. I recently explained why that is needed and how it works here: Sort array by distance near user location from firebase
Related
As for similar questions on this topic and on ChildEventListener, there is no relevant answer, so heres mine.
I have a local SQLite DB which holds all the data, I also have Firebase realtime database which I'm updating with new entries or real time changes across all users. I'm currently doing it with the use of ChildEventListener as follows:
DatabaseReference rootRef = FirebaseDatabase.getInstance().getDatabase().getReference();
DatabaseReference childRef = rootRef.child("my_root");
ChildEventListener eventListener = new ChildEventListener()
{
....
};
childRef.addChildEventListener(eventListener);
As for functionality, With this code I can get realtime changes on childs, get new entries, deleted childs and everything I need but there is one problem. When this specific activity with the listener loads up, the onChildAdded listener gets called enormous amounts of times for every child on this root, as stated on the documentation:
child_added is triggered once for each existing child and then again every time a new child is added to the specified path
So I though to gain focus on the items that I really need and I have done it with:
rootRef.orderByKey().startAt("-WhatTF123456789")...
But then I have lost my CRUD capabilities because it's listening to the new entries and not all of them.
So I came up with a solution. Keep node with all the changes that has been made to the FireBase database and a node with all the users that have read and made the changes to the local DB to know who needs an update, Then use addChildEventListener to this specific node. But that seems redundant.
What is my options to handle this kind of situation?
The onChildAdded listener gets called enormous amounts of times for every child on this root.
As you already mentioned and as the docs states, this is the expected behaviour. Usually, is not recommended to attach a ChildEventListener on a node (root node) that contains huge amount of data. Please be careful about this practice because when downloading large amount of data, you can get erros like: OutOfMemoryError. This is happening because you implicitly download the entire node that you are listening to, along with all the data beneath it. That data might be present as simple properties or, as complex objects. So it can be considered a waste of resource and bandwidth. In this case, the best approach is to flatten the database as much as possible. If you are new to NoSQL databases, this practice is called denormalization and is a common practice when it comes to Firebase. For a better understanding, I recommend you take a look at:
This video, Denormalization is normal with the Firebase Database.
Official docs regarding Best practices for data structure in Firebase realtime database.
My answer from this post: What is denormalization in Firebase Cloud Firestore?
This article, Structuring your Firebase Data correctly for a Complex App.
This article, NoSQL data modeling techniques.
Please also note that when you are duplicating data, there is one thing that need to keep in mind. In the same way you are adding data, you need to maintain it. With other words, if you want to update/detele an item, you need to do it in every place that it exists.
I also recommend you to see the last part of my answer from the following post:
What is the correct way to structure this kind of data in firestore?
It is for Cloud Firestore but same rules apply to Firebase realtime database.
But then I have lost my CRUD capabilities because it's listening to the new entries and not all of them.
Everything in Firebase is about listeners. You cannot get realtime updates for objects within a node, unless you are listening to them. So you cannot limit the results and expect to get updates from objects that you are not listening to. If you need to get updates for all objects within a node, you need to listen to all of them. Because this approach isn't practical at all, you can either use denormalization as explained above or to restrict the results by using queries that can help you limit the amount of data that you get from the database. Regarding your solutions, the second one is much preferred but you can also consider another approach which would be to load data in smaller chunks according to a timestamp property, or according to any other property that you need.
Edit: According to your comment:
Can you please provide tests for each solution (1.denormalization, 2.my solution) examine use of bandwidth and resources and which one is really preferred?
All data is modeled to allow the use-cases that an app requires. Unfortunately, I cannot do tests because it really depends on the use-case of the app and the amount of data that it contains. This means that what works for one app, may be insufficient for another app. So the tests might not be correct for everyone. The denormalization process or your solution is entirely dependent on how you intend to query the database. In the list above, I have added a new resource which is an answer of mine regarding the denormalization tehnique in NoSQL databases. Hope it will also help feature visitors.
I would make a root node with the name, for example, MaintenanceUpdate.
All clients are subscribed to changes here.
As soon as MaintenanceUpdate becomes = true, all clients unsubscribe from changes to the main "database". And then (when MaintenanceUpdate = false) are re-subscribed again.
At this time you are updating the database.
I have similar requirements, with Firebase and Room, while I've solved it alike this:
public class BaseModel extends BaseObservable implements IDataModel {
/** Sqlite default PK */
private int itemId = 0;
/** Firebase uniqueId */
#ColumnInfo(name = SqliteBaseHelper.KEY_FIREBASE_UNIQUE_ID)
protected String uniqueId = null;
/** Firebase lastSync */
#ColumnInfo(name = SqliteBaseHelper.KEY_FIREBASE_LAST_SYNC)
protected long lastSync = 0;
...
}
this means, when a local record has a KEY_FIREBASE_UNIQUE_ID which is null and the KEY_FIREBASE_LAST_SYNC is 0, it has to be inserted into Firebase - else it would check, when running a synchronization AsyncTask, if the local or remote record needs to be updated. this is because the main issue is, that when inserting remotely, the ChildEventListener will attempt to synchronize duplicates to the same client - unless having such indicators for the synchronization status in place, locally and remotely. the local primary keys might vary across the clients (depending for how long they were offline and how many records where locally inserted during the offline state), while the synthetic KEY_FIREBASE_UNIQUE_ID is used for identifying them; it's the "key to success".
So I am wondering about how necessary it is to store variables in the realtime firebase database if I want all Users at access the same dynamic variable.
So for instance, I have a arraylist that stores the list of open games, and if I want this list to update in realtime for every user should this List in firebase realtime database?
Sorry for the simplicity of the question
Yes, it may be a simple question, but it surely pops in head of everyone, once.
I think for updating any list dynamically in real time, would require you to access any kind of database.
It is not necessary to have it on Firebase database, but any database online, that can tell every open instance of your app that the list has to be updated at a particular instant.
The main reason of why you need it to be on database is updating it in real time and that too dynamically.
If it's not dynamic, meaning the content that you need, can be hardcoded then one way would be placing everything you need in your code and using timer or something like that to fire at particular moments to update things in your app.
Also that aside, sorting, storing and changing data is much simpler using a database, which also becomes one more reason for you to use a database like Firebase to keep content of your app that has to be updated frequently in real time.
You can know more about database in this Google link, I found.
I have enabled the Firebase Persistance in my application. If I am setting a value to a child such as
child.setValue("XYZ");
I am not adding value to the parent tree. I am just updating the value of one child. So here, the value will be updated again and again by the user as he uses the application like many times a day. So, my question is, if user do not have inter-net connection for days, will this thing generate bug as Firebase is storing these things in cache. Does all the data get stored offline with mechanism something like commits in git or just the latest value is stored. I am asking this thing because it's kind of cache so if firebase stores data with all the logs and values the child gets then it can make my application buggy and slow as it will carry all the cache all the time.
If you are getting offline and you are updating a single record than, when your getting back online, only your last update will be updated on the server. Let's take an example. You have a product in which you store a timestamp. Every time you make an update, you change that timestamp with the current timestamp. If you are offline and you edit that product several times, when you'll be back online, only the last timpstamp will be added on the server.
But remember, this not happening when you add new data. When you do this, all the new data is added on the server accordingly to time you have added. This happening also when you delete.
Hope it helps.
I am java developer and my application is in iOS and android.I have created web service for that and it is in restlet Framework as JDBC as DB connectivity.
My problem is i have three types of data it is called intersection like current + Past + Future.and this intersection contain list of user as a data.There is single web service for giving all users to device as his/her intersection.I have implement pagination but server has to process all of his/her intersections and out of this giving (start-End) data to device.I did this because there are chances that past user may also come in current.This the total logic.
But as intersection grows in his/her profile server has to process all user.so it become slow and this is obvious.also device call this web service in every 5 minutes.
please provide better suggestion to handle this scenario.
Thanks in advance.
Ketul Rathod
It's a little hard to follow your logic, but it sounds like you can probably benefit from caching your results on the server.
If it makes sense, after every time you process the users data on the server, save the results (to a file, to a database table, whatever). Then, in 5min, if there are no changes, simply return the same. If there were changes, retrieve from cache (optionally invalidating the cache in the process), append those changes to what is cached, re-save the results in the cache, and return the results.
If this is applicable to your workflow, your server-side processing time will be significantly less.
Till now, I have coded apps which load listviews completely from web(parsing,etc) or completely from local database. What I intend to develop is a listview which will load from existing local database, and check the web database and fetch only those entries which do not exist in the local cache, and then update the local cache with those new entries.
I had a naive idea of implementing it. There would be a single value in local data about the number of entries in local database. Similarly, there would be a value of number of entries on the web database. Then we can exactly fetch the required number of entries from the web, instead of fetching the entire list again.
Is there some better/easier way to do it?
I found a solution to this. The better thing to do would be to do this checking on a middle layer server. It would be like a 3 layer application. There would be a middle layer which would store all the posts seen by the current user. So next time user wants to update himself, the middle layer can check what results were posted back to him last time