Really hoping someone may be able to point me in the right direction for this one. We have an app that uses Realm to manage our data locally on the device.
Through general use there will be data inputted to the device and then this will be attempted to be sent to the live service we have on our api. However due to the location of the users we cannot be sure if this has been uploaded before we re-download the data the next time the app is opened.
We need it to work just like Git in a way. You can't pull without first committing your changes. But instead of committing we are pushing up.
We believe the live data should take priority here so if there is a change on live it should get pulled down, but if there is something null on live but we have it locally then we should change that.
Is there something we can use to code this functionality, can't believe we are the first to face this issue. Either that or a flow for how we should get the data before the user can start working.
An example:
We are looking to save the imei number that the user enters. This will save to the realm db at the time and attempt to update the live database. If this is not possible when the app is restarted and the imei numbers are pulled we don't what that to be possible until the data we have locally that is missing on live to be pushed.
Imei.java (Realm model)
public class Imei extends RealmObject {
#PrimaryKey
public int id;
public int deviceId;
public String imei;
}
When we get a response on the fetch:
#Override
public void onResponse(JSONObject response) {
Gson gson = new GsonBuilder()
.setFieldNamingPolicy(FieldNamingPolicy.LOWER_CASE_WITH_UNDERSCORES)
.create();
ImeiResponse imeiResponse = gson.fromJson(response.toString(), ImeiResponse.class);
for (Imei imei : imeiResponse.imeis()) {
addToDB(imei);
}
}
And then we use addToDb, which is where I would imagine that some kind of check would take place as if we were able to pull then we have a good enough connection to push the data we have locally
public void addToDB(Imei imei)
{
Realm realm = Realm.getDefaultInstance();
realm.beginTransaction();
realm.copyToRealmOrUpdate(imei);
realm.commitTransaction();
}
Just in case someone stumbles across this question we did eventually find a solution.
Any time we need to save data we are adding it into a request queue table and then attempting to run each of the requests. If they pass they are removed, if they fail they are deleted.
Every time we then go to pull data we check that requests queue first and run if there is anything in there, if they continue to fail you then don't pull.
This seemed the best solution for us. Hope it helps
Related
I am trying to do live streaming example app, where I can live update the list in the browser. I want to return all elements and then still listening (don't stop the stream) when new item is add to the database. Then I want to show new item in the browser. My current solution all the time prints all items (second by second) but I think there is better solution, when I can a) find the difference in list from last processing repository.findAll() and return only currList - prevList b) I can listen to some kind of events? Like inserting to table and add new item to still opened stream.
Here is my current code:
#RestController
#RequestMapping("/songs")
public class SongController {
private final SongRepository songRepository;
public SongController(SongRepository songRepository) {
this.songRepository = songRepository;
}
#GetMapping(produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<Song> getAllSongs() {
return Flux.interval(Duration.ofSeconds(1))
.flatMap(x -> songRepository.findAll());
}
#PostMapping
public Mono<Song> addSong(#RequestBody Song song) {
return songRepository.save(song);
}
}
Here is how it looks like now:
As you can see, Its obviously looped, and I just need plain list with 7 elements on begining and then +1 element every time I post new song (by addSong()).
I don't need a entire ready solution, I just don't know what should I use.
Thank you in advance, cheers
In my experience there are three options that have different pros and cons.
You could create a web socket connection from the browser to your backend service. This will create a bi-directional connection that will allow you push updates from the server to your browser. In this instance whenever you add a song you would then write that song to the web socket connection and handle that on the browser side, so adding it to the list in the browser.
The cons of this are in my experience web socket connections are finicky and aren't the most stable or reliable.
You could use server side events. I haven't used this personally but I have heard this can be a viable options for pushing events from the server to the browser. https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
You could poll the endpoint. I know this approach gets a lot of hate in recent years but it is a viable options. The benefit with polling the endpoint is that it is resilient to failures. If your backend is overloaded and can't respond for one request it will likely be able to respond to a subsequent request. Also there are ways of improving commonly used endpoints so you're not hammering your database like a cache or something of that nature.
TL;DR
Every time my Fiestore admin server reboots my document listener is triggered for all documents even if I have already listened to the document and processed it. How do I get around this?
End TL;DR
I'm working on building a backend for my Firestore chat application. The basic idea is that whenever a users enters a chat message through a client app the backend server listens for new messages and processes them.
The problem I'm running into is that whenever I reboot my app server the listener is triggered for all of the existing already processed chats. So, it will respond to each chat even though it has already responded previously. I would like the app server to only respond to new chats that it hasn't already responded to.
One idea I have for a work around is to put a boolean flag on each chat document. When the backend processes the chat document it will set the flag. The listener will then only reply to chats that don't have the flag set.
Is this a sound approach or is there a better method? One concern I have is that every time I reboot my app server I will be charged heavily to re-query all of the previous chats. Another concern I have is that listening seems memory bound? If my app scales massively will I have to store all chat documents in memory? That doesn't seem like it will scale well...
//Example listener that processes chats based on whether or not the "hasBeenRepliedTo" flag is set
public void startFirestoreListener() {
CollectionReference docRef = db.collection("chats");
docRef.addSnapshotListener(new EventListener<QuerySnapshot>() {
#Override
public void onEvent(#javax.annotation.Nullable QuerySnapshot queryDocumentSnapshots, #javax.annotation.Nullable FirestoreException e) {
if(e != null) {
logger.error("There was an error listening to changes in the firestore chats collection. E: "+e.getLocalizedMessage());
e.printStackTrace();
}
else if(queryDocumentSnapshots != null && !queryDocumentSnapshots.isEmpty()) {
for(ChatDocument chatDoc : queryDocumentSnapshots.toObjects(ChatDocument.class)) {
if(!chatDoc.getHasBeenRepliedTo() {
//Do some processing
chatDoc.setHasBeenRepliedTo(true); //Set replied to flag
}
else {
//No-op, we've already replied to this chat
}
}
}
}
});
}
Yes, to avoid getting each document all the time, you will have to construct a query that yields only the documents that you know have been processed.
No, you are not charged to query documents. You are charged only to read them, which will happen if your query yields documents.
Yes, you will have to be able to hold all the results of a query in memory.
Your problem will be much easier to solve if you use Cloud Functions to receive events for each new document in a collection. You won't have to worry about any of the above things, and instead just worry about writing a Firestore trigger that does what you want with each new document, and paying for those invocations.
After deleting data from my Firestore Database, it takes my Android app some time to realize that the data was deleted, and I assume that it's happening due the auto data cache. My app has nothing to do with offline usage and I'd like to disable this feature...
I have added this in my custom Application Class:
import android.app.Application;
import com.google.firebase.firestore.FirebaseFirestore;
import com.google.firebase.firestore.FirebaseFirestoreSettings;
public class ApplicationClass extends Application {
#Override
public void onCreate() {
super.onCreate();
FirebaseFirestore db=FirebaseFirestore.getInstance();
FirebaseFirestoreSettings settings = new FirebaseFirestoreSettings.Builder()
.setPersistenceEnabled(false)
.build();
db.setFirestoreSettings(settings);
}
}
The problem occurs after turning off the internet connection and than turning it back on (while the app is still running, in the background or not)- the Firestore module seems to lose connection to the server, and it makes the opposite operation than the intended one - instead of stop taking data from the cache, it takes data from the cache only.
For example, debugging this code will always show that isFromCache is true and documentSnapshot is empty (even though that on the server side - it's not empty):
usersRef.document(loggedEmail).collection("challenges_received").get().addOnSuccessListener(new OnSuccessListener<QuerySnapshot>() {
#Override
public void onSuccess(QuerySnapshot documentSnapshots) {
boolean isFromCache=documentSnapshots.getMetadata().isFromCache();
if (!documentSnapshots.isEmpty()) {
}
}
});
Is this normal behavior?
Is there another way to disable the data cache in Cloud Firestore?
EDIT:
Adding: FirebaseFirestore.setLoggingEnabled(flase); (instead of the code above) in the custom Application Class gives the same result.
According to Cloud Firestore 16.0.0 SDK update, there is now a solution to this problem:
You are now able to choose if you would like to fetch your data from the server only, or from the cache only, like this (an example for server only):
DocumentReference documentReference= FirebaseFirestore.getInstance().document("example");
documentReference.get(Source.SERVER).addOnSuccessListener(new OnSuccessListener<DocumentSnapshot>() {
#Override
public void onSuccess(DocumentSnapshot documentSnapshot) {
//...
}
});
For cache only, just change the code above to Source.CACHE.
By default, both methods still attempt server and fall back to the cache.
I just ran a few tests in an Android application to see how this works. Because Firestore is currently still in beta release and the product might suffer changes any time, i cannot guarantee that this behaviour will still hold in the future.
db.collection("tests").document("fOpCiqmUjAzjnZimjd5c").get().addOnCompleteListener(new OnCompleteListener<DocumentSnapshot>() {
#Override
public void onComplete(#NonNull Task<DocumentSnapshot> task) {
DocumentSnapshot documentSnapshot = task.getResult();
System.out.println("isFromCache: " + documentSnapshot.getMetadata().isFromCache());
}
});
Regarding the code, is the same no matter if we're getting the data from the cache or you are connected to the servers.
When I'm online it prints:
isFromCache: false
When I'm offline, it prints:
isFromCache: true
So, for the moment, there is no way to stop the retrieval of the data from the cache while you are not connected to the server, as you cannot force the retrieval of the data from the cache while you're connected to the server.
If instead I use a listener:
db.collection("tests").document("fOpCiqmUjAzjnZimjd5c").addSnapshotListener(new DocumentListenOptions().includeMetadataChanges(), new EventListener<DocumentSnapshot>() {
#Override
public void onEvent(DocumentSnapshot documentSnapshot, FirebaseFirestoreException e) {
System.out.println("listener.isFromCache: " + documentSnapshot.getMetadata().isFromCache());
}
});
I get two prints when I'm online:
listener.isFromCache: true
listener.isFromCache: false
Firestore is desinged to retrieve data from the chache when the device is permanently offline or while your application temporarily loses its network connection and for the moment you cannot change this behaviour.
As a concusion, an API that does something like this, currently doesn't exist yet.
Edit: Unlike in Firebase, where to enable the offline persistence you need use this line of code:
FirebaseDatabase.getInstance().setPersistenceEnabled(true);
In Firestore, for Android and iOS, offline persistence is enabled by default.
Using the above line of code, means that you tell Firebase to create a local (internal) copy of your database so that your app can work even if it temporarily loses its network connection.
In Firestore we find the opposite, to disable persistence, we need to set the PersistenceEnabled option to false. This means that you tell Firestore not to create a local copy of your database on user device, which in term means that you'll not be able to query your database unless your are connected to Firebase servers. So without having a local copy of your database and if beeing disconected, an Exception will be thrown. That's why is a good practice to use the OnFailureListener.
Update (2018-06-13): As also #TalBarda mentioned in his answer this is now possible starting with the 16.0.0 SDK version update. So we can achieve this with the help of the DocumentReference.get(Source source) and Query.get(Source source) methods.
By default, get() attempts to provide up-to-date data when possible by waiting for data from the server, but it may return cached data or fail if you are offline and the server cannot be reached. This behavior can be altered via the Source parameter.
So we can now pass as an argument to the DocumentReference or to the Query the source so we can force the retrieval of data from the server only, chache only or attempt server and fall back to the cache.
So something like this is now possible:
FirebaseFirestore db = FirebaseFirestore.getInstance();
DocumentReference docIdRef = db.collection("tests").document("fOpCiqmUjAzjnZimjd5c");
docIdRef.get(Source.SERVER).addOnSuccessListener(new OnSuccessListener<DocumentSnapshot>() {
#Override
public void onSuccess(DocumentSnapshot documentSnapshot) {
//Get data from the documentSnapshot object
}
});
In this case, we force the data to be retrieved from the server only. If you want to force the data to be retrieved from the cache only, you should pass as an argument to the get() method, Source.CACHE. More informations here.
FirebaseFirestoreSettings settings = new FirebaseFirestoreSettings.Builder()
.setPersistenceEnabled(false)
.build();
dbEventHome.setFirestoreSettings(settings);
By setting this it is fetching from server always.
In Kotlin:
val db:FirebaseFirestore = Firebase.firestore
val settings = firestoreSettings {
isPersistenceEnabled = false
}
db.firestoreSettings = settings
// Enable Firestore logging
FirebaseFirestore.setLoggingEnabled(flase);
// Firestore
mFirestore = FirebaseFirestore.getInstance();
In general: the Firebase client tries to minimize the number of times it downloads data. But it also tries to minimize the amount of memory/disk space it uses.
The exact behavior depends on many things, such as whether the another listener has remained active on that location and whether you're using disk persistence. If you have two listeners for the same (or overlapping) data, updates will only be downloaded once. But if you remove the last listener for a location, the data for that location is removed from the (memory and/or disk) cache.
Without seeing a complete piece of code, it's hard to tell what will happen in your case.
Alternatively: you can check for yourself by enabling Firebase's logging [Firebase setLoggingEnabled:YES];
try this For FireBase DataBase
mDatabase.getReference().keepSynced(false);
FirebaseDatabase.getInstance().setPersistenceEnabled(false);
In Kotlin;
val settings = FirebaseFirestoreSettings.Builder()
with(settings){
isPersistenceEnabled = false
}
Firebase.firestore.firestoreSettings = settings.build()
Now, I save data in datastore of google cloud.
module-backend
#Entity
public class ComprarProducto {
#Id
private String producto;
private boolean comprado;
public ComprarProducto() {
}
public ComprarProducto(String producto) {
this.producto = producto;
this.comprado = false;
}
...
}
Example data of CompraProducto
This data is displayed in a listview. When I add one item, the list is updated and saved in datastore.For save data I use AsyncTask in app.
Then, If I add one item in device1, I want to see that listview as both device1 as device2 is updated, but I don't know how to do it.
I use too GCM (google cloud messaging) and I've got to send a message to devices but only this. I want to see the data update in the device 2 without taking any action.
I want to see the data update in the device 2 without taking any
action.
This is not possible without somehow notifying from the backend server that new data has been saved to datastore. Sending a push notification to device2 (or all other devices or whatever) would definitely be the correct way to do it. What you do is have the push notification tell device2 to query datastore.
If you really don't want to use GCM you COULD have device2 just query datastore every couple minutes but that is not recommended. The correct way to do it is to use GCM to give your device2 a "tickle" to indicate that there is new data in datastore and to query accordingly.
Not sure what kind of app you are building but these are some options.
I am working on a Spring-MVC based web-app which uses Cometd for chat purposes. For real-time management of which user is online, we are sending notifications when the user is online. So when window is closed, then notifications don't appear and after 30 seconds it is determined that the user is offline or not reachable.
Now the problem happens when user is over multiple browsers. Lets just keep it for 2 now. So, after 10 minutes we are setting user status to 'Away from Keyboard'(AFK). But if the user is online in one browser, then we are getting a blinking status, for few seconds because of the browser in 'Idle Mode', we get a AFK, and from the active machine we get an 'Available' status.
How can we solve this problem? Any ideas, suggestions. I thought of using a boolean flag, and couple with IP address, which will be checked before overwriting the notification, but it has a problem of stale notifications.
This is my code for sending out notifications for online to all listeners(Friends of user).
Code :
#Listener(value = "/service/online")
public void OnlineNotifications(ServerSession remote, ServerMessage.Mutable message) {
Person sender = this.personService.getCurrentlyAuthenticatedUser();
Map<String, Object> input = message.getDataAsMap();
String onlineStatus = (String) input.get("status");
Map<String, Object> output = new HashMap<>();
output.put("status", onlineStatus);
output.put("id", sender.getId());
ServerChannel serverChannel = bayeux.createChannelIfAbsent("/online/" + sender.getId()).getReference();
serverChannel.setPersistent(true);
serverChannel.publish(serverSession, output);
}
Any ideas are welcome. Thanks a lot. :-)
Your application can easily associate multiple devices with the same userName.
You can do this from your SecurityPolicy, just associate the userName string with a List<ServerSession> or whatever data structure fits better your case.
Your online state for a particular userName is therefore a function of all the ServerSessions of this particular userName.
As such, it's a mapping from the userName to a list of online states.
If you imagine user bob logged in from two browsers, the mapping for its online status can be:
"bob" -> ["afk", "online"]
To compute the online status for bob you just run through all the online statuses for each ServerSession and if there is at least one online then it's online, if all are afk then it's away from the keyboard, and so on.
You can implement this logic in several ways (e.g. storing the online status as a ServerSession attribute, use a different data structure, cache the per-user online status, etc.), but you get the idea.
Whenever you get an online status change from one ServerSession, you update your server-side data structure, and then recompute the online status from all ServerSessions for that user, and that is what you send to the other users.