I am creating an app that logs data. I am creating documents that have the data and sending those documents to a couchbase server. Or I am trying to anyways. One major concern I have is how do I confirm a document is stored on the server so that it can be immediately deleted on the device? I am hoping there is a quick and efficient way to do this. The end result is to have a thread constantly checking if there is a connection to couchbase, and if so start sending data up to clear it off the device. Most documentation seems to be regarding syncying the database, however I don't want to do this because I don't want to keep a copy of the data on the device. It would take up too much storage. Thanks for any help.
EDIT: For clarification, I currently have the app storing many data points in documents. I want to send these documents to a couchbase server. I don't want to "sync" the documents, but rather just insert them into the database then immediately delete them off the device. How would one go about doing this? Most examples I have seen typically sync documents such as profile information where changes can be made in various synced databases and all those changes would appear in every database. Instead I want a 1 way relationship with the database were information is sent, confirmed as received, then immediately deleted from the device.
There are at least a few possibilities.
If you are expecting a solid network connection, or are ok with handling errors yourself, you can achieve this with a direct REST call to Sync Gateway. You can, of course, always write your own REST server that talks directly to Couchbase Server, too.
The second way relies on an older version of Couchbase Lite. Couchbase Lite 2.x is a major rewrite of the product. As of the current shipping version (2.1), it does not support this approach, so you'll need to use the 1.x version (1.3 or later, IIRC). See further down for how to approach this with 2.1.
Set up a push only replication. After replication, cycle through the docs and purge all the ones that are not still pending. (This uses the isDocumentPending method on the Replication class. That's the key piece not available as of 2.1.) You can either run one shot replications and do this after the replication completes, or monitor the replication state of a continuous replication.
Purging a document from the local CB Lite database effectively makes it act as if it never existed on that device. By running a push only replication, you don't have to worry about the docs getting sent back to the device.
Using 2.1, you can't as easily determine if a document has been replicated. So you need to run a replication to completion while avoiding a race condition with writing something new.
One approach here is to pause writing documents, run a one shot replication, then purge the documents before starting up again. You could also work out something with alternating databases, or tracking documents yourself somehow, etc.
For completeness, if you were in a situation where you had a mixed use, that is, wanted only some documents pushed up off the device and forgotten, and some synced, you would control this through Sync Gateway channels.
I don't know Lite and Sync Gateway well enough, but from a Server perspective:
You could use the new Eventing service in Couchbase. When a document is created in bucket A, you could write an event to copy that document over to bucket B. Then, if the documents are deleted on the device, it wouldn't matter if they get deleted from bucket A.
I have a bucket "staging" and a bucket "final". I created a function called "moveIt" with "final" (I aliased as 'f').
The OnUpdate function could be as simple as:
function OnUpdate(doc, meta) {
f[meta.id] = doc;
}
My main concern would be the timing. I don't think there's an easy way for your mobile app to know that the event has finished copying a document before you decide to delete it in Lite and start a Sync. But it might be worth a try. Check out the docs to learn more about the Eventing service details.
In Couchbase Lite 2.5, you can use replicated events to detect when a document has synced (pushed to server or pulled from server). You can register a callback on the Couchbase Lite replicator to detect if the documents have been pushed to the sync gateway and then use the purge API to locally purge
Related
I am developing an app that uses a database to store the data on the server, but I am trying to save some of the data locally, and in the event of no internet connection being established, save new data locally to the device and synchronize any changes to the server when an connection is re-established. What is the best and most efficient way to do this?
I have been looking at Androids Room persistence library and it seems like the logical choice, but I am not sure how it goes about synchronizing changes to/from the local storage database. I have looked at multiple threads and forums for help, but have had no luck so far. Please help.
One way is to build your own sync adapter: https://developer.android.com/training/sync-adapters
You will need to handle most of the sync logic between the client and the server, but that allows you to use any database technology in the server. From the docs:
A sync adapter doesn't automate any data transfer tasks. If you want to download data from a server and store it in a content provider, you have to provide the code that requests the data, downloads it, and inserts it in the provider. Similarly, if you want to send data to a server, you have to read it from a file, database, or provider, and send the necessary upload request. You also have to handle network errors that occur while your data transfer is running.
A sync adapter doesn't automatically handle conflicts between data on the server and data on the device. Also, it doesn't automatically detect if the data on the server is newer than the data on the device, or vice versa. Instead, you have to provide your own algorithms for handling this situation.
Use firestore and enable offline data persistence. https://firebase.google.com/docs/firestore/manage-data/enable-offline
I am trying to develop a mechanism for Sync data with app & server, like Google Drive/Dropbox for one of my android based application where data are stored locally on device and I would like to sync data with server.
Ways thought -
1) Observer which looks for change and call web service
2) Background service which runs in background and check for changes
3) Set Alarm which checks at particular time and sync all the data with server
I have not started to which option to go with, I would like to get experts view on this and like to get some guidance so that I can achieve the sync mechanism in best possible way for my app.
All suggestion are welcome.
Thinking on same line as you have been,the aim should be to achieve a balance between the number of times the server is queried for same set of information AND the data consistency.
for this, I would fire a GCM message from my server to the device for which the data on the server has changed, I would maintain a count for these Update Messages(No Notification genereated). If this count Exceeds the minimum-threshold-count-value, I would immediately call for an UPDATE. Or otherwise, if this count still is less than the threshold-count-value, for a certain period, that I would call the threshold-waiting-period, then too I call for an UPDATE.
The UPDATE would be using Sync Adapters and Services. This link explains its basics.
Hope this prooves to be helpful!
...so that I can achieve the sync mechanism in best possible way for my app
Well, the BEST mechanism depends completely on what kind of application, you are developing. Also, your option 1 and 2 seems very similar to me except few implementation changes. However, how frequently your device data is changing, that also matters a lot. If you have authentication mechanism, you can configure sync up, at the time, whenever you login to your app. Like in case of confidential data, sync up should happen immediately.
If data is managed well based on authentication and authorization based on user roles or so, you also need to take care of synchronization among them. For example, one user has updated an entity which is not yet sync and another user tries to update the same, then first user will see his changes are not synced or has been lost.
The best way (means having least drawbacks) to do this would be sync up trigger at a particular interval of time OR at every login time. (still as I said, depends on your app).
Hope this would clear some or all of your doubts.
I am developing an Android application where users will see lists of Groups and join 1-* Groups. Within the Groups will be members and users can message each other within the group they have joined. Sending messages 1 person to many and reply all.
I want this to support multiple devices. So, if i join a group and someone sends me a message, i want to see that message on my phone and tablet. If both have the app installed.
I want the user to be able to see group details while offline.
I have been doing some research and drawing diagrams on how this could work.
What are my options for how to set this up with a remote database?
Here are a couple of the options that i have finally narrowed down to.
Have the remote DB store everything. Have a local DB on the device that reflects the remote DB. Have the remote DB contain a version number. At some point, either scheduled or triggered, the app compares the version numbers in the local and remote and if the local is out of date it will update to the current remote DB.
Have the remote DB store everything. Have a local DB on the device that reflects everything in the remote DB except the messaging system. This would behave similar to #1. Except the tables and stuff that are related to the messaging system would "somehow, gcm maybe" send a message to all devices associated with the user that a message relating to the user has changed and the app will then update to the most recent version of the remote messaging tables in the remote database. Then their local database contains the new data and their "message box" would be updated.
What would be the best way to do this? How is this done currently in most systems? Are there more options then what i listed?
Remote DB: MySql
Local DB: sqlite
I am aware of some similar questions here on SO, but i am wanting to know if either of these options or something else would be best for my specific scenario. Mainly how to handle the messaging part.
Thanks
I think Option 1 is a pretty standard way of handling this problem.
A DB version # could work, but if it changes frequently you will end up resync all the data very often. Its probably better if you just store "Last Modified Date" on records and only pull down things that have changed since the last sync. That way you can return a smaller set of new records.
You would only want to pull down public information (the groups) and the information specific to the user. The remote DB would have everything, but the local DB would only consist of data that is related to the user.
I found this tutorial that walks through how this might be implemented using php on the remote server for API access. You could streamline this process by using an ORM or "Out of the Box" API solution.
I'm sure there are more elegant solutions out there for syncing a local Android DB with a remote but this is likely the most practical approach for the non-enterprise solutions.
I want to create java desktop application, which stores it's data offline in a database (not just some config files). The application should work fine when the user is offline. When the user becomes online, the offline database should be able to sync with the online master.
Any ideas which technologies can be used to achieve this?
This has been discussed on stackoverflow a lot and it usually boils down to: don't roll out your own solution - This is a very specialized field - Look up SymmetricDS. It does what you want.
One of my fav discussions is Strategy for Offline/Online data synchronization
Use one of the available pure java DB implementations as a local DB. Use any other DB as a remote one.
Implement logic that tries to connect to remote DB and fall backs to local one on failure. If it connects successfully to remote DB implement the data synchronization.
When the local application operates, it should not only change database, but also log the changes. That changes are sent to the server when the connection is available. Also, the application receives logged changes stored on the server (from other application instances).
The main problem is how to merge changes made by different instances. There can be 3 variants:
1) Each application instance can modify only its private part of the whole database. Your are lucky, no merging needed, and server can store only logs and not run the whole database.
2) modifications always can be merged automatically (for example, application can add a value to a common variable, but cannot set it directly). The server runs the whole database, accepts partial logs from clients, generates its own log and sends it to clients.
3) Clients are allowed to do arbitrary modifications. This leads to potential conflicts. In case of conflicts, one of conflicting changes should be rejected. That means, that if a client made local modifications, that modifications can be rejected later by the server. The user interface must reflect this issue. In the rest, this is similar to the variant 2.
Architecture:
A bunch of clients send out messages to a server which is behind a VIP. Obviously this server poses an availability risk.
The client monitors a resource and the server is responsible to take action based on the what status the majority of the clients report to it and hence the need for only 1 server/leader.
I am thinking of adding another server as a backup on the VIP, which gets turned on only when the first server fails. However when the backup comes up it would have no information to process and would lose time waiting for clients to report and waiting for the required thresholds etc.
Problem:
What is the best and easiest way to have two servers share client state information with only one receiving client traffic?
Solution1:
I thought of have having the server forward client state information to backup server and in the event of a failure when the backup server comes up, it can take it from there.
Is there any other way to do this? I thought of having a common/shared place to store state information where both servers can read client state information from. But this doesn't work well as the shared space is a single point of failure too.
One option is to use a write-ahead log. Essentially, any modification you make to your state gets sent over to the backup server, which replays the change on its own copy of the state. As long as it can keep up with the streaming log, the backup is always up-to-date.
This is the approach generally used by most databases; if you use one as your backend, you may be able to get support for this with little work.
Be careful to have a plan to recover from communication failure - either save the log to disk and resend the missing portion, or send a snapshot of the state, plus all log entries since the snapshot on reconnect.
There are various distributed caching products which do the kind of thing you're talking about here. Some are supplied with App Servers, such as WebSphere's dynacache and Object Grid. In fact ObjectGrid can be used in JSE, no need for an App Server.
Those distributed cache products use various push and pull models with pub-sub messaging to achieve consistency across the instances. Working for IBM I'm a fan of ObjectGrid, but more impartant, I'm fan of not reinventing wheels. My take is that this stuff can get quite complex and hence finding something off-the shelf might save a load of work - there are links to various Open Source solutions here.
The is very much dependent on how available your solution needs to be (how many 9's). There is a spectrum of solution.
A lightweight one could be crafted around Memcache: extremely fast distributed state facility. As example, it is used extensively on Google AppEngine.