How to synchronize data with a remote database in case of stand-alone applications?
- I am creating a "TODO" application that uses Service Workers to cache the responses to requests, and in case the user is disconnected, the cached data is displayed to the user.
- The server provides a REST-end endpoint that has POST, PUT, DELETE, and GET endpoints exposed for resources.
- Considering that when a user is offline and submits a TODO item, I save it in a local indexed format, but I cannot send this POST request to the server as there is no network connection. The same is true for PUT, DELETE requests when the user updates or deletes an existing TODO item.
Questions
- What patterns are used to synchronize pending requests with a REST-ful server on reconnection?
source to share
At the moment I can think of two approaches and it depends on what storage options you are using on your server.
If you are using a DBMS to back up all data:
The problem with autonomous systems in this approach is the potential for conflict that you might run into when publishing new data or updating existing data.
As a first measure to avoid conflicts, you will need to create unique identifiers for all of your clients' objects so that they remain unique when hosted on the server and persist in the database. For this, you can reliably rely on UUIDs to create unique identifiers for objects. UUID guarantees the uniqueness of the system in a distributed system and depending on which implementation language you will have methods for generating UUIDs without any problems.
Create your local database so you can use UUID as primary key in your local database. On the server end, you can use both types: an auto-appending and indexing integer type, a primary key, and a VARCHAR type for storing UUIDs. The primary key on the server uniquely identifies the objects in that table, and the UUID uniquely identifies records between tables and databases.
Therefore, when sending your object to the server during synchronization, you just need to check if any object with a UDID is present and take appropriate action. When you fetch objects from the server, send both the primary key of the object from your table and the UDID for the objects. So when you serialize the response in model objects, or store them in your local database, you can specify the objects that were synchronized with the ones that don't, since the objects that need to be synchronized won't have a primary key in your local database , just a UUID.
It may happen that your server is down and refuses to save data during synchronization. In this case, you can store an integer variable in your objects that will contain a count of the number of synchronization attempts. If this number exceeds a certain value, say 3, you move on to sync the next object. Now, what you do with non-sync objects is your policy you have for objects like a solution that you can drop or only store them locally.
If you are not using an RDBMS
As an alternative approach, instead of storing all objects, you can store the transactions that each client performs locally on the server. Each client synchronizes transactions, and when fetching, you get the current state, working with all transactions from the bottom up. This is very similar to what Git uses. It saves changes to your repository as transactions, such as what was added (or removed) and by whom. The current state of the repository for each user is processed from transactions. This approach will not lead to conflicts, but as you can see, it is a little difficult to develop.
source to share
What patterns are used to synchronize pending requests with a REST-ful server on reconnection?
The Background Sync API would be appropriate for this scenario. This allows web applications to sync data in the background. In doing so, it can defer actions until the user has a reliable connection, ensuring that whatever the user wants to send is actually sent. Even if the user moves or closes the browser, the action is taken and you can notify the user if you want.
Since you are saving to IndexDB, you can register for a sync event when a user adds, removes, or updates a TODO item
function addTodo(todo) {
return addToIndeDB(todo).then(() => {
// Wait for the scoped service worker registration to get a
// service worker with an active state
return navigator.serviceWorker.ready;
}).then(reg => {
return reg.sync.register('add-todo');
}).then(() => {
console.log('Sync registered!');
}).catch(() => {
console.log('Sync registration failed :(');
});
}
You have registered a type sync event add-todo
that you will listen to in the service worker, and then when you receive that event, you will receive data from IndexDB and POST to your Restful API.
self.addEventListener('sync', event => {
if (event.tag == 'add-todo') {
event.waitUntil(
getTodo().then(todos => {
// Post the messages to the server
return fetch('/add', {
method: 'POST',
body: JSON.stringify(todos),
headers: { 'Content-Type': 'application/json' }
}).then(() => {
// Success!
});
})
})
);
}
});
This is just an example of how you could achieve this using background synchronism. Note that you have to handle conflict resolution on the server.
You can use PouchDB on the client and Couchbase or CouchDB on the server. With PouchDB on the client, you can persist data on the client and set it to automatically sync / replicate data whenever the user is online. When the database is in sync and conflicting changes occur, CouchDB will detect this and mark the affected document with a special attribute "_conflicts":true
. It determines which one it will use as the latest version, and save the rest as the previous revision of that entry. It is not trying to merge a conflicting revision. It's up to you how the merge should be done in your application. It's not that much different from Couchbase. See the links below for more information on conflict resolution.
- Managing conflicts with CouchDB
- Understanding CouchDB Conflict
- Couchbase Conflict Resolution
- Demystifying Conflict Resolution in Couchbase Mobile
I used pouchDB and couchbase / couchdb / IBM, but I did it through Hoodie It has external user authentication - box, handles conflict management and a few more. Think of it as a backend. In your TODO app, Hoodie will fit perfectly. I wrote something about how to use Hoodie, see links below:
source to share