Simple Offline Data Synchronization for Mobile Web and PhoneGap Applications

Being able to work offline is an expected feature of mobile applications. For data-driven applications, it means that you — the developer — will have to store (a subset of) your application data locally, and implement a data synchronization mechanism that keeps your local and server data in sync.

In this article, I describe a simple data synchronization strategy that uses the device’s (or browser’s) SQLite database. The implementation currently leverages the Web SQL API (even though the W3C is no longer actively maintaining the spec) because both iOS and Android support it, but they don’t support IndexedDB, the official alternative. However, the API described below — getLastSync(), getChanges(), applyChanges() — defines a generic synchronization contract, and the solution can be expanded and made “pluggable”: You could create different synchronization objects, each providing a different implementation of these methods. You could then choose which object to plug in based on the context and the platform your application is running on.

Try it in the Playground

Before looking at the code, you can try some offline syncing in this a hosted playground:

1. Open the Offline Client Playground in Chrome or Safari (they both support Web SQL).
2. Click the Synchronize button.
3. Look at the log (the textarea in the middle of the screen): Because it’s the first time you use the application, all the employees have been downloaded from the server and inserted in your local SQLite database.
4. Clear the log, click the Synchronize button, and look at the log again: because you now have an up-to-date local version of the data, the server didn’t return any change and your local database remains unchanged.
5. In another tab, open the Server Admin PlayGround.
6. Modify an existing employee and click Save. (Don’t worry, it’s using your own session-based data set).
7. Go back to the Offline Client tab, click Synchronize, and notice that the server returned one change, and that it was applied to your local database.
8. Go back to the Server Admin tab and modify (create, update, delete) other employees. Switch back to the Offline Client tab, click Synchronize, and see how these changes are applied to your local database.
9. You can also use the Resources Tab in the Chrome Developer Tools to inspect your local database.

Server API

The only piece of infrastructure you need at the server side is an API that returns the items that have changed (created, updated, or deleted) since a specific moment in time expressed as a timestamp.

Here is the RESTful API call used in my application:

http://coenraets.org/offline-sync/api/employees?modifiedSince=2012-03-01 10:20:56

The format of the data returned by the server is up to you and is part of the contract between the client and the server. In this application, the server returns the changes as an array of JSON objects. The server-side technology (RoR, PHP, Java, .NET, …) and database system (SQL, NoSQL, …) you use to generate the list of changes is also totally up to you. I provide a simple PHP implementation as part of the source code. That implementation manages a session-based data set that provides an isolated and transient playground. In a real-life application, you’d obviously get the data from some sort of database.

Client API

At the client side, our synchronization API consists of three methods.

getLastSync()

A method that returns a timestamp to be used as the query parameter for the next synchronization request. A common practice is to persist a timestamp after each synchronization request. But things can go wrong and the timestamp itself can get out-of-sync. I prefer to “recalculate” the lastSync timestamp before each synchronization request.

getLastSync: function(callback) {
    this.db.transaction(
        function(tx) {
            var sql = "SELECT MAX(lastModified) as lastSync FROM employee";
            tx.executeSql(sql, this.txErrorHandler,
                function(tx, results) {
                    var lastSync = results.rows.item(0).lastSync;
                    callback(lastSync);
                }
            );
        }
    );
}

getChanges()

This is a wrapper around an Ajax call to the server-side API that returns the items that have changed (created, updated, or deleted) since a specific moment in time defined in the modifiedSince parameter.

getChanges: function(syncURL, modifiedSince, callback) {

    $.ajax({
        url: syncURL,
        data: {modifiedSince: modifiedSince},
        dataType:"json",
        success:function (changes) {
            callback(changes);
        },
        error: function(model, response) {
            alert(response.responseText);
        }
    });

}

applyChanges()

A method that persists the changes in your local data store. Notice that SQLite supports a convenient “INSERT OR REPLACE” statement so that you don’t have to determine if you are dealing with a new or existing employee before persisting it.

applyChanges: function(employees, callback) {
    this.db.transaction(
        function(tx) {
            var l = employees.length;
            var sql =
                "INSERT OR REPLACE INTO employee (id, firstName, lastName, title, officePhone, deleted, lastModified) " +
                "VALUES (?, ?, ?, ?, ?, ?, ?)";
            var e;
            for (var i = 0; i < l; i++) {
                e = employees[i];
                var params = [e.id, e.firstName, e.lastName, e.title, e.officePhone, e.deleted, e.lastModified];
                tx.executeSql(sql, params);
            }
        },
        this.txErrorHandler,
        function(tx) {
            callback();
        }
    );
}

Synchronization Logic

With these server and client APIs in place, you can choreograph a data synchronization process as follows:

sync: function(syncURL, callback) {

    var self = this;
    this.getLastSync(function(lastSync){
        self.getChanges(syncURL, lastSync,
            function (changes) {
                self.applyChanges(changes, callback);
            }
        );
    });

}

Final Notes

• This solution currently supports unidirectional (server to client) data synchronization. It could easily be expanded to support bidirectional synchronization.
• This solution currently implements “logical deletes”: items are not physically deleted from the table, but the value of their “deleted” column is set to true.
• As mentioned above, you could replace the Web SQL implementation with another data access strategy. For example, take a look at Brian Leroux’ Lawnchair for another local persistence solution.

Source Code

The source code is available in this GitHub repository.