This is the translation of my article that first appeared on Habrahabr.

Couchbase and Couchbase Lite

When developing data-driven mobile applications, we often encounter the customer's wish to fully access all of the app’s features, including changing the data, when the device is offline. The changes made to the data also have to sync with the backend when the device goes online. The backend is also concurrently accessed by desktop and web frontend applications which may also modify the data.

Public cloud synchronization is not always viable, especially when security concerns are in place, and customer wishes to keep all of their data on their private servers. In this article I’ll describe my experience of solving this task by using Couchbase database on the server and Couchbase Lite database in the mobile application with two-way replication between them.

The Couchbase database is a document-oriented distributed NoSQL database that ensures high performance by writing data into memory first, eventually persisting it onto the disk. Couchbase enables strong consistency between the nodes in a clustered environment by making them independent and equal, while each document being bound to a certain node. Couchbase is queried with indexed views that implement the MapReduce pattern.

Couchbase Lite is a lightweight version of Couchbase that is intended for desktop and mobile applications and is able to replicate with Couchbase server. Couchbase Lite is implemented on iOS, Android, Java and .NET platforms, so it can be used not only in mobile but also in desktop applications. It’s worth mentioning that the iOS version of Couchbase Lite currently has several advantages against other platforms. For instance, there is full-text search, and also automatic mapping of documents to Objective C and Swift objects.

For synchronization of Couchbase and Couchbase Lite, a CouchDB-almost-compatible replication protocol is used. Almost — because the authors don’t guarantee complete compatibility due to obscure documentation of CouchDB protocol which they even had to partly reverse. This protocol is implemented in Sync Gateway — a REST-based replication service. All clients that wish to sync data should connect to the central database using this service.

Couchbase Server installation and setup

Couchbase installation

The installation process of Couchbase differs between platforms and is described in the documentation. Let’s assume the database is already installed on localhost. The default location of admin console is http://localhost:8091/. Let’s go there and create a bucket named "demo" which we’ll use for storing our documents. To do that, open Data Buckets tab and click Create New Data Bucket button.

Enter the bucket name "demo" and limit it’s memory quota to 100 MB.

When all is done, a new bucket named demo will appear in the list of buckets, with a green circle beside it that indicates its normal activity.

Click the Documents button and observe that the newly created bucket is empty.

Sync Gateway setup

Sync Gateway installation and setup are described in the documentation. Here I’ll provide a sync-gateway-config.json file that will allow you to run the sample application that we’ll develop in this article:

{
     "interface":":4984",
     "adminInterface":"0.0.0.0:4985",
     "log": ["CRUD+", "REST+", "Changes+", "Attach+"],
     "databases":{
          "demo":{
               "bucket":"demo",
               "server":"http://localhost:8091",
               "users": {
                    "GUEST": {"disabled": false, "admin_channels": ["*"]}
               },
               "sync":`function(doc) {channel(doc.channels);}`
          }
     }
}

After running the Sync Gateway with this config file, you should observe the following log showing that the demo bucket is ready for acting as our central data synchronization storage:

23:27:02.411961 Enabling logging: [CRUD+ REST+ Changes+ Attach+]
23:27:02.412547 ==== Couchbase Sync Gateway/1.0.3(81;fa9a6e7) ====
23:27:02.412559 Configured Go to use all 8 CPUs; setenv GOMAXPROCS to override this
23:27:02.412604 Opening db /demo as bucket "demo", pool "default", server 
23:27:02.413160 Opening Couchbase database demo on 
23:27:02.601456 Reset guest user to config
23:27:02.601467 Starting admin server on 0.0.0.0:4985
23:27:02.603461 Changes+: Notifying that "demo" changed (keys="{_sync:user:}") count=2
23:27:02.604248 Starting server on :4984 ...

Refresh the page with the bucket document list, and you should see some internal Sync Gateway documents there which IDs start with _sync:

Console application

The code of the console application is available on GitHub together with the mobile application. It is mainly intended for demonstrating and testing the interaction of mobile and desktop databases and is comprised of a simple Java application that connects to an embedded Couchbase Lite database, which is also implemented in Java. The application is able to create a local document with an image attachment and a timestamp_added attribute. It also initiates replication of local changes to Couchbase Server.

Mobile application

The mobile application will show thumbnails of pictures that were added in a console application, persisted to the local database and replicated to the mobile database via server database. The process of creating this mobile application is described here in full. I chose the iOS platform for the mobile application as it is has better support for the Couchbase Lite API. The language used here is Swift.

Creating a project and adding dependencies

First let’s create a simple Single View Application:

To attach the couchbase-lite-ios library to the project, let's use the CocoaPods dependency manager. The CocoaPods installation is described in its documentation. Let’s initialize CocoaPods in the project directory:

pod init

Add the couchbase-lite-ios dependency to Podfile:

target 'CouchbaseSyncDemo' do
     pod 'couchbase-lite-ios', '~> 1.0'
end

Install the specified library into the project:

pod install

Now you should reopen the project as a workspace (CouchbaseSyncDemo.xcworkspace). Now add a bridging header file so you can use the CocoaPods-installed Objective C libraries in your Swift classes. To do that, add to the project the following header file, naming it CouchbaseSyncDemo-Bridging-Header.h:

#ifndef CouchbaseSyncDemo_CouchbaseSyncDemo_Bridging_Header_h
#define CouchbaseSyncDemo_CouchbaseSyncDemo_Bridging_Header_h
#import "CouchbaseLite/CouchbaseLite.h"
#endif

Specify this file in your Build Settings:

UI stub

Inherit the automatically generated ViewController class from the UICollectionViewController:

class ViewController: UICollectionViewController {

Open Main.storyboard and switch the default ViewController to a Collection View Controller, dragging it from the Object Library and redirecting the Storyboard Entry Point to it. In the Custom Class section of the Identity Inspector specify the generated ViewController. Also select the Collection View Cell and in its Attribute Inspector specify "cell" as its Reuse Identifier. The result is shown on the following screenshot:

Initializing and starting the replication

Create a class CouchbaseService that will incapsulate the database-related functionality and implement it as a singleton:

private let CouchbaseServiceInstance = CouchbaseService()

class CouchbaseService {

     class var instance: CouchbaseService {
          return CouchbaseServiceInstance
     }

}

Now open the demo database in the constructor of this class and start continuous pull replication. If the application is run inside the emulator, and Couchbase Server is running on the same machine, then we can use localhost as the address for replication. The continuous flag ensures that the replication runs continuously via long polling mechanism. You should also create the "images" view for extracting the list of all images:

private let pull: CBLReplication
private let database: CBLDatabase

private init() {

     // create or open the database
     database = CBLManager.sharedInstance().databaseNamed("demo", error: nil)

     // initiate pull replication
     let syncGatewayUrl = NSURL(string: "http://localhost:4984/demo/")
     pull = database.createPullReplication(syncGatewayUrl)
     pull.continuous = true;
     pull.start()

     // create a view of all documents in the database
     database.viewNamed("images").setMapBlock({(doc: [NSObject : AnyObject]!, emit: CBLMapEmitBlock!) -> Void in
          emit(doc["timestamp_added"], nil)
     }, version: "1")
}

Couchbase Lite views

Couchbase view is an indexed and automatically refreshed result of execution of a pair of functions — map and (optionally) reduce — on all of the documents in the bucket. Here the view is specified only by its map function that for each document returns its creation timestamp as the key. The key in views is also used to sort the view’s results, so the images will always be sorted by the time they were added. The version parameter specifies the view’s version and has to be changed each time we change the view’s code. The change in the version is a signal for Couchbase to rebuild the view using the new version of the code.

Views in Couchbase can be queried. A specific type of queries is a live query, which results in an automatically updated array of documents. Thanks to Objective C and Swift’s KVO feature, we can observe this array’s changes and update the interface of our application when new data arrives via replication.

As a matter of fact, this way of tracking the changes may only signal the fact that the query results changed, but not the concrete added or deleted records. Such information would allow us to minimize the updates to interface — and gladly, Couchbase Lite provides it via the kCBLDatabaseChangeNotification event. This event signals of all new revisions that are added to the database. But in this example I decided to use the more simple live query mechanism.

Dealing with the data

Let’s add to CouchbaseService class a function for executing live query to our images view:

func getImagesLiveQuery() -> CBLLiveQuery {
     return database.viewNamed("images").createQuery().asLiveQuery()
}

The iOS implementation of Couchbase Lite stands out from other platforms by its automatic bi-directional mapping of documents to object models. This mapping leverages dynamic features of Objective C. A Swift implementation of this mapping is as follows:

@objc
class ImageModel: CBLModel {

     @NSManaged var timestamp_added: NSString

     var imageInternal: UIImage?

     var image: UIImage? {
          if (imageInternal == nil) {
               imageInternal = UIImage(data: self.attachmentNamed("image").content)
          }
          return imageInternal
     }

}

The timestamp_added attribute is dynamically linked to the corresponding field in the document, and the attachmentNamed: function allows us to receive binary data attached to the document. To convert the document to its object model, we can use the ImageModel constructor.

Binding interface and data

All that’s left to do is to subscribe ViewController to live query refresh and process this refresh by reloading the collection view. The images attribute keeps the list of documents converted to object models.

private var images: [ImageModel] = []

private var query: CBLLiveQuery?

override func viewDidAppear(animated: Bool) {
     query = CouchbaseService.instance.getImagesLiveQuery()
     query!.addObserver(self, forKeyPath: "rows", options: nil, context: nil)
}

override func observeValueForKeyPath(keyPath: String, ofObject object: AnyObject, change: [NSObject : AnyObject], context: UnsafeMutablePointer) {
     if object as? NSObject == query {
          images.removeAll()
          var rows = query!.rows
          while let row = rows.nextRow() {
               images.append(ImageModel(forDocument: row.document))
          }
          collectionView?.reloadData()
     }
}

The UICollectionViewDataSource protocol methods are quite typical and self-explanatory, except that we use the "cell" reuse identifier that we specified for the collection view cell in the storyboard earlier.

override func collectionView(collectionView: UICollectionView, numberOfItemsInSection section: Int) -> Int {
     return images.count
}

override func collectionView(collectionView: UICollectionView, cellForItemAtIndexPath indexPath: NSIndexPath) -> UICollectionViewCell {
     let cell = collectionView.dequeueReusableCellWithReuseIdentifier("cell", forIndexPath: indexPath) as! UICollectionViewCell
     cell.backgroundView = UIImageView(image:images[indexPath.item].image)
     return cell
}

Running the application

Now let’s see what we’ve achieved. Let’s run the console application. By issuing the start command inside the console application, we’re starting the replication; with the attach command we can create several documents with images.

    start

CBL started
апр 15, 2015 11:41:14 PM com.github.oxo42.stateless4j.StateMachine publicFire
INFO: Firing START
push event: PUSH replication event. Source: com.couchbase.lite.replicator.Replication@144c1e50 Transition: INITIAL -> RUNNING Total changes: 0 Completed changes: 0
апр 15, 2015 11:41:15 PM com.github.oxo42.stateless4j.StateMachine publicFire
push event: PUSH replication event. Source: com.couchbase.lite.replicator.Replication@144c1e50 Transition: RUNNING -> IDLE Total changes: 0 Completed changes: 0
INFO: Firing WAITING_FOR_CHANGES

    attach http://upload.wikimedia.org/wikipedia/commons/4/41/Harry_Whittier_Frees_-_What%27s_Delaying_My_Dinner.jpg

Saved image with id = 8e357b3c-1c7f-4432-b91d-321dc1c9fd9d
push event: PUSH replication event. Source: com.couchbase.lite.replicator.Replication@144c1e50 Total changes: 1 Completed changes: 0
push event: PUSH replication event. Source: com.couchbase.lite.replicator.Replication@144c1e50 Total changes: 1 Completed changes: 1

The data is replicated to the mobile device and gets displayed right away:

Summary

In this article I demonstrated synchronization of data between server side and mobile application by means of Couchbase and Couchbase Lite. This allows us to create a mobile application that can be fully functional while the device is offline. In my future articles I’ll explore document revisions and replication protocol of Couchbase Lite more closely and test it for bad connectivity, sudden backgrounding of application and other perils of mobile app development.

Exception With Butterfly Wings

Thursday, May 7, 2015

Offline data access and synchronization in a mobile application with Couchbase Lite