Couchbase Sync Gateway is a key component of the Couchbase Mobile stack. It is an internet-facing synchronization mechanism that securely syncs data across devices as well as between devices and the cloud. Couchbase Mobile uses a websocket based replication protocol.
The core functions of the Sync Gateway include:
This tutorial will demonstrate how to -
"Live Queries"
or Query events within your Couchbase Lite clients to be asynchronously notified of changes.We will be using Xamarin (iOS/Android/UWP) apps as examples of Couchbase Lite enabled clients.
You can learn more about the Sync Gateway here.
This tutorial assumes familiarity with building apps with Xamarin, more specifically Xamarin.Forms using C# and XAML.
If you are unfamiliar with the basics of Couchbase Lite, it is recommended that you walk through the following tutorials:
For iOS/Mac development, you will need a Mac running MacOS 11 or 12
iOS/Mac (Xcode 12/13) - Download latest version from the Mac App Store or via Xcodes
Note: If you are using an older version of Xcode, which you need to retain for other development needs, make a copy of your existing version of Xcode and install the latest Xcode version. That way you can have multiple versions of Xcode on your Mac. More information can be found in Apple's Developer Documentation. The open source Xcodes project simplifies this process.
Note: You can not edit or debug UWP projects with Visual Studio for Mac and you can't edit or debug Mac projects with Visual Studio for PC.
We will be working with a simple "User Profile" app which we introduced in the Fundamentals Tutorial and extended in the Query tutorial.
In this tutorial, we will be further extending that app to support data sync. It will do the following:
sync function
).The User Profile demo app is a Xamarin.Forms based solution that supports iOS and Android mobile platforms along with the UWP desktop platform. The solution utilizes various design patterns and principles such as MVVM, IoC, and the Repository Pattern.
The solution consists of seven projects:
.ipa
file..apk
file..exe
file.Before diving into the code for the apps, it is important to point out the Couchbase Lite dependencies within the solution. The Couchbase.Lite Nuget package is included as a reference within four projects of this solution:
The Couchbase.Lite
Nuget package contains the core functionality for Couchbase Lite. In the following sections you will dive into the capabilities it the package provides.
git clone https://github.com/couchbase-examples/dotnet-xamarin-cblite-userprofile-sync
UserProfileDemo.sln
. The project would be located at /path/to/dotnet-xamarin-cblite-userprofile-sync/src
.open UserProfileDemo.sln
If have followed along the tutorial on Query tutorial, you can skip this section and proceed to the Backend Installation section as we have not made any changes to the Data model for this tutorial.
Couchbase Lite is a JSON Document
Store. A Document
is a logical collection of named fields and values.The values are any valid JSON types. In addition to the standard JSON types, Couchbase Lite supports some special types like Date
and Blob
. While it is not required or enforced, it is a recommended practice to include a "type" property that can serve as a namespace for related.
The app deals with a single Document
with a "type" property of "user". The document ID is of the form "user::demo@example.com".
An example of a document would be
{
"type":"user",
"name":"Jane Doe",
"email":"jane.doe@earth.org",
"address":"101 Main Street",
"image":CBLBlob (image/jpg),
"university":"Missouri State University"
}
The "user" Document is encoded to a class
named UserProfile.
public class UserProfile
{
public string type => "user";
public string Id { get; set; }
public string Name { get; set; }
public string Email { get; set; }
public string Address { get; set; }
public byte[] ImageData { get; set; }
public string Description { get; set; }
public string University { get; set; }
}
The app comes bundled with a collection of documents of type "university". Each Document
represents a university.
{
"type":"university","web_pages": [
"http://www.rpi.edu"
],
"name": "Rensselaer Polytechnic Institute",
"alpha_two_code": "US",
"state-province": null,
"domains": [
"rpi.edu"
],
"country": "United States"
}
The "university" Document
is encoded to a class
named University.
public class University
{
public string Name { get; set; }
public string Country { get; set; }
}
We will install Couchbase Server and Sync Gateway using Docker.
Couchbase Server and Sync Gateway Server need to communicate with each other over the network. A network bridge in docker allows network traffic between servers. Create a docker network bridge named workshop.
docker network ls
docker network create -d bridge workshop
We have a custom docker image priyacouch/couchbase-server-userprofile:7.0.0-dev
of Couchbase Server, which creates an empty bucket named userprofile and an RBAC user admin with sync gateway role.
Alternatively, you can follow the instructions in our documentation — see: Get Started - Prepare, to install Couchbase Server and configure it with the relevant bucket.
docker stop cb-server && docker rm cb-server
docker run -d --name cb-server \
--network workshop \
-p 8091-8094:8091-8094 -p 11210:11210 \
priyacouch/couchbase-server-userprofile:7.0.0-dev
The server could take a few minutes to deploy and fully initialize; so be patient.
docker logs -f cb-server
When the setup is completed, you should see output similar to that shown in below:
a. Open up http://localhost:8091 in a browser. b. Sign in as Administrator and password in login page. c. Go to "buckets" menu and confirm "userprofile" bucket is created
Now we will install, configure, and run Sync Gateway.
When using Sync Gateway, we can opt to provide a bootstrap configuration -- see: Sync Gateway Configuration. We would then provision database, sync and other configuration using the Admin REST endpoints Alternatively, we can continue to run in legacy-mode, using the Pre-3.0 configuration.
In this tutorial - for the purposes of backward compatibility - we will run 3.x using its legacy configuration option. That is, we will be running with the disable_persistent_config
option in the configuration file set to true
. You can, if you wish, run a 2.8 version of Sync Gateway instead.
The configuration files corresponding to this sample application are shown in Table 1. They are available in the github repo hosting the app, which you cloned - look in: /path/to/cloned/repo/dotnet-xamarin-cblite-userprofile-sync/src/
Table 1. Available configuration files
Release | Filename |
---|---|
3.x | sync-gateway-config-userprofile-demo-3-x-legacy.json |
2.x | sync-gateway-config-userprofile-demo-2-x.json |
Let us configure and launch Sync Gateway in a Docker container.
cd /path/to/cloned/repo/dotnet-xamarin-cblite-userprofile-sync/src
docker stop sync-gateway && docker rm sync-gateway
Launch Sync Gateway in a Docker container using directions below based on the version you are using.
Configuring and running Sync Gateway 3.x in Docker using the configuration in sync-gateway-config-userprofile-demo-3-x-legacy.json
.
Note the use of disable_persistent_config
in the configuration file to force legacy configuration mode.
docker run -p 4984-4986:4984-4986 \
--network workshop \
--name sync-gateway \
-d \
-v `pwd`/sync-gateway-config-userprofile-demo-3-x-legacy.json \
/etc/sync_gateway/sync_gateway.json \
couchbase/sync-gateway:3.0.0-enterprise \
/etc/sync_gateway/sync_gateway.json
Configuring and running Sync Gateway 2.8.
docker run -p 4984-4986:4984-4986 \
--network workshop \
--name sync-gateway \
-d \
-v `pwd`/sync-gateway-config-userprofile-demo-2-x.json:\
/etc/sync_gateway/sync_gateway.json \
couchbase/sync-gateway:2.8.4-enterprise \
/etc/sync_gateway/sync_gateway.json
Configure and run Sync Gateway 3.0 in legacy mode.
docker run -p 4984-4986:4984-4986 ^
--network workshop ^
--name sync-gateway ^
-d -v %cd%/sync-gateway-config-userprofile-demo-3-x-legacy.json:^
/etc/sync_gateway/sync_gateway.json ^
couchbase/sync-gateway:3.0.0-enterprise ^
/etc/sync_gateway/sync_gateway.json
Configuring and running Sync Gateway 2.8.
docker run -p 4984-4986:4984-4986 ^
--network workshop ^
--name sync-gateway ^\
-d ^
-v %cd%/sync-gateway-config-userprofile-demo-2-x.json:^
etc/sync_gateway/sync_gateway.json ^
couchbase/sync-gateway:2.8.4-enterprise ^
/etc/sync_gateway/sync_gateway.json
Now we can confirm that the Sync Gateway is up and running.
docker logs -f sync-gateway
You will see a series of log messages. Make sure there are no errors.
{"couchdb":"Welcome","vendor": { "name":"Couchbase Sync Gateway", "version":"3.0" },
"version":"Couchbase Sync Gateway/3.0.0(460;26daced) EE"}
Now that we have the server and the sync gateway installed, we can verify data sync between Couchbase Lite enabled apps.
A key component of the sync process is the Sync Function and we will first look at how that can be set-up to control how data sync works.
The Sync Function is a Javascript function that is specified as part of the Sync Gateway Configuration. It handles Authorization, Data Validation, Data Routing, and Access Countrol.
Open the your configuration file using a text editor of your choice. It will be located in the repo at /path/to/cloned/repo/dotnet-xamarin-cblite-userprofile-sync/src
.
Locate the sync
setting in the file you used.
Now you can follow along with the rest of the sections below.
We use Basic Authentication in our application. The Id of the user making the request is specified in the Authorization
header.
Locate the /*Authorization*/
section of the Sync Function. You will see we are using the Sync functions requireUser()
API to verify that the email
property specified in the Document matches the Id of the user making the request.
function sync(doc, oldDoc) {
....
/* Authorization */
// Verify the user making the request is the same as the one in doc's email
requireUser(doc.email);
.....
}
In this case, we are doing some basic validation of the contents of the Document:
/* Data Validation */
// Validate the presence of email field.
// This is the "username"
validateNotEmpty("email", doc.email);
// Validate that the document Id _id is prefixed by owner
var expectedDocId = "user" + "::" + doc.email;
if (expectedDocId != doc._id) {
// reject document
throw({forbidden: "user doc Id must be of form user::email"});
}
email
property is not null. If it's null, we throw a JS exception (see validateNotEmpty()
function)Id
of the Document is of the required format (i.e. "user::demo@example.com"). We throw an exception if that's not the case.email
property value has not changed. Again, we throw an exception if that's not the case.NOTE: You can learn more about the Sync Function in the documentation here: Sync Function API.
Channels
are a mechanism to "tag" documents. They are typically used to route/seggregate documents based on the contents of those document.
When combined with the access()
and requireAccess()
API, the channel() API can be used to enforce Access Control.
As we shall see in a later section, clients can use channels to pull only a subset of documents.
/* Routing */
// Add doc to the user's channel.
var email = getEmail();
var channelId = "channel."+ username;
channel(channelId);
We can enforce access control to channels using the access ( ) API. This approach ensures that only users with access to a specific channel will be able to retrieve documents in the channel.
// Give user read access to channel
access(username, channelId);
Two-way Replication between the app and the Sync Gateway is enabled when user logs into the app.
StartReplicationAsync
method.public async Task StartReplicationAsync(
string username,
string password,
string[] channels,
ReplicatorType replicationType = ReplicatorType.PushAndPull,
bool continuous = true)
ReplicatorConfig
instance that specifies the source and target database and you can optionally, override the default configuration settings.var configuration = new ReplicatorConfiguration(database, targetUrlEndpoint) // <1>
{
ReplicatorType = replicationType, // <2>
Continuous = continuous, // <3>
Authenticator = new BasicAuthenticator(username, password), // <4>
Channels = channels?.Select(x => $"channel.{x}").ToArray() // <5>
};
<1> Initialize with Source
as the local Couchbase Lite database and the remote
target as the Sync Gateway
<2> Replication type
of PushAndPull
indicates that we require two-way sync. A value of .Pull
specifies that we only pull data from the Sync Gateway. A value of .Push
specifies that we only push data.
<3> The Continuous
mode is specified to be true which means that changes are synced in real-time. A value of false which implies that data is only pulled from the Sync Gateway.
<4> This is where you specify the authentication credentials of the user. In the Authorization section, we discussed that the Sync Gateway can enforce authorization check using the RequireUser
API.
<5> The Channels
are used to specify the channels to pull from. Only documents belonging to the specified channels are synced. This is subject to Access Control rights enforced at the Sync Gateway. This means that if a client does not have access to documents in a channel, the documents will not be synched even if the client specifies it in the replicator configuration.
Replicator
with the ReplicatorConfiguration
_replicator = new Replicator(configuration);
Replicator
to be asynchronously notified of state changes. This could be useful for instance, to inform the user of the progress of the replication. This is an optional step._replicatorListenerToken = _replicator.AddChangeListener(OnReplicatorUpdate);
OnReplicatorUpdate
void OnReplicatorUpdate(object sender, ReplicatorStatusChangedEventArgs e)
{
var status = e.Status;
switch (status.Activity)
{
case ReplicatorActivityLevel.Busy:
Console.WriteLine("Busy transferring data.");
break;
case ReplicatorActivityLevel.Connecting:
Console.WriteLine("Connecting to Sync Gateway.");
break;
case ReplicatorActivityLevel.Idle:
Console.WriteLine("Replicator in idle state.");
break;
case ReplicatorActivityLevel.Offline:
Console.WriteLine("Replicator in offline state.");
break;
case ReplicatorActivityLevel.Stopped:
Console.WriteLine("Completed syncing documents.");
break;
}
if (status.Progress.Completed == status.Progress.Total)
{
Console.WriteLine("All documents synced.");
}
else
{
Console.WriteLine($"Documents {status.Progress.Total - status.Progress.Completed} still pending sync");
}
}
_replicator.Start();
When user logs out of the app, the replication is stopped before the database is closed.
Stop
function.public void StopReplication()
_replicator.RemoveChangeListener(_replicatorListenerToken);
_replicator.Stop()
NOTE: All open replicators must be stopped before database is closed. There will be an exception if you attempt to close the database without closing the active replicators.
Couchbase Lite applications can set up live queries in order to be asynchronously notified of changes to the database that affect the results of the query. This can be very useful, for instance, in keeping a UI View up-to-date with the results of a query.
In our app, the user profile view is kept up-to-date using a live query that fetches the user profile data used to populate the view. This means that, if the replicator pulls down changes to the user profile, they are automatically reflected in the view.
To see this:
GetAsync
function. Calling this method and passing in a value for the Action<UserProfile>
named userProfileUpdated
implies that the caller wishes to be notified of any changes to query results via delegation.public async Task<UserProfile> GetAsync(string userProfileId, Action<UserProfile> userProfileUpdated)
QueryBuilder
API. If you are unfamiliar with this API, please check out our Query tutorial._userQuery = QueryBuilder
.Select(SelectResult.All())
.From(DataSource.Database(database))
.Where(Meta.ID.EqualTo(Expression.String(userProfileId)));
_userQueryToken = _userQuery.AddChangeListener(
(object sender, QueryChangedEventArgs e) => // <1>
{
if (e?.Results != null && e.Error == null)
{
foreach (var result in e.Results.AllResults())
{
var dictionary = result.GetDictionary("userprofile"); // <2>
if (dictionary != null)
{
userProfile = new UserProfile // <3>
{
Name = dictionary.GetString("name"), // <4>
Email = dictionary.GetString("email"),
Address = dictionary.GetString("address"),
University = dictionary.GetString("university"),
ImageData = dictionary.GetBlob("imageData")?.Content
};
}
}
if (userProfile != null)
{
userProfileUpdated.Invoke(userProfile);
}
}
});
<1> Attach a listener callback to the query. Attaching a listerner automatically makes it live so any time there is a change in the user profile data in the underlying database, the callback would be invoked.
<2> The SelectResult.all()
method is used to query all the properties of a document. In this case, the document in the result is embedded in a dictionary where the key is the database name, which is "userprofiles". So, we retrieve the DictionaryObject
at key "userprofiles".
<3> Create an instance of UserProfile. This will be populated with the query results.
<4> We use appropriate type getters to retrieve values and populate the UserProfile instance
Tip: If you are running the application in Android emulator(s) then you will need to change the URL of the remote Sync Gateway in DatabaseManager.cs.
- Find and uncomment the following line:
readonly Uri _remoteSyncUrl = new Uri("ws://10.0.2.2:4984"); 2. Comment out the standard line: readonly Uri _remoteSyncUrl = new Uri("ws://localhost:4984");
In this exercise, we will observe how changes made on one app are synced across to the other app
In this exercise, we will observe changes made via Sync Gateway are synced over to the apps
Make sure you have completed Exercise 1. This is to ensure that you have the appropriate user profile document (with document Id of "user::demo@example.com") created through the app and synced over to the Sync Gateway.
Open the command terminal and issue the following command to get the user profile document via GET Document REST API. We will be using curl
to issue the request. If you haven't done so, please install curl as indicated in the Prerequisites section.
curl -X GET http://localhost:4984/userprofile/user::demo@example.com --user demo@example.com
NOTE: This GET retrieves the userprofile document with the id user::demo@example.com
{
"_attachments": { <2>
"blob_1": {
"content_type": "image/jpeg",
"digest": "sha1-S8asPSgzA+F+fp8/2DdIy4K+0U8=",
"length": 14989,
"revpos": 2,
"stub": true
}
},
"_id": "user::demo@example.com",
"_rev": "2-3a76cfa911e2c54d1e82b29dbffc7f4e5a9bc265", //<1>
"address": "",
"email": "demo@example.com",
"image": {
"@type": "blob",
"content_type": "image/jpeg",
"digest": "sha1-S8asPSgzA+F+fp8/2DdIy4K+0U8=",
"length": 14989
},
"name": "",
"type": "user",
"university": "Missouri State University"
}
If you had updated an image via the mobile app, you should see an "_attachments" property. This entry holds an array of attachments corresponding to each image blob entry added by the mobile app. This property is added by the Sync Gateway when it processes the document. You can learn more about how image Blob types are mapped to attachments here.
Record the revision Id of the document. You will need this when you update the document
NOTE: We chose to show how to update the address field via the REST API. You can choose to update any other profile information if you like. You will be prompted to enter the users password when you submit the curl command.
curl -X PUT \
'http://localhost:4985/userprofile/user::demo@example.com?rev=3-12d203d6024c8b844c5ed736c726ac63379e05dc' \
-H 'Accept: application/json' \
-H 'Cache-Control: no-cache' \
-H 'Content-Type: application/json' \
-d '{
"address": "101 Main Street", //<1>
"email": "demo@example.com",
"image": {
"@type": "blob",
"content_type": "image/jpeg",
"digest": "sha1-S8asPSgzA+F+fp8/2DdIy4K+0U8=",
"length": 14989
},
"name": "",
"type": "user",
"university": "Missouri State University"
}'
Here, in the
PUT
, we specify the:
- user id (
user::demo@example.com
)- revision Id (from the previous step
3-033fcbaf269d65a9247067be76d664f1111d033b
) to select the item we want to update
Confirm that you get a HTTP "201 Created" status code
As soon as you update the document via the Sync Gateway REST API, confirm that the changes show up in the mobile app on the simulator.
Data conflicts are inevitable in an environment where you can potentially have multiple writes updating the same data concurrently. Couchbase Mobile supports Automated Conflict Resolution.
You can learn more about automated conflict resolution in this blog Document Conflicts & Resolution.
Congratulations on completing this tutorial!
This tutorial walked you through an example of how to use a Sync Gateway to synchronize data between Couchbase Lite enabled clients. We discussed how to configure your Sync Gateway to enforce relevat access control, authorization and data routing between Couchbase Lite enabled clients.
Check out the following links for further details