Couchbase Sync Gateway is a key component of the Couchbase Mobile stack. It is an Internet-facing synchronization mechanism that securely syncs data across devices as well as between devices and the cloud. Couchbase Mobile uses a websocket based replication protocol.
The core functions of the Sync Gateway include:
In this part of our learning path, you will walk you through setting up Couchbase Server and Sync Gateway so that in future steps we can replicate documents from the mobile app to Couchbase Server and vice versa. You will learn the fundamentals of:
In this step of the learning path, we will be using docker and docker compose and should have both of these installed before continuing. Docker Desktop provides these tools and UI for Mac, Windows, and Linux.
You might also want a text editor outside of Android Studio to review the docker and docker compose config files. Any text editor will work including Android Studio, but for many developers Visual Studio Code is a good solution as it provides extensions for Docker config file formatting and YAML support, which is a file format Docker Compose uses. You can download Visual Studio code here. If you are using Visual Studio code make sure you install the Docker extension. The YAML Language Support is another great extension that can be a major quality of life improvement if you use YAML files a lot.
NOTE: This part of the learning path is longer than normal as there are several configuration files to review. For developers looking to try out the containers without reviewing the configuration files, you can skip to the Try It Out section.
Learn Couchbase Lite with Kotlin and Jetpack Compose
repository from GitHub.git clone https://github.com/couchbase-examples/android-kotlin-cbl-learning-path.git
Docker and Docker Compose will be used to create a Couchbase Server container that has a one node cluster setup with a bucket, a user for sync gateway to perform replication between the sync gateway server and server, and indexes for the bucket.
This is accomplished by creating a custom Dockerfile that defines the Couchbase Server base image along with a shell scripts to perform the automation of the Couchbase Server cluster setup and importing of sample data.
FROM couchbase:latest AS stage_base
COPY init-cbserver.sh /opt/couchbase/init/
COPY sample-data.json /opt/couchbase/init/
base
image for this container. We are using the couchbase:latest
image. As of Couchbase Server 7.1 - ARM64 and X86 images are provided.Next open the init-cbserver.sh file found in the same folder as the Dockerfile. This shell script is well documented with comments before each line, however we will still go through the script from a high level.
#!/bin/bash
# used to start couchbase server - can't get around this as docker compose
# only allows you to start one command - so we have to start couchbase like the Dockerfile would
# https://github.com/couchbase/docker/blob/master/enterprise/couchbase-server/7.1.1/Dockerfile#L88
/entrypoint.sh couchbase-server &
NOTE Sleep statements make sure that things complete before moving on to the next step. The 10-second delay is set after Couchbase Server is started to make sure the cluster is completely running before moving on to the next command. The delay is conservative and could be shortened based on the speed of your computer.
sleep 10s
/opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1 \
--cluster-username $COUCHBASE_ADMINISTRATOR_USERNAME \
--cluster-password $COUCHBASE_ADMINISTRATOR_PASSWORD \
--services data,index,query \
--cluster-ramsize $COUCHBASE_RAM_SIZE \
--cluster-index-ramsize $COUCHBASE_INDEX_RAM_SIZE \
--index-storage-setting default
/opt/couchbase/bin/couchbase-cli bucket-create -c localhost:8091 \
--username $COUCHBASE_ADMINISTRATOR_USERNAME \
--password $COUCHBASE_ADMINISTRATOR_PASSWORD \
--bucket $COUCHBASE_BUCKET \
--bucket-ramsize $COUCHBASE_BUCKET_RAMSIZE \
--bucket-type couchbase
/opt/couchbase/bin/couchbase-cli user-manage \
--cluster http://127.0.0.1 \
--username $COUCHBASE_ADMINISTRATOR_USERNAME \
--password $COUCHBASE_ADMINISTRATOR_PASSWORD \
--set \
--rbac-username $COUCHBASE_RBAC_USERNAME \
--rbac-password $COUCHBASE_RBAC_PASSWORD \
--roles mobile_sync_gateway[*] \
--auth-domain local
/opt/couchbase/bin/curl -v http://localhost:8093/query/service \
-u $COUCHBASE_ADMINISTRATOR_USERNAME:$COUCHBASE_ADMINISTRATOR_PASSWORD \
-d 'statement=CREATE INDEX idx_projects_team on projects(team)'
sleep 2s
/opt/couchbase/bin/curl -v http://localhost:8093/query/service \
-u $COUCHBASE_ADMINISTRATOR_USERNAME:$COUCHBASE_ADMINISTRATOR_PASSWORD \
-d 'statement=CREATE INDEX idx_projects_type on projects(type)'
sleep 2s
/opt/couchbase/bin/curl -v http://localhost:8093/query/service \
-u $COUCHBASE_ADMINISTRATOR_USERNAME:$COUCHBASE_ADMINISTRATOR_PASSWORD \
-d 'statement=CREATE INDEX idx_projects_projectId on projects(projectId)'
/opt/couchbase/bin/cbimport json --format list \
-c http://localhost:8091 \
-u $COUCHBASE_ADMINISTRATOR_USERNAME \
-p $COUCHBASE_ADMINISTRATOR_PASSWORD \
-d "file:///opt/couchbase/init/sample-data.json" -b 'projects' -g %projectId%
Now that we have reviewed how the Couchbase Server will be created, let's review how the Sync Gateway server will be created.
FROM couchbase/sync-gateway:latest AS stage_base
COPY sync-gateway.json /etc/sync_gateway/config.json
base
image for this container. We are using the sync-gateway:latest
image.The Sync Gateway server uses a configuration file when it starts to load in all important settings. In the demo app, this file is setup in Legacy Pre-3.0 Configuration for sake of simplicity. Let's review the configuration file by sections.
"interface":":4984",
"adminInterface":":4985",
"log": ["*"],
"logging": {
"log_file_path": "/var/tmp/sglogs",
"console": {
"log_level": "debug",
"log_keys": ["*"]
},
"error": {
"enabled": true,
"rotation": {
"max_size": 20,
"max_age": 180
}
},
"warn": {
"enabled": true,
"rotation": {
"max_size": 20,
"max_age": 90
}
},
"info": {
"enabled": false
},
"debug": {
"enabled": false
}
},
"disable_persistent_config":true,
"server_tls_skip_verify": true,
"use_tls_server": false,
Note: To use TLS with something like self-signed certificates would greatly increase the complexity and length of this tutorial. For this reason the tutorial shows the configuration without certificates.
"databases": {
"projects": {
"import_docs": true,
"bucket":"projects",
"server": "couchbase://couchbase-server",
"enable_shared_bucket_access":true,
"delta_sync": {
"enabled":false
},
"num_index_replicas":0,
"username": "admin",
"password": "P@$$w0rd",
"users": {
"demo@example.com": { "password": "P@ssw0rd12", "admin_roles": ["team1"] },
"demo1@example.com": { "password": "P@ssw0rd12", "admin_roles": ["team1"] },
"demo2@example.com": { "password": "P@ssw0rd12", "admin_roles": ["team2"] },
"demo3@example.com": { "password": "P@ssw0rd12", "admin_roles": ["team2"] },
"demo4@example.com": { "password": "P@ssw0rd12", "admin_roles": ["team3"] },
"demo5@example.com": { "password": "P@ssw0rd12", "admin_roles": ["team3"] },
"demo6@example.com": { "password": "P@ssw0rd12", "admin_roles": ["team4"] },
"demo7@example.com": { "password": "P@ssw0rd12", "admin_roles": ["team4"] },
"demo8@example.com": { "password": "P@ssw0rd12", "admin_roles": ["team5"] },
"demo9@example.com": { "password": "P@ssw0rd12", "admin_roles": ["team5"] },
"demo10@example.com": { "password": "P@ssw0rd12", "admin_roles": ["team6"] },
"demo11@example.com": { "password": "P@ssw0rd12", "admin_roles": ["team6"] },
"demo12@example.com": { "password": "P@ssw0rd12", "admin_roles": ["team7"] },
"demo13@example.com": { "password": "P@ssw0rd12", "admin_roles": ["team8"] },
"demo14@example.com": { "password": "P@ssword12", "admin_roles": ["team9"] },
"demo15@example.com": { "password": "P@ssw0rd12", "admin_roles": ["team10"] }
},
"roles": {
"team1": { "admin_channels": [ "channel.team1" ] },
"team2": { "admin_channels": [ "channel.team2" ] },
"team3": { "admin_channels": [ "channel.team3" ] },
"team4": { "admin_channels": [ "channel.team4" ] },
"team5": { "admin_channels": [ "channel.team5" ] },
"team6": { "admin_channels": [ "channel.team6" ] },
"team7": { "admin_channels": [ "channel.team7" ] },
"team8": { "admin_channels": [ "channel.team8" ] },
"team9": { "admin_channels": [ "channel.team9" ] },
"team10": { "admin_channels": [ "channel.team10" ] }
},
The next part of the configuration is the import_filter. The import_filter is a string value of Javascript code that controls whether a document written to the Couchbase Server bucket should be made available to Couchbase Mobile clients.
function(doc){
console.log("********Processing import filter - documents from couchbase server");
if (doc.type == 'project'
|| doc.type == 'location'
|| doc.type == 'user'
|| . doc.type == 'audit'){
return true;
}
return false;
}
The sync section of the configuration is used to define custom business logic. The sync is a string value of Javascript code that will run every time a new document, revision, or deletion is added to a database. The sync function will examine the document and custom business logic can then calculate things like access control to limit which users can see which documents. The demo app is a simple example of custom business logic. See the Sync Function API and Access Control How-To guides for more detailed information and a listing of other API functions available.
function sync(doc, oldDoc) { // <1>
/* Data Validation */
validateNotEmpty("type", doc.type); // <2>
if (doc.type == 'location'){ // <3>
console.log("********Processing Location Docs - setting it to global/public");
channel('!'); // <3>
} else { // <4>
console.log("********Processing Team Docs");
validateNotEmpty("team", doc.team); // <5>
if (!isDelete()) { // <6>
/* Routing -- add channel routing rules here for document */
var team = getTeam(); // <7>
var channelId = "channel." + team; // <8>
console.log("********Setting Channel to " + channelId);
channel(channelId); // <9>
/* Authorization - Access Control */
requireRole(team); // <10>
access("role:team1", "channel.team1"); // <11>
access("role:team2", "channel.team2"); // <11>
access("role:team3", "channel.team3"); // <11>
access("role:team4", "channel.team4"); // <11>
access("role:team5", "channel.team5"); // <11>
access("role:team6", "channel.team6"); // <11>
access("role:team7", "channel.team7"); // <11>
access("role:team8", "channel.team8"); // <11>
access("role:team9", "channel.team9"); // <11>
access("role:team10", "channel.team10"); // <11>
}
}
// get type property
function getType() {
return (isDelete() ? oldDoc.type : doc.type);
}
// get email Id property
function getTeam() {
return (isDelete() ? oldDoc.team : doc.team);
}
// Check if document is being created/added for first time
function isCreate() {
// Checking false for the Admin UI to work
return ((oldDoc == false) || (oldDoc == null || oldDoc._deleted) && !isDelete());
}
// Check if this is a document delete
function isDelete() {
return (doc._deleted == true);
}
// Verify that specified property exists
function validateNotEmpty(key, value) {
if (!value) {
throw({forbidden: key + " is not provided."});
}
}
}
!
to the channel function will make the document global and giving all users access to the document.The docker-compose.yml file is provide to configure and build the Couchbase Server and Sync Gateway containers that we have reviewed.
couchbase-server:
build: ./couchbase-server
ports:
- 8091-8096:8091-8096
- 11210:11210
environment:
- CLUSTER_NAME=couchbase-demo
- COUCHBASE_ADMINISTRATOR_USERNAME=Administrator
- COUCHBASE_ADMINISTRATOR_PASSWORD=P@$$w0rd12
- COUCHBASE_BUCKET=projects
- COUCHBASE_BUCKET_RAMSIZE=512
- COUCHBASE_RBAC_USERNAME=admin
- COUCHBASE_RBAC_PASSWORD=P@$$w0rd
- COUCHBASE_RBAC_NAME=admin
- COUCHBASE_RAM_SIZE=2048
- COUCHBASE_INDEX_RAM_SIZE=512
hostname: couchbase-server
container_name: couchbase-server
working_dir: /opt/couchbase
stdin_open: true
tty: true
networks:
- workshop
entrypoint: [""]
command: sh -c "/opt/couchbase/init/init-cbserver.sh"
sync-gateway:
build: ./sync-gateway
ports:
- 4984-4986:4984-4986
hostname: sync-gateway
container_name: sync-gateway
depends_on:
- couchbase
working_dir: /docker-syncgateway
stdin_open: true
tty: true
networks:
- workshop
Finally, we define the network configuration and driver to use.
networks:
workshop:
driver: bridge
Now that we have reviewed all the files that are used to create the containers, open a terminal window or command prompt depending on your platform of choice.
If you are running in Mac or standard PC laptop with Windows or Linux on it you can follow the directions below:
docker-compose up -d
The docker containers should start downloading, then build, and finally start up
You can check the status of docker using either Docker Desktop or the terminal commands.
Docker Desktop Users should see a container listing after launching the app. The group name of the containers is the name of the directory of the code repo on your computer where you ran docker compose. In the example it's named android-kotlin-cbl-learning-path
.
docker-compose ls
docker container ls
Docker Desktop Users can select each container in Docker Desktop to get detained information and logs about the container running to validate the containers were built properly or use the terminal to gather information.
Docker Desktop Users - select the couchbase-server container. You should see logging information.
docker container logs couchbase-server
Administrator : P@$w0rd12
Starting Couchbase Server -- Web UI available at http://<ip>:8091
and logs available in /opt/couchbase/var/lib/couchbase/logs
SUCCESS: Cluster initialized
SUCCESS: Bucket created
SUCCESS: User admin set
* Trying 127.0.0.1:8093...
* Connected to localhost (127.0.0.1) port 8093 (#0)
* Server auth using Basic with user 'Administrator'
> POST /query/service HTTP/1.1
> Host: localhost:8093
> Authorization: Basic QWRtaW5pc3RyYXRvcjpQQCR3MHJkMTI=
> User-Agent: curl/7.78.0-DEV
> Accept: */*
> Content-Length: 58
> Content-Type: application/x-www-form-urlencoded
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Length: 238
< Content-Type: application/json; version=7.1.0-N1QL
< Date: Thu, 05 May 2022 16:29:41 GMT
<
{
"requestID": "fd7ead5a-15bb-449b-ae64-c5f416186267",
"signature": null,
"results": [
],
"status": "success",
"metrics": {"elapsedTime": "1.682457959s","executionTime": "1.682413042s","resultCount": 0,"resultSize": 0,"serviceLoad": 5}
}
* Connection #0 to host localhost left intact
* Trying 127.0.0.1:8093...
* Connected to localhost (127.0.0.1) port 8093 (#0)
* Server auth using Basic with user 'Administrator'
> POST /query/service HTTP/1.1
> Host: localhost:8093
> Authorization: Basic QWRtaW5pc3RyYXRvcjpQQCR3MHJkMTI=
> User-Agent: curl/7.78.0-DEV
> Accept: */*
> Content-Length: 58
> Content-Type: application/x-www-form-urlencoded
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Length: 236
< Content-Type: application/json; version=7.1.0-N1QL
< Date: Thu, 05 May 2022 16:29:46 GMT
<
{
"requestID": "80880668-04f1-4970-815c-88e533399855",
"signature": null,
"results": [
],
"status": "success",
"metrics": {"elapsedTime": "2.67268896s","executionTime": "2.67262546s","resultCount": 0,"resultSize": 0,"serviceLoad": 5}
}
* Connection #0 to host localhost left intact
* Trying 127.0.0.1:8093...
* Connected to localhost (127.0.0.1) port 8093 (#0)
* Server auth using Basic with user 'Administrator'
> POST /query/service HTTP/1.1
> Host: localhost:8093
> Authorization: Basic QWRtaW5pc3RyYXRvcjpQQCR3MHJkMTI=
> User-Agent: curl/7.78.0-DEV
> Accept: */*
> Content-Length: 68
> Content-Type: application/x-www-form-urlencoded
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Length: 238
< Content-Type: application/json; version=7.1.0-N1QL
< Date: Thu, 05 May 2022 16:29:50 GMT
<
{
"requestID": "ad9a5337-53ac-4f02-bfda-6c1d0bda1c5a",
"signature": null,
"results": [
],
"status": "success",
"metrics": {"elapsedTime": "2.566747917s","executionTime": "2.566666001s","resultCount": 0,"resultSize": 0,"serviceLoad": 5}
}
* Connection #0 to host localhost left intact
JSON `file:///opt/couchbase/init/sample-data.json` imported to `http://localhost:8091` successfully
Documents imported: 9 Documents failed: 0
docker container logs sync-gateway
Open your web browser of choice and navigate to the following URL:
Log into the portal using the same username and password that was displayed in the top of the Couchbase Server docker logs:
Username: Administrator
Password: P@$w0rd12
From the Cluster > Dashboard page click on Buckets link on the navigation menu on the left side of the screen.
To validate the Sync Gateway server we will use the REST API interface.
{"couchdb":"Welcome","vendor":{"name":"Couchbase Sync Gateway","version":"3.0"},"version":"Couchbase Sync Gateway/3.0.0(541;46803d1) EE"}
Congratulations on completing this step of the learning path! In this section, we walked through how to use Docker Compose and Docker Desktop to build and start a container running Couchbase Server and a container runnnig Sync Gateway. Continue on to the next step to learn how to setup replication to sync information between the mobile demo app and Sync Gateway to Couchbase Server.