cbbackup

cbbackup

The cbbackup tool creates a copy of data from an entire running cluster, an entire bucket, a single node, or a single bucket on a single functioning node.

Syntax

The basic syntax is:

cbbackup [options] [backup-dir] [destination] -u [admin] -p [password]

Where:

  • [backup-dir]

    The directory for the backup files to be stored. If an empty directory doesn't exist, it will be created; the parent directory must exist.

The following syntax example includes a full backup and two incremental backups for a cluster:

cbbackup http://HOST:8091 /backup-42 -u Administrator -p password
cbbackup http://HOST:8091 /backup-42 -u Administrator -p password
cbbackup http://HOST:8091 /backup-42 -u Administrator -p password			
Note: If the backup directory exists, the following backup command will be automatically incremented. The only way to generate a full backup is to choose an unused directory. If backing up a Couchbase Server Community Edition cluster, then incremental backup is not available, all backups will be full backups.

The following syntax example include a full backup, two differential backups, and one accumulative backup for a single node.

cbbackup couchbase://HOST:8091 /backup-43 [-m full] --single-node -u Administrator -p password
cbbackup couchbase://HOST:8091 /backup-43 [-m diff] --single-node -u Administrator -p password 
cbbackup couchbase://HOST:8091 /backup-43 [-m diff] --single-node -u Administrator -p password  
cbbackup couchbase://HOST:8091 /backup-43 [-m accu] --single-node -u Administrator -p password 

Syntax example of a full backup plus two differentials and one accumulative for a single node.

cbbackup couchbase://HOST:8091 /backup-43 [-m full] --single-node -u Administrator -p password \  
cbbackup couchbase://HOST:8091 /backup-43 [-m diff] --single-nodecbbackup \
couchbase://HOST:8091 /backup-43 [-m diff] --single-node -u Administrator -p password \
cbbackup couchbase://HOST:8091 /backup-43 -m accu --single-node -u Administrator -p password]			
Note: A full backup task is always triggered for a new sink location no matter what backup mode is specified.

Description

The backup process writes a copy of data onto the disk. To create a backup using cbbackup, the node or cluster needs to be functional.

The cbbackup, cbrestore, and cbtransfer tools do not communicate with external IP addresses for server nodes outside of a cluster. Backup, restore, or transfer operations are performed on data from a node within a Couchbase Server cluster. They only communicate with nodes from a node list obtained within a cluster. This also means that if Couchbase Server is installed with a default IP address, an external hostname cannot be used to access it.

This tool has several different options which you can use to:

  • Backup all buckets in an entire cluster
  • Backup one named bucket in a cluster
  • Backup all buckets on a node in a cluster
  • Backup one named buckets on a specified node
Tip: We recommended that cbbackup output be generated on a file system local to the server. Specifically, back up to a dedicated partition. A dedicated partition prevents the backup from filling the Couchbase Server data stores and operating system partitions.
CAUTION:
Avoid routing the cbbackup output to remote share file systems (NFS). This is because cbbackup files are based on sqlite format which have issues being written to remote file systems.

By default, cbbackup backs up all global secondary indexes in a cluster. If you are using cbbackup to backup a single node in the cluster then global secondary index definitions will only be backed up if the node has the index service enabled. If you backup a single index node, then all index definitions across the cluster will be stored in the backup, not just indexes on that one node.

Note: To backup global secondary indexes, the cluster administrator username and password must be passed as parameters to cbbackup.

As with map-reduce views, cbbackup only stores the definitions of these indexes (i.e which fields are being indexed), not the contents of the index itself. These definitions are stored in the file index.json, located in the backup directory.

This tool is located in the following directories:

Operating system Location
Linux /opt/couchbase/bin/cbbackup
Windows C:\Program Files\Couchbase\Server\bin\cbbackup
Mac OS X /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/cbbackup

Options

The following are the command options:

Table 1. cbbackup options
Parameters Description
-h, --help Command line help.
-b BUCKET_SOURCE, --bucket-source=BUCKET_SOURCE Single named bucket from source cluster to transfer. If the backup directory only contains a single bucket, then that bucket is automatically used.
--single-node Use a single server node from the source only, not all server nodes from the entire cluster. This single server node is defined by the source URL.
-m MODE, --mode=MODE Backup mode. The mode option takes any one of the following values:
full
Perform a full backup
diff
Perform a differential incremental backup, which backs up only the changes since the last full or incremental backup.
accu
Perform a cumulative incremental backup, which backs up all changes since the last full backup.
-i ID, --id=ID Transfer only items that match a vbucket ID.
-k KEY, --key=KEY Transfer only items with keys that match a regular expression.
-n, --dry-run No actual transfer. Just validate parameters, files, connectivity and configurations.
-u USERNAME, --username=USERNAME REST username for source cluster or server node.
-p PASSWORD, --password=PASSWORD REST password for cluster or server node.
-s, --ssl Transfer data with SSL enablec.
-t THREADS, --threads=THREADS Number of concurrent workers threads performing the transfer.
-v, --verbose Verbose logging. More v's provide more verbosity. Max: -vvv.
--silent Reduce logging verbosity to only include errors.
-x EXTRA, --extra=EXTRA Provide extra, uncommon configuration parameters. Comma-separated key=val(key-val)* pairs.

The following are extra, specialized command options with the cbbackup -x parameter.

Table 2. cbbackup -x options
-x options Description
backoff_cap=10 Maximum backoff time during the rebalance period.
batch_max_bytes=400000 Transfer this # of bytes per batch.
batch_max_size=1000 Transfer this # of documents per batch.
cbb_max_mb=100000 Split backup file on destination cluster if it exceeds the MB.
conflict_resolve=1 By default, disable conflict resolution.

* This option doesn't work in Couchbase Server versions 4.0 and 4.1 but will be re-implemented in version 4.1.1 and in subsequent versions.

data_only=0 For value 1, transfer only data from a backup file or cluster.
design_doc_only=0 For value 1, transfer only design documents from a backup file or cluster. Default: 0.
max_retry=10 Max number of sequential retries if the transfer fails.
mcd_compatible=1 For value 0, display extended fields for stdout output.
nmv_retry=1 0 or 1, where 1 retries transfer after a NOT_MY_VBUCKET message. Default: 1.
recv_min_bytes=4096 Amount of bytes for every TCP/IP batch transferred.
rehash=0 For value 1, rehash the partition id's of each item. This is required when transferring data between clusters with different number of partitions, such as when transferring data from an Mac OS X server to a non-Mac OS X cluster.
report=5 Number batches transferred before updating progress bar in console.
report_full=2000 Number batches transferred before emitting progress information in console.
seqno=0 By default, start seqno from beginning.
try_xwm=1 Transfer documents with metadata. Default: 1. Value of 0 is only used when transferring from 1.8.x to 1.8.x.
uncompress=0 For value 1, restore data in uncompressed mode.
Note: Back up only design documents which includes view and secondary index definitions from a cluster or bucket with the option, design_doc_only=1. Restore only design documents with cbrestore design_doc_only=1.

Examples

Example 1:

An entire cluster can be backed up. This includes all of the data buckets and all design documents which includes view and secondary index definitions. To backup an entire cluster:

cbbackup http://HOST:8091 ~/backups -u Administrator -p password

Where ~/backups is the directory where you want to store the data. When this operation is performed, be aware that cbbackup creates the following directory structure and files in the ~/backups directory assuming there two buckets in the cluster named my_name and sasl and two nodes N1 and N2 :

~/backups
        bucket-my_name
            N1
            N2
            design.json
            index.json
        bucket-sasl
            N1
            N2
            design.json
            index.json
            

Where bucket-my_name and bucket-sasl are directories containing data files, view definitions, index definitions and where N1 and N2 are two sets of data files for each node in the cluster.

Example 2:

To backup a single bucket in a cluster:

cbbackup http://HOST:8091 /backups/backup-20120501 -u Administrator -p password \ 
 -b default

In this case, the default bucket in the cluster is specified and the data from the default bucket is backed up.

Example 3:

To backup all data from multiple buckets on a single node:

> cbbackup http://HOST:8091 /backups/ -u Administrator -p password \   
 --single-node

Example 4:

To backup data from a single bucket on a single node:

cbbackup http://HOST:8091 /backups -u Administrator -p password \
 --single-node  -b bucket_name

Example 5:

To specify the keys that are backed up using the - k option. For example, to backup all keys from a bucket with the prefix ‘object’:

> cbbackup http://HOST:8091 /backups/backup-20120501 -u Administrator -p password \ 
 -b bucket_name -k '^object.*'

Example 6:

The following example creates a backup copy of all design documents which includes view definitions and secondary index definitions from foo-bucket and store this as design.json in the directory ~/backup/foo-bucket. If a bucket is not specified, design documents for all buckets in the cluster are backed up.

cbbackup http://10.5.2.117:8091 ~/backup -x design_doc_only=1 -u Administrator -p password

Response

The following example response shows only design documents for all buckets being backed up. In this case, the source node had two (2) buckets including the default bucket.

transfer design doc only. bucket msgs will be skipped.
transfer design doc only. bucket msgs will be skipped.
done       

In the following output, two design documents were backed up.

[
   {
      "controllers":{
         "compact":"/pools/default/buckets/default/ddocs/_design%2Fddoc1/controller/compactView",
         "setUpdateMinChanges":"/pools/default/buckets/default/ddocs/_design%2Fddoc1/controller/setUpdateMinChanges"
      },
      "doc":{
         "json":{
            "views":{
               "view1":{
                  "map":"function(doc){emit(doc.key,doc.key_num);}"
               },
               "view2":{
                  "map":"function(doc,meta){emit(meta.id,doc.key);}"
               }
            }
         },
         "meta":{
            "rev":"1-6f9bfe0a",
            "id":"_design/ddoc1"
         }
      }
   },
   {
      "controllers":{
         "compact":"/pools/default/buckets/default/ddocs/_design%2Fddoc2/controller/compactView",
         "setUpdateMinChanges":"/pools/default/buckets/default/ddocs/_design%2Fddoc2/controller/setUpdateMinChanges"
      },
      "doc":{
         "json":{
            "views":{
               "dothis":{
                  "map":"function (doc, meta) {\n  emit(meta.id, null);\n}"
               }
            }
         },
         "meta":{
            "rev":"1-4b533871",
            "id":"_design/ddoc2"
         }
      }
   },
   {
      "controllers":{
         "compact":"/pools/default/buckets/default/ddocs/_design%2Fdev_ddoc2/controller/compactView",
         "setUpdateMinChanges":"/pools/default/buckets/default/ddocs/_design%2Fdev_ddoc2/controller/setUpdateMinChanges"
      },
      "doc":{
         "json":{
            "views":{
               "dothat":{
                  "map":"function (doc, meta) {\n  emit(meta.id, null);\n}"
               }
            }
         },
         "meta":{
            "rev":"1-a8b6f59b",
            "id":"_design/dev_ddoc2"
         }
      }
   }
]

Example 7:

The following example requests a full backup of all the data on the specified cluster:

cbbackup -m full http://example.com:8091 /backups/backup-1 -u Administrator -p password

After an initial full backup, incremental backups can be performed. This example requests a differential incremental backup of all the data on the specified cluster:

cbbackup -m diff http://example.com:8091 /backups/backup-1 -u Administrator -p password

This example requests a cumulative incremental backup of all the data on the specified cluster:

cbbackup -m accu http://example.com:8091 /backups/backup-1 -u Administrator -p password