cbbackupwrapper and cbrestorewrapper

cbbackupwrapper and cbrestorewrapper

Two new tools are included with Couchbase Server Enterprise Edition that improve throughput for the database backup process: cbbackupwrapper and cbrestorewrapper.

cbbackupwrapper and cbrestorewrapper were specifically created to address the single worker thread limitation for one backup/restore target (Bucket, Node). They should be used instead of the standard backup tools, cbbackup and cbrestore.

cbbackupwrapper and cbrestorewrapper are designed to provide a parallel processing approach to backing up a Couchbase Server cluster. At a technical level, they divide each vBucket space into a subset of vBuckets for each backup process and thus improve the speed and efficiency.

Important: In a future release of Couchbase Server, these two tools will be folded into cbbackup and cbrestore respectively and then deprecated.

Syntax

The basic syntax is:

cbbackupwrapper CLUSTER BACKUPDIR OPTIONS
cbrestorewrapper BACKUPDIR CLUSTER OPTIONS 

Options

The following are the command options:

Table 1. Basic command options for cbbackupwrapper
Option Description
-h, --help Show this help message and exit
-b BUCKET_SOURCE, --bucket-source=BUCKET_SOURCE Specify the bucket to backup. Defaults to all buckets
--single-node Use a single server node from the source only
-u USERNAME, --username=USERNAME REST username for source cluster or server node. Default is Administrator
-p PASSWORD, --password=PASSWORD REST password for source cluster or server node
-s, --ssl Transfer data with SSL enabled
-v, --verbose Enable verbose messaging
--path=PATH Specify the path to cbbackup. Defaults to current directory
--port=PORT Specify the bucket port. Defaults to 11210.
-n NUMBER, --number=NUMBER Specify the number of vbuckets per process. Defaults to 100.
-P PARALLELISM, --parallelism=PARALLELISM Number of vbucket backup jobs to run at a time. Defaults to 1.
-x EXTRA, --extra=EXTRA Provide extra, uncommon config parameters; comma-separated key=val(,key=val)* pairs.
-m MODE, --mode=MODE Backup mode: full, diff or accu [default:diff]
Table 2. Additional configuration options for cbbackupwrapper
Option Description
backoff_cap=10 Max backoff time during rebalance period
batch_max_bytes=400000 Transfer this # of bytes per batch
batch_max_size=1000 Transfer this # of documents per batch
cbb_max_mb=100000 Split backup file on destination cluster if it exceeds MB
data_only=0 For value 1, only transfer data from a backup file or cluster
dcp_consumer_queue_length=1000 A DCP client needs a queue for incoming documents/messages. A large length is more efficient, but memory proportional to length*avg. doc size. Below length 150, performance degrades significantly.
design_doc_only=0 For value 1, transfer design documents only from a backup file or cluster
flow_control=1 For value 0, disable flow control to improve throughput.
max_retry=10 Max number of sequential retries if transfer fails
recv_min_bytes=4096 Amount of bytes for every TCP/IP call transferred
rehash=0 For value 1, rehash the partition IDs of each item. Rehashing is needed when transferring data between clusters with a different number of partitions, such as when transferring data from an OS X server to a non-OS X cluster.
report=5 Number batches transferred before updating progress bar in console
report_full=2000 Number batches transferred before emitting progress information in console
seqno=0 By default, start seqno from beginning.
uncompress=0 For value 1, restore data in uncompressed mode.
Table 3. Basic command options for cbrestorewrapper
Option Description
-h, --help Show this help message and exit
-B BUCKET_DESTINATION Allows you to specify the destination bucket for a restore (assuming that the bucket in the backup is named differently).
-b BUCKET_SOURCE, --bucket-source=BUCKET_SOURCE Specify the bucket to restore. Defaults to all buckets
-u USERNAME, --username=USERNAME REST username for source cluster or server node. Default is Administrator
-p PASSWORD, --password=PASSWORD REST password for source cluster or server node
-s, --ssl Transfer data with SSL enabled
-v, --verbose Enable verbose messaging
--path=PATH Specify the path to cbrestore. Defaults to current directory
--port=PORT Specify the bucket port. Defaults to 11210.
--from-date=FROM_DATE Restore data from the date specified as yyyy-mm-dd. By default, all data from the very beginning will be restored.
--to-date=TO_DATE Restore data till the date specified as yyyy-mm-dd. By default, all data collected will be restored.
-x EXTRA, --extra=EXTRA Provide extra, uncommon config parameters; comma-separated key=val(,key=val)* pairs.
Table 4. Additional configuration options for cbrestorewrapper
Option Description
backoff_cap=10 Max backoff time during rebalance period
batch_max_bytes=400000 Transfer this # of bytes per batch
batch_max_size=1000 Transfer this # of documents per batch.
cbb_max_mb=100000 Split backup file on destination cluster if it exceeds MB.
conflict_resolve=1 By defaault, enable conflict resolution.
data_only=0 For value 1, only transfer data from a backup file or cluster.
dcp_consumer_queue_length=1000 A DCP client needs a queue for incoming documents/messages. A large length is more efficient, but memory proportional to length*avg. doc size. Below length 150, performance degrades significantly.
design_doc_only=0 For value 1, transfer design documents only from a backup file or cluster.
flow_control=1 For value 0, disable flow control to improve throughput.
max_retry=10 Max number of sequential retries if transfer fails.
recv_min_bytes=4096 Amount of bytes for every TCP/IP call transferred.
rehash=0 For value 1, rehash the partition IDs of each item; this is needed when transferring data between clusters with different number of partitions, such as when transferring data from an OS X server to a non-OS X cluster.
report=5 Number batches transferred before updating progress bar in console.
report_full=2000 Number batches transferred before emitting progress information in console.
seqno=0 By default, start seqno from beginning.
uncompress=0 For value 1, restore data in uncompressed mode.