cbtransfer tool

cbtransfer tool

The cbtransfer tool is used to transfer data between clusters and to/from files.


In addition to transferring data between clusters and to/from files, the cbtransfer tool can also be used to create a copy of data from a node that is no longer running. This tool is the underlying, generic data transfer tool upon which cbbackup and cbrestore are built. It is a lightweight extract-transform-load (ETL) tool that can move data from a source to a destination. The source and destination parameters are similar to URLs or file paths.

Note: The most important way to use this tool is for transferring data from a Couchbase node that is no longer running to a cluster that is running. The cbbackup , cbrestore , and cbtransfer tools do not communicate with external IP addresses for server nodes outside of a cluster. Backup, restore, or transfer operations are performed on data from a node within a Couchbase cluster. They only communicate with nodes from a node list obtained within a cluster. If Couchbase Server is installed with a default IP address, an external hostname cannot be used to access it.
Note: Couchbase Server does not transfer design documents. To back up a design document, use cbbackup to store the information and cbrestore to read it back into memory.

The tool is at the following location:

Operating system Location
Linux /opt/couchbase/bin/
Windows C:\Program Files\Couchbase\Server\bin\
Mac OS X /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/

CLI command and parameters

Basic syntax for this command:

cbtransfer [options] source destination

The following are the command options:

Table 1. cbtransfer options
Parameters Description
-h, --help Command line HELP.
-b BUCKET_SOURCE Single named bucket from a source cluster to transfer.
-B BUCKET_DESTINATION , --bucket-destination=BUCKET_DESTINATION Single named bucket on the destination cluster that receives the transfer. You can transfer to a bucket with a different name than your source bucket; if you do not provide a name, it defaults to the same name as the bucket-source.
-i ID , --id=ID Transfer only items that match the vbucket ID.
-k KEY , --key=KEY Transfer only items with keys that match the regexp . 1
-n , --dry-run No actual transfer occurs, just the validation of parameters, files, connectivity, and configurations.
-u USERNAME , --username=USERNAME REST username for the source cluster or the server node.
-p PASSWORD , --password=PASSWORD REST password for the cluster or the server node.
-t THREADS , --threads=THREADS Number of concurrent worker threads performing the transfer. Memcached uses 75% of the number of cores reported of the system (with a minimum of 4 cores) 2 .
-v , --verbose Verbose logging; provide more verbosity.
-x EXTRA , --extra=EXTRA Provide extra, uncommon configuration parameters.
--single-node Transfer from a single server node in a source cluster. This single server node is a source node URL.
--source-vbucket-state=SOURCE_VBUCKET_STATE Only transfer from the source vbuckets occurs in this state, such as active (default) or replica . Must be used with the Couchbase cluster as a source.
--destination-vbucket-state=DESTINATION_VBUCKET_STATE Only transfer to destination vbuckets in this state, such as active (default) or replica . Must be used with Couchbase cluster as the destination.
--destination-operation=DESTINATION_OPERATION Perform this operation on transfer. set will override an existing document; add will not override; get will load all keys transferred from a source cluster into the caching layer at the destination.
Important: By default, the cbtransfer tool will use set and override any existing documents.
/path/to/filename Export a .csv file from the server or import a .csv file to the server.

1 : Regex (Reglar Expression Syntax) specifies a set of strings that matches it; the functions in this module let you check if a particular string matches a given regular expression (or if a given regular expression matches a particular string, which comes down to the same thing). For a complete reference, see https://docs.python.org/2/library/re.html .

2 : To tune memcached to adjust dynamically the number of worker threads, do one of the following:
  • Export MEMCACHED_NUM_CPUS=number of threads you want before starting Couchbase Server.
  • Use the -t <number > command line argument.
  • Specify it in the configuration file that is read during startup. Be aware that when started from the full server this file is regenerated every time, and you will loose the modifications.

The following are extra, specialized command options with the cbtransfer -x parameter.

Table 2. cbtransfer -x options
-x options Description
backoff_cap=10 Maximum backoff time during the rebalance period.
batch_max_bytes=400000 Transfer this # of bytes per batch.
batch_max_size=1000 Transfer this # of documents per batch.
cbb_max_mb=100000 Split backup file on destination cluster if it exceeds the MB.
conflict_resolve=1 By default, disable conflict resolution.
data_only=0 For value 1, transfer only data from a backup file or cluster.
design_doc_only=0 For value 1, transfer only design documents from a backup file or cluster. Default: 0.
max_retry=10 Max number of sequential retries if the transfer fails.
mcd_compatible=1 For value 0, display extended fields for stdout output.
nmv_retry=1 0 or 1, where 1 retries transfer after a NOT_MY_VBUCKET message. Default: 1.
recv_min_bytes=4096 Amount of bytes for every TCP/IP batch transferred.
rehash=0 For value 1, rehash the partition IDs of each item. This is required when transferring data between clusters with different number of partitions, such as when transferring data from a Mac OSX server to a non-Mac OSX cluster.
report=5 Number of batches transferred before updating progress bar in the console.
report_full=2000 Number of batches transferred before emitting progress information in the console.
seqno=0 By default, start seqno from beginning.
try_xwm=1 Transfer documents with metadata. Default: 1. Value of 0 is only used when transferring from 1.8.x to 1.8.x.
uncompress=0 For value 1, restore data in the uncompressed mode.


The following is the basic syntax:

cbtransfer [options] source destination

The following are syntax examples:

cbtransfer http://SOURCE:8091 /backups/backup-42
cbtransfer /backups/backup-42 http://DEST:8091
cbtransfer /backups/backup-42 couchbase://DEST:8091
cbtransfer http://SOURCE:8091 http://DEST:8091
cbtransfer file.csv http://DEST:8091

Example: Transferring data to a cluster

The following example and response transfers data from a non-running node to a running cluster:



The response shows 10000 total documents transferred in batch size of 1088 documents each.

[####################] 100.0% (10000/10000 msgs)
bucket: bucket_name, msgs transferred...
      : total | last | per sec
batch : 1088 | 1088 | 554.8
byte : 5783385 | 5783385 | 3502156.4
msg : 10000 | 10000 | 5230.9

Example: Transferring data to standard output

The following example and response sends all the data from a node to standard output:

cbtransfer stdout:

set pymc40 0 0 10
set pymc16 0 0 10
set pymc9 0 0 10
set pymc53 0 0 10
set pymc34 0 0 10

Example: Exporting and importing CSV files

The cbtransfer tool is also used to import and export csv files. Data is imported into Couchbase Server as documents and documents are exported from the server into comma-separated values. Design documents associated with vBuckets are not included.

In these examples, the following records are in the default bucket where re-fdeea652a89ec3e9 is the document ID, 0 are flags, 0 is the expiration, and the CAS value is 4271152681275955. The actual value is the hash starting with "{""key"".......

 ""name"":""fdee c3e"",
 ""achievements"":[77, 149, 239, 37, 76],""body"":""xc4ca4238a0b923820d

This example exports these items to a .csv file. All items are transferred from the default bucket, -b default available at the node http://localhost:8091 and put into the /data.csv file. If a different bucket is provided for the -b option, all items are exported from that bucket. Credentials are required for the cluster when exporting items from a bucket in the cluster.

cbtransfer http://[localhost]:8091 csv:./data.csv -b default -u Administrator -p password

The following example response is similar to that in other cbtransfer scenarios:

[####################] 100.0% (10000/10000 msgs)
bucket: default, msgs transferred...
       : total | last | per sec
 batch : 1053 | 1053 | 550.8
 byte : 4783385 | 4783385 | 2502156.4
 msg : 10000 | 10000 | 5230.9
2013-05-08 23:26:45,107: mt warning: cannot save bucket design on a CSV destination

The following example syntax shows 1053 batches of data transferred at 550.8 batches per second. The tool outputs “cannot save bucket design….” to indicate that no design documents were exported. To import information from a.csv file to a named bucket in a cluster:

cbtransfer /data.csv http://[hostname]:[port] -B bucket_name -u Administrator -p password

If the .csv file is not correctly formatted, the following error displays during import:

w0 error: fails to read from csv file, .....