Monitoring Statistics

Monitoring Statistics

Within the Data Bucket tab, information and statistics about buckets and nodes is displayed for the entire Couchbase Server cluster. The information is aggregated from all the server nodes within the configured cluster for the selected bucket.

The following functionality is available through this display, and is common to all the graphs and statistics display within the web console.

  • Bucket Selection - The Data Buckets selection list lets you select which of the buckets configured on your cluster is to be used as the basis for the graph display. The statistics shown are aggregated over the whole cluster for the selected bucket.
  • Server Selection - The Server Selection option enables you to limit the display to an individual server or entire cluster. The individual node selection displays information for the node. The all server nodes selection displays information for the entire cluster.
  • Interval Selection - The Interval Selection at the top of the main graph changes interval display for all graphs displayed on the page. For example, selecting Minute shows information for the last minute, continuously updating. As the selected interval increases, the amount of statistical data displayed will depend on how long your cluster has been running.
  • Statistic Selection - All of the graphs within the display update simultaneously. Clicking on any of the smaller graphs will promote that graph to be displayed as the main graph for the page.
  • Individual Server Selection - Clicking the blue triangle next to any of the smaller statistics graphs enables you to show the selected statistic individual for each server within the cluster, instead of aggregating the information for the entire cluster.

Individual Bucket Monitoring

Bucket monitoring within the Couchbase Web Console has been updated to show additional detailed information. The following statistic groups are available for Couchbase bucket types.

  • Summary - The summary section provides a quick overview of the cluster activity.
  • vBucket Resources - This section provides detailed information on the vBucket resources across the cluster, including the active, replica and pending operations.
  • Disk Queues - Disk queues show the activity on the backend disk storage used for persistence within a data bucket. The information displayed shows the active, replica and pending activity.
  • TAP Queues - This section provides information on the activity within the TAP queues across replication, rebalancing and client activity.
  • XDCR Destination - The XDCR Destination section show you statistical information about the Cross Datacenter Replication (XDCR), if XDCR has been configured.
  • View Stats - This section lets you monitor the statistics for each production view configured within the bucket or system.
  • Top Keys - This section shows a list of the top 10 most actively used keys within the selected data bucket.

For Memcached bucket types, the Memcached statistic summary is provided.

Bucket Monitoring — Summary Statistics

The summary section is designed to provide a quick overview of the cluster activity. Each graph (or selected graph) shows information based on the currently selected bucket.

The following graph types are available:

Statistics Description
ops per second The total number of operations per second on this bucket.
cache miss ratio Ratio of reads per second to this bucket which required a read from disk rather than RAM.
creates per second Number of new items created in this bucket per second.
updates per second Number of existing items updated in this bucket per second.
XDCR ops per sec Number of XDCR related operations per second for this bucket.
disk reads per sec Number of reads per second from disk for this bucket.
temp OOM per sec Number of temporary out of memory conditions per second.
gets per second Number of get operations per second.
sets per second Number of set operations per second.
deletes per second Number of delete operations per second.
items Number of items (documents) stored in the bucket.
disk write queue Size of the disk write queue.
docs data size Size of the stored document data.
docs total disk size Size of the persisted stored document data on disk.
doc fragmentation % Document fragmentation of persisted data as stored on disk.
XDC replication queue Size of the XDCR replication queue.
total disk size Total size of the information for this bucket as stored on disk, including persisted and view index data.
view data size Size of the view data information.
views total disk size Size of the view index information as stored on disk.
views fragmentation % Percentage of fragmentation for a given view index.
view reads per second Number of view reads per second.
memory used Amount of memory used for storing the information in this bucket.
high water mark High water mark for this bucket (based on the configured bucket RAM quota).
low water mark Low water mark for this bucket (based on the configured bucket RAM quota).
disk update time Time required to update data on disk.

Monitoring vBucket Resources

The vBucket statistics provide information for all vBucket types within the cluster across three different states. Within the statistic display the table of statistics is organized in four columns, showing the Active, Replica and Pending states for each individual statistic. The last column provides the total value for each statistic.

The Active column displays the information for vBuckets within the Active state. The Replica column displays the statistics for vBuckets within the Replica state (that is currently being replicated). The Pending columns shows statistics for vBuckets in the Pending state, that is while data is being exchanged during rebalancing.

These states are shared across all the following statistics. For example, the graph new items per sec within the Active state column displays the number of new items per second created within the vBuckets that are in the active state.

The individual statistics, one for each state, shown are:

Statistics Description
vBuckets The number of vBuckets within the specified state.
items Number of items within the vBucket of the specified state.
% resident

Percentage of items within the vBuckets of the specified state that are resident (in RAM).

new items per second Number of new items created in vBuckets within the specified state.
Note: The new items per second is not valid for the Pending state.
ejections per second Number of items ejected per second within the vBuckets of the specified state.
user data in RAM Size of user data within vBuckets of the specified state that are resident in RAM.
metadata in RAM Size of item metadata within the vBuckets of the specified state that are resident in RAM.

Monitoring Disk Queues

The Disk Queues statistics section displays the information for data being placed into the disk queue. Disk queues are used within Couchbase Server to store the information written to RAM on disk for persistence. Information is displayed for each of the disk queue states, Active, Replica, and Pending.

The Active column displays the information for the Disk Queues within the Active state. The Replica column displays the statistics for the Disk Queues within the Replica state (that is currently being replicated). The Pending columns shows statistics for the disk Queues in the Pending state, that is while data is being exchanged during rebalancing.

These states are shared across all the following statistics. For example, the graph fill rate within the Replica state column displays the number of items being put into the replica disk queue for the selected bucket.

The displayed statistics are:

Type Description
items The number of items waiting to be written to disk for this bucket for this state.
fill rate The number of items per second being added to the disk queue for the corresponding state.
drain rate The number of items actually written to disk from the disk queue for the corresponding state.
average age The average age of items (in seconds) within the disk queue for the specified state.

Monitoring TAP Queues

The TAP queues statistics are designed to show information about the TAP queue activity, both internally, between cluster nodes and clients. The statistics information is therefore organized as a table with columns showing the statistics for TAP queues used for replication, rebalancing, and clients.

The statistics in this section are detailed below:

Statistics Description
tap senders Number of TAP queues in this bucket for internal (replica), rebalancing or client connections.
items Number of items in the corresponding TAP queue for this bucket.
drain rate Number of items per second being sent over the corresponding TAP queue connections to this bucket.
back-off rate Number of back-offs per second sent when sending data through the corresponding TAP connection to this bucket.
backfill remaining Number of items in the backfill queue for the corresponding TAP connection for this bucket.
remaining on disk Number of items still on disk that need to be loaded in order to service the TAP connection to this bucket.

Memcached Buckets

For Memcached buckets, Web Console displays a separate group of statistics:

The Memcached statistics are:

Statistics Description
Operations per sec. Total operations per second serviced by this bucket.
Hit Ratio % Percentage of get requests served with data from this bucket.
Memory bytes used Total amount of RAM used by this bucket.
Items count Number of items stored in this bucket.
RAM evictions per sec. Number of items per second evicted from this bucket.
Sets per sec. Number of set operations serviced by this bucket.
Gets per sec. Number of get operations serviced by this bucket.
Net. bytes TX per sec Number of bytes per second sent from this bucket.
Net. bytes RX per sec. Number of bytes per second sent into this bucket.
Get hits per sec. Number of get operations per second for data that this bucket contains.
Delete hits per sec. Number of delete operations per second for data that this bucket contains
Incr hits per sec. Number of increment operations per second for data that this bucket contains.
Decr hits per sec. Number of decrement operations per second for data that this bucket contains.
Delete misses per sec.

Number of delete operations per second for data that this bucket does not contain.

Decr misses per sec. Number of decrement operations per second for data that this bucket does not contain.
Get misses per sec. Number of get operations per second for data that this bucket does not contain.
Incr misses per sec.

Number of increment operations per second for data that this bucket does not contain.

CAS hits per sec. Number of CAS operations per second for data that this bucket contains.
CAS badval per sec. Number of CAS operations per second using an incorrect CAS ID for data that this bucket contains.
CAS misses per sec. Number of CAS operations per second for data that this bucket does not contain.

Monitoring Outgoing XDCR

The Outgoing XDCR shows the XDCR operations that are supporting cross datacenter replication from the current cluster to a destination cluster.

You can monitor the current status for all active replications in the Ongoing Replications section under the XDCR tab:

The Ongoing Replications section shows the following information:

Column Name Description
Bucket The source bucket on the current cluster that is being replicated.
From Source cluster name.
To Destination cluster name.
Status Current status of replications.
When Indicates when replication occurs.

The Status column indicates the current state of the replication configuration. The status options include:

  • Starting Up - The replication process has just started, and the clusters are determining what data needs to be sent from the originating cluster to the destination cluster.
  • Replicating - The bucket is currently being replicated and changes to the data stored on the originating cluster are being sent to the destination cluster.
  • Failed - Replication to the destination cluster has failed. The destination cluster cannot be reached. The replication configuration may need to be deleted and recreated.

Under the Data Buckets tab, you can click on a named Couchbase bucket and find more statistics about replication for that bucket. Couchbase Web Console displays statistics for the particular bucket; on this page you can find two drop-down areas: Outgoing XDCR and Incoming XDCR Operations. Both provides statistics about ongoing replication for the particular bucket. Under theOutgoing XDCR panel if you have multiple replication streams you will see statistics for each stream.

outbound xdcr stats 2.2
The statistics shown are:
Statistics Description
outbound XDCR mutation Number of changes in the queue waiting to be sent to the destination cluster.
mutations checked Number of document mutations checked on source cluster.
mutations replicated Number of document mutations replicated to the destination cluster.
data replicated Size of data replicated in bytes.
active vb reps Number of parallel, active vBucket replicators. Each vBucket has one replicator which can be active or waiting. By default you can only have 32 parallel active replicators at once per node. Once an active replicator finishes, it will pass a token to a waiting replicator.
waiting vb reps Number of vBucket replicators that are waiting for a token to replicate.
secs in replicating

Total seconds elapsed for data replication for all vBuckets in a cluster.

secs in checkpointing Time working in seconds including wait time for replication.
checkpoints issued Total number of checkpoints issued in replication queue. By default active vBucket replicators issue a checkpoint every 30 minutes to keep track of replication progress.
checkpoints failed Number of checkpoints failed during replication. This can happen due to timeouts, due to network issues or if a destination cluster cannot persist quickly enough.
mutations in queue Number of document mutations waiting in replication queue.
XDCR queue size Amount of memory used by mutations waiting in replication queue. In bytes.
mutation replication rate Number of mutations replicated to destination cluster per second.
data replication rate Bytes replicated to destination per second.
ms meta ops latency Weighted average time for requesting document metadata in milliseconds.
mutations replicated optimistically Total number of mutations replicated with optimistic XDCR.
ms docs ops latency Weighted average time for sending mutations to destination cluster in milliseconds.
percent completed Percent of total mutations checked for metadata.

Be aware that if you use an earlier version of Couchbase Server, such as Couchbase Server 2.0, only the first three statistics appear and have the labels changes queue, documents checked, and documents replicated respectively. You can also get XDCR statistics using the Couchbase REST API. All of the statistics in Web Console are based on statistics via the REST API or values derived from them.

Monitoring Incoming XDCR

The Incoming XDCR Operations section shows the XDCR operations that are coming into to the current cluster from a remote cluster.

The statistics shown are:
Statistics Description
metadata reads per sec. Number of documents XDCR scans for metadata per second. XDCR uses this information for conflict resolution.
sets per sec. Set operations per second for incoming XDCR data.
deletes per sec. Delete operations per second as a result of the incoming XDCR data stream.
total ops per sec. Total of all the operations per second.

Monitoring View Statistics

The View Stats show information about individual design documents within the selected bucket. One block of stats will be shown for each production-level design document.

The statistics shown are:
Statistics Description
data size Size of the data required for this design document.
disk size Size of the stored index as stored on disk.
view reads per sec.

Number of read operations per second for this view.