Disk throughput and sizing

Disk throughput and sizing

Couchbase Server decouples RAM from the I/O layer to enable high scaling.

Decoupling enables high scaling at very low and consistent latencies and very high write loads without affecting the client application performance.

Couchbase Server implements an append-only format and a built-in automatic compaction process. Requirements of your disk subsystem are broken down into two components: size and I/O.

Disk size requirements

Disk size requirements are impacted by the Couchbase file write format, append-only, and the built-in automatic compaction process. Append-only format means that every write (insert/update/delete) creates a new entry in the file(s).

The required disk size increases from the update and delete workload and then shrinks as the automatic compaction process runs. The size increases because of the data expansion rather than the actual data using more disk space. Heavier update and delete workloads increases the size more dramatically than heavy insert and read workloads.

Size recommendations are available for key-value data only. If views and indexes or XDCR are implemented, contact Couchbase support for analysis and recommendations.

Depending on the workload, the required disk size is 2–3x your total dataset size (active and replica data combined).

Important: The disk size requirement of 2-3x your total dataset size applies to key-value data only and does not take into account other data formats and the use of views and indexes or XDCR.

I/O requirements

I/O is a combination of the sustained write rate, the need to compact the database files, and anything else that requires disk access. Couchbase Server automatically buffers writes to the database in RAM and eventually persists them to disk. Because of this, the software can accommodate much higher write rates than what a disk is able to handle. However, sustaining these writes eventually requires enough I/O to get it all down to disk.

To manage I/O, configure the thresholds and schedule when the compaction process starts (or doesn’t start), keeping in mind that the successful completion of compaction is critical to keeping the disk size in check. Disk size and disk I/O become critical to size correctly when using views and indexes and Cross Data Center Replication (XDCR). They are also important for backups and any other operations outside of Couchbase Server that need space or are accessing the disk.

Tip: Use the available configuration options to separate data files, indexes and the installation/configuration directories on separate drives and devices to ensure that I/O and space are allocated effectively.