Disk write queue

Disk write queue

Disk writing is implemented as a 2-queue system: commit to DRAM and then queued to be written to disk

Couchbase Server is a persistent database which means that part of monitoring the system is understanding how we interact with the disk subsystem.

Since Couchbase Server is an asynchronous system, any mutation operation is committed first to DRAM and then queued to be written to disk. The client is returned an acknowledgment almost immediately so that it can continue working. There is replication involved here too, but we’re ignoring it for the purposes of this discussion.

Disk writing is implemented as a 2-queue system and are tracked by the stats. The first queue is where mutations are immediately placed. Whenever there are items in that queue, our “flusher” (disk writer) comes along and takes all the items off of that queue, places them into the other one and begins writing to disk. Since disk performance is so dramatically different from RAM, new writes can be continuously accepted while the system is (possibly slowly) writing new ones to the disk.

The flusher will process 250k items at a time, then perform a disk commit and continue this cycle until its queue is drained. When it has completed everything in its queue, it will either grab the next group from the first queue or essentially sleep until there are more items to write.

Monitoring the disk write queue

There are basically two ways to monitor the disk queue, at a high-level from the Web UI or at a low-level from the individual node statistics.

  1. From the Web UI, click on Monitor Data Buckets and select the particular bucket that you want to monitor.
  2. Click “Configure View” in the top right corner and select the “Disk Write Queue” statistic. Closing this window shows that there is a new mini-graph.

This graph is showing the Disk Write Queue for all nodes in the cluster. To get a deeper view into this statistic, monitor each node individually using the ‘stats’ output. The statistics to watch are ep_queue_size (where new mutations are placed) and flusher_todo (the queue of items currently being written to disk).