Best Practices for Virtualized Platforms

Best Practices for Virtualized Platforms

Virtualized platforms such as AWS, VMWare, Azure, and Docker containers are popular ways of achieving hardware scalability to complement Couchbase Server's scalability. In this section, we describe some considerations when running Couchbase Server on a virtualized platform. These considerations are common across any virtualized platform or container infrastructure, and are not specific to Couchbase Server.

Avoid a Single Point of Failure

Couchbase Server's resilience and high-availability are achieved through creating a cluster of independent nodes and replicating data between them so that any individual node failure doesn't lead to loss of access to your data. In a virtualized environment, if you were to run multiple nodes on the same host or physical hardware, you are inadvertently re-introducing a single point of failure. In environments where you control the VM placement, we recommend that each Couchbase Server node runs on a different piece of physical hardware.

Sizing your Virtualized Platform

Physical environments tend to provide greater performance, predictability and simplicity over virtualized environments. Virtualization carries much greater flexibility but does add an additional layer of complexity which can make both sizing correctly and debugging performance issues more difficult. We recommend that you have at least 4 cores and, if these are virtualized, make sure these are dedicated to the VM rather than shared across multiple VMs. For more information on the hardware requirements, see Hardware Requirements.

Disk performance is also an important aspect - and there are many different technologies you can choose. The general rule of thumb is to make sure that your disk has sufficient throughput to handle the CRUD operations as well as any indexing, compaction and backup activities that will be required.

Avoid Over-committing your Resource

One of the major benefits of virtualization is the ability to share physical resources and even over-commit on these resources (meaning each VM can think it has more disk/RAM/CPU than is actually physically available). One side effect of this is that an individual VM's performance can vary based on what load other VMs are placing upon the physical resources. If your application requires consistent performance, ensure that you guard against over-committing when configuring your VMs. We recommend ensuring your Couchbase Server VMs have dedicated physical resource and the hardware resource is not over-committed.

AutoFailover Threshold

Due to the fact the system is running virtualized on top of a hypervisor or Docker engine, there will be a minor CPU performance overhead. In certain circumstances this can mean that our default auto-failover timeout can be a little too sensitive. We recommend increasing the threshold from 30 to 45 or even 60 seconds. depending how CPU-intensive your workload is.

Live Migration

Some virtualization environments allow VMs to be migrated between physical nodes and/or between storage backends. When moving the VM there is the potential for small pauses and network disruption which may affect the scheduler and could trigger a failover. When changing the backend storage, disk queues, compaction or indexing may be affected, which could have a knock-on effect on general performance. We recommend that you use Couchbase's built-in rebalance mechanism for maintenance. If it is absolutely necessary to perform a migration, then disable auto-failover beforehand and be prepared for a performance impact during the migration. Pausing / Resuming and snapshotting virtual machines can also have the same effect, these actions should only be performed on a Couchbase Server node which has been removed from the cluster.

Additional Considerations for Containers

This section lists the additional considerations that are applicable only to containers.

Map Couchbase Node Specific Data to a Local Folder

A Couchbase Server Docker container writes all persistent and node-specific data to the directory /opt/couchbase/var by default. It is recommended to map this directory to a directory on the host file system using the -v option to get persistence and performance when using "docker run".

By mapping the directory /opt/couchbase/var to a directory outside the container with the -v option, you can delete the container and recreate it later without losing your data from Couchbase Server. You can even update to a container running a later release/version of Couchbase Server without losing your data.

In a standard Docker environment using a union file system, leaving /opt/couchbase/var inside the container results in some amount of performance degradation.

Note: If you have SELinux enabled, mounting the host volumes in a container requires an extra step. Assuming you are mounting the ~/couchbase directory on the host file system, you need to run the following command once before running your first container on that host:
mkdir ~/couchbase && chcon -Rt svirt_sandbox_file_t ~/couchbase

Increase ULIMIT in Production Deployments

Couchbase Server normally expects the following changes to ulimits:

ulimit -n 40960        # nofile: max number of open files
ulimit -c unlimited    # core: max core file size
ulimit -l unlimited    # memlock: maximum locked-in-memory address space

These ulimit settings are necessary when running under heavy load. If you are just doing light testing and development, you can omit these settings, and everything will still work. To set the ulimits in your container, you need to run Couchbase Docker containers with the following additional --ulimit flags:

docker run -d --ulimit nofile=40960:40960 
--ulimit core=100000000:100000000 --ulimit memlock=100000000:100000000 
--name db -p 8091-8094:8091-8094 -p 11210:11210 couchbase

Since unlimited is not supported as a value, it sets the core and memlock values to 100 GB. If your system has more than 100 GB RAM, increase this value to match the available RAM on the system.

Note: The --ulimit flags only work on Docker 1.6 or later.