X.509 for TLS

X.509 for TLS

Couchbase Server uses X.509 certificates to encrypt Cross Data Center Replication (XDCR) and other client-server communications.

Couchbase Server Enterprise Edition supports X.509 certificates, for Transport Layer Security (TLS).

With X.509 certificates, Couchbase Server can strongly encrypt Cross Data Center Replication and other client-server communications.

Full administrators can manage certificates, using the Couchbase CLI tools (as described in ssl-manage) and REST API (as described in Security API).

CA-based certificates can be managed and rotated without client downtime.

The following overview of Couchbase certificate-management assumes the reader's prior knowledge of TLS/SSL, PKI certificates including X.509 certificates, and Certificate Authorities (CAs).

When to Use X.509 Certificates

An X.509 certificate does more than just distribute the public key: it is signed by a trusted (internal or third-party) CA, and thereby verifies the identity of the server, assuring clients that their information is not being sent to a rogue server.

Therefore, scenarios potentially requiring the use of X.509 certificates include:

  • In production, where clients have to go through the internet.
  • When transferring sensitive data on the wire between application and Couchbase Server, or between data centers (XDCR).
  • When mandated by compliance regulations.

CA Hierarchies Supported by Couchbase

Couchbase customers can associate Couchbase Server with their CA hierarchies. The CA at the top of a hierarchy is called the root authority, or root CA. Two types of CA hierarchy are supported by Couchbase: single-tier and n-tier.

Single-Tier Hierarchy
In its simplest form, the single-tier hierarchy starts with a root CA.

In the case represented by the figure immediately above, the root CA is also the issuing CA. All certificates immediately below the root certificate inherit its trustworthiness, and can be used to secure systems.

This is the simplest form of CA hierarchy: however, most enterprises use the more complex, N-tier CA hierarchy, as described next.

N-Tier Hierarchy
In many production deployments, a hierarchy has multiple CAs.

In a multi-tier hierarchy, the root CA issues certificates to the intermediate CAs, which in turn generate intermediate certificates: these are used to sign client certificates, such as a cluster certificate:
  • Trusted root CA > Intermediate CA > Cluster certificate
  • Trusted root CA > Intermediate CA 1 > Intermediate CA 2.... > Intermediate CA n > Cluster certificate

When you need to regenerate the intermediate certificate, ensure that the chain can be verified up to the root CA.

All intermediate certificates should be installed on your server: otherwise, some clients will assume that the connection is not secure. This results in 'untrusted' warnings like the following:

To avoid such warnings, a server should always provide a complete trust chain. The trust chain contains your certificate, concatenated with all intermediate certificates.

Configuring X.509

This section explains how to configure X.509 certificates for TLS in Couchbase Server. Note that choosing a root CA, the CA hierarchy, and obtaining a certificate from that CA chain to set up a Couchbase cluster are not within the scope of this document.

X.509 Certificate Requirements and Best Practices

Here are the basic requirements for using X.509 certificates in Couchbase:

  • The certificate must be in available in the .pem format.
  • The certificate must be an RSA key certificate.
  • The current system time must fall between the times set in the certificate's properties valid from and valid to.
  • Common name: This can be a certificate with a nodename (preferable), IP address, URI (www.example.com), or URI with a subject alternative name (SAN) certificate (example.com and example.net).
  • The node certificate must be designated for server authentication, by setting the optional field of the certificate's property enhanced key usage to Server Authentication.

Recommended, best practices include:

  • To avoid man-in-the-middle attacks, do not use wildcards with IP addresses in the certificate common name.
  • Use an RSA key-length of 2048 bits or higher. (As computing capabilities increase, longer RSA keys provide increased security.)

The certificate chain must be valid from the node certificate up to the root certificate: this can be verified using the OpenSSL validate certificate test.

Validating Server Identity

The HTTPS specification mandates that HTTPS clients must be capable of verifying the identity of the server. This requirement can potentially affect how you generate your X.509 certificates. The HTTPS specification defines a generic mechanism for verifying the server identity, known as the HTTPS URL integrity check, which is the standard mechanism used by Web browsers.

HTTPS URL integrity check

The basic idea of the URL integrity check is that the server certificate's identity must match the server hostname. This integrity check has an important impact on how you generate X.509 certificates for HTTPS: the certificate identity (usually the certificate subject DN’s common name) must match the name of the host on which Couchbase Server is deployed.

The URL integrity check is designed to prevent man-in-the-middle attacks.

Specify the certificate identity for the URL integrity check in one of the following ways:

Using the commonName
The usual way to specify the certificate identity (for the purpose of the URL integrity check) is through the Common Name (CN) in the subject DN of the certificate.
Using the subjectAltName
If you deploy a certificate on a multi-homed host, however, it may be practical to allow the certificate to be used with any of the multi-homed host names. In this case, it is necessary to define a certificate with multiple, alternative identities, and this is only possible using the subjectAltName certificate-extension.
The HTTPS protocol also supports, in host names, the wildcard character *. For example, you can define the subjectAltName as follows:

This certificate identity matches any three-component host name in the domain couchbase.com.

Note: As a best practice, try to avoid using the wildcard character in the domain name. Be sure never to do this accidentally by forgetting to type the dot (.) delimiter in front of the domain name. For example, if you specified *couchbase.com, your certificate could be used in any domain that ends with the string couchbase.

Couchbase Cluster Certificate

The Couchbase cluster certificate is the root CA's public key ca.pem. In the configuration steps shown in the following sections, ca.pem is the CA public key that should be configured in Couchbase as the cluster certificate.

When you load the cluster certificate into Couchbase, it is first checked to make sure it is a valid X.509 certificate. Next, if the per-node certificates are not signed by the cluster certificate, a warning is shown for each node during configuration. As the per-node certificates are updated, such that they are signed by the cluster certificate, the warning for each node goes away.

Per Node Certificate

The Couchbase cluster certificate is used to sign per-node Couchbase certificates, each containing the following:

  • The node private key, which is named pkey.key as shown in the configuration steps below.
  • The node public key certificate file, which is named pkey.pem as shown in the configuration steps below.
  • The certificate chain file based on the supported CA hierarchy, This file is named chain.pem as shown in the configuration steps below.
Table 1. Private and public keys you need to have
  Key name Description
Server-side files ca.pem Root CA public key or the cluster certificate.
int.pem Intermediate public key. There can be one or more intermediate public keys in the hierarchy.
pkey.key Node private key per node (private key of the node). Each node in the cluster must have its private key.
pkey.pem Node public key (public key of the node). Each node in the cluster must have its public key.
chain.pem Concatenated chain file (chain file). This file contains the node public key and the intermediate public keys that signed first the node key (pkey.pem) and then each other. This file does not contain the CA public key.
Client-side files ca.pem CA public key, which should be configured on the client
chain.pem Concatenated chain file (chain file)

Prepare for Configuration

Note: If your CA authority supports automatic generation of certificates, you can skip the X.509 configuration steps.

Before you start configuring X.509 certificates for your nodes, assess your needs.

Where will you put the configured CA, intermediate, and node keys?
All the keys and certificates are generated in a directory named SSLCA, which can be located anywhere on your machine.
The generated private node key ( pkey.key) and chain certificate ( chain.pem) must be posted in a specific place that is in the certificate trust path (such as /Users/<username>/Library/Application\ Support/Couchbase/var/lib/couchbase/inbox/ on MacOSX, or /opt/couchbase/var/lib/couchbase/inbox/) on Linux.
Do you have one or more nodes in the cluster?
  • With one node, you will generate one node directory inside the directory SSLCA that will contain the private node key (pkey.key) and the certificate chain file (chain.pem). The node public key (pkey.pem) is included in the chain file.
  • With multiple nodes, you need to add an appropriate number of node directories with distinctive names, such as node-sales, node-hr, or whatever your situation requires.
Do you have one or more intermediate CAs in your trust path?
With only one CA, create one directory named int. If you have multiple intermediate CAs, be sure to name them in a way that will allow you to stack them properly in the chain file, such as int1, int2, and so on.
This order will show that the intermediate CA closest to the node (which signed the node certificate) has the higher number, or in the sample below int2.

For example:

Configure X.509 Certificates

The following steps show how to configure X.509 certificates on Ubuntu 16: a root certificate is created with a single intermediate certificate and a single node certificate; and a chain-certificate is created from the intermediate and node certificates. The chain certificate and node private key are then made active for the current Couchbase Server-node.

Proceed as follows, using the sudo command where appropriate.

  1. Create environment variables for the naming of a directory-structure, within which will reside the certificates you create for root, intermediate, and node.
    export TOPDIR=SSLCA
    export ROOT_DIR=rootdir
    export NODE_DIR=nodedir
    export INT_DIR=intdir

    Note that in cases where multiple intermediate and/or node certificates are to be included in the certificate-chain, multiple intermediate and/or directories are required — one for each intermediate or certificate.

  2. Create environment variables for each of the certificate-files to be created.
    export ROOT_CA=ca
    export INTERMEDIATE=int
    export NODE=pkey
    export CHAIN=chain

    Note that in cases where multiple intermediate and/or node certificates are to be included in the certificate-chain, additional environment-variable definitions — one for each of the additional intermediate and/or node certificates — are required.

  3. Create environment variables for the administrator-credentials to be used for certificate-management, the IP address at which the Couchbase Server-node is located, and the username required for role-based access to a particular resource.
    export ADMINCRED=Administrator:password
    export ip=
    export USERNAME=travel-sample

    Note that in this example, the username is specified as travel-sample, which is typically associated with the Bucket Full Access role, on the bucket travel-sample. For access to be fully tested, ensure that the travel-sample user has indeed been defined on the Couchbase Server-node, and is associated with the Bucket Full Access role. (See Authorization for more information on RBAC.)

  4. Create a directory-structure in which, within a top-level directory named SSLCA, three subdirectories reside — rootdir, intdir, and nodedir — respectively to hold the certificates you create for root, intermediate, and node.
    mkdir ${TOPDIR}
    cd ${TOPDIR}
    mkdir ${ROOT_DIR}
    mkdir ${INT_DIR}
    mkdir ${NODE_DIR}

  5. Generate the root private key file (ca.key) and the public key file (ca.pem):
    cd ${ROOT_DIR}
    openssl genrsa -out ${ROOT_CA}.key 2048
    openssl req -new -x509 -days 3650 -sha256 -key ${ROOT_CA}.key \
    -out ${ROOT_CA}.pem -subj '/C=UA/O=MyCompany/CN=MyCompanyRootCA'

  6. Generate, first, the intermediate private key (int.key); and secondly, the intermediate certificate signing-request (int.csr):
    cd ../${INT_DIR}
    openssl genrsa -out ${INTERMEDIATE}.key 2048
    openssl req -new -key ${INTERMEDIATE}.key -out ${INTERMEDIATE}.csr \
    -subj '/C=UA/O=MyCompany/CN=MyCompanyIntermediateCA'

  7. Create the extension file v3_ca.ext; in order to add extensions to the certificate, and to generate the certificate signing-request:
    cat <<EOF>> ./v3_ca.ext
    subjectKeyIdentifier = hash
    authorityKeyIdentifier = keyid:always,issuer:always
    basicConstraints = CA:true

  8. Generate the intermediate public key (int.pem), based on the intermediate certificate signing-request (int.csr), and signed by the root public key (ca.pem).

    openssl x509 -req -in ${INTERMEDIATE}.csr \
    -CA ../${ROOT_DIR}/${ROOT_CA}.pem -CAkey ../${ROOT_DIR}/${ROOT_CA}.key \
    -CAcreateserial -CAserial ../${ROOT_DIR}/rootCA.srl -extfile ./v3_ca.ext \
    -out ${INTERMEDIATE}.pem -days 365

  9. Generate, first, the node private key (pkey.key); secondly, the node certificate signing-request (pkey.csr); and thirdly, the node public key (pkey.pem).
    cd ../${NODE_DIR}
    openssl genrsa -out ${NODE}.key 2048  
    openssl req -new -key ${NODE}.key -out ${NODE}.csr \
    -subj "/C=UA/O=MyCompany/CN=${USERNAME}"
    openssl x509 -req -in ${NODE}.csr -CA ../${INT_DIR}/${INTERMEDIATE}.pem \
    -CAkey ../${INT_DIR}/${INTERMEDIATE}.key -CAcreateserial \
    -CAserial ../${INT_DIR}/intermediateCA.srl -out ${NODE}.pem -days 365
  10. Generate the certificate chain-file, by concatenating the node and intermediate certificates. This allows the client to verify the intermediate certificate against the root certificate.
    cd ..
    cat ./${NODE_DIR}/${NODE}.pem ./${INT_DIR}/${INTERMEDIATE}.pem > ${CHAIN}.pem

    Note that if multiple intermediate certificates are specified for concatenation in this way, the concatenation-order must correspond to the order of signing. Thus, the node certificate, which appears in the first position, has been signed by the intermediate certificate, which therefore appears in the second position: and in cases where this intermediate certificate has itself been signed by a second intermediate certificate, the second intermediate certificate must appear in the third position, and so on.

    Note also that the root certificate is never included in the chain.

  11. Manually copy the node private key (pkey.key) and the chain file (chain.pem) to the inbox folder of the Couchbase Server-node:
    mkdir /opt/couchbase/var/lib/couchbase/inbox/
    cp ./${CHAIN}.pem /opt/couchbase/var/lib/couchbase/inbox/${CHAIN}.pem
    chmod a+x /opt/couchbase/var/lib/couchbase/inbox/${CHAIN}.pem
    cp ./${NODE_DIR}/${NODE}.key /opt/couchbase/var/lib/couchbase/inbox/${NODE}.key
    chmod a+x /opt/couchbase/var/lib/couchbase/inbox/${NODE}.key
  12. Upload the root certificate, and activate it:
    curl -X POST --data-binary "@./${ROOT_DIR}/${ROOT_CA}.pem" \
    curl -X POST http://${ADMINCRED}@${ip}:8091/node/controller/reloadCertificate

    Note that alternatively, the following command-line interfaces can be used:

    couchbase-cli ssl-manage -c ${ip}:8091:8091 -u Admnistrator -p password \
    couchbase-cli ssl-manage -c ${ip}:8091 -u Admnistrator -p password \
  13. For the current Couchbase Server-node, enable the client certificate:
    curl -X POST --data-binary "state=enable" \
    curl -X POST --data-binary "delimiter=" \
    curl -X POST --data-binary "path=subject.cn" \
    curl -X POST --data-binary "prefix=" \

    For further information on certificate-deployment, see ssl-manage and Encryption On-the-Wire API.

Provide Certificate-Based Authentication for a Java Client

Once the root certificate for a Couchbase Server-node has been deployed, a Java client can authenticate by means of an appropriately prepared keystore.

For an overview, see Certificate-Based Authentication

Proceed as follows. Note that these instructions assume use of the Ubuntu 16 environment configured in the preceding section, Configure X.509 Certificates.

  1. Define environment variables for the name of the keystore to be created, and its password.
    export KEYSTORE_FILE=my.keystore
    export STOREPASS=storepass

  2. If necessary, install a package containing the keytool utility:
    sudo apt install openjdk-9-jre-headless

  3. Within the top-level, SSLCA directory that you created, generate the keystore:
    keytool -genkey -keyalg RSA -alias selfsigned \
    -keystore ${KEYSTORE_FILE} -storepass ${STOREPASS} -validity 360 -keysize 2048 \
    -noprompt  -dname "CN=${USERNAME}, OU=None, O=None, L=None, S=None, C=US" -keypass ${STOREPASS}

  4. Generate the certificate signing-request:
    keytool -certreq -alias selfsigned -keyalg RSA -file my.csr \
    -keystore ${KEYSTORE_FILE} -storepass ${STOREPASS} -noprompt

  5. Generate the client certificate, signing it with the intermediate private key:
    openssl x509 -req -in my.csr -CA ./${INT_DIR}/${INTERMEDIATE}.pem \
    -CAkey ./${INT_DIR}/${INTERMEDIATE}.key -CAcreateserial -out clientcert.pem -days 365

  6. Add the root certificate to the keystore:
    keytool -import -trustcacerts -file ./${ROOT_DIR}/${ROOT_CA}.pem \
    -alias root -keystore ${KEYSTORE_FILE} -storepass ${STOREPASS} -noprompt

  7. Add the intermediate certificate to the keystore:
    keytool -import -trustcacerts -file ./${INT_DIR}/${INTERMEDIATE}.pem \
    -alias int -keystore ${KEYSTORE_FILE} -storepass ${STOREPASS} -noprompt

  8. Add the client certificate to the keystore:
    keytool -import -keystore ${KEYSTORE_FILE} -file clientcert.pem \
    -alias selfsigned -storepass ${STOREPASS} -noprompt

This concludes preparation of the Java client's keystore. Copy the file (in this case, my.keystore) to a location on a local filesystem from which the Java client can access it.

Rotating X.509

Certificate rotation is needed when:

  • A certificate expires.
  • You move from an old CA authority to a new.
  • There is a change in the policy of the certificates issued by the CA.
  • A widespread breach of security has occurred in your system.

Certificate-renewal should be planned well before a certificate expires. X.509 certificate-rotation in Couchbase is an online operation that does not require a node or cluster restart: applications maintain continued access to Couchbase Server, experiencing no downtime due to the rotation operation.

How to Rotate a Couchbase Server X.509 Certificate

  1. Generate a new certificate.

    Before you rotate a certificate, you need to generate a new certificate.

    Typically, your Certificate Authority will give you a self-service option to re-issue certificates. If this is not the case, you can manually regenerate a new X509 certificate.

    1. Renew the root CA certificate

      The root certificate authority is the topmost CA in a CA hierarchy. Its validity period is typically the longest in the hierarchy: between 10 and 20 years.

      Note: When you renew the root CA, you have the option of reusing its existing private key. If you keep the same private key on your root CA, all certificates can continue to validate successfully against the new root; all that is required of you is to trust the new root.
    2. Generate the root CA for the first time
      openssl genrsa -out ca.key 2048
      openssl req -new -x509  -days 3650 -sha256 -key ca.key -out ca.pem \
      -subj '/C=UA/O=My Company/CN=My Company Root CA'
    3. After ten years, the renewal time for the root CA comes up.
      • Renew the root CA using the existing ca.key:
        openssl req -new -key ca.key -out newcsr.csr 
        openssl x509 -req -days 3650 -sha256 -in newcsr.csr \
        -signkey newca.key -out newca.pem
      • Generate a completely new root CA:
        openssl genrsa -out newca.key 2048
        openssl req -new -x509  -days 3650 -sha256 -key newca.key \
        -out newca.pem -subj '/C=UA/O=My Company/CN=My Company Root CA' 
    4. Renew the intermediate certificates.

      For the intermediate CAs, a possible strategy might be to renew them for a year to six months before they expire, and reuse the existing key.

      By replacing the old chain file with the new chain file (which contains the updated intermediate certificates), rotation of the intermediate certificates can be performed:
      cat pkey.pem ../int/newint.pem \
      <possibly other intermediate CAs> > chain.pem 
  2. Deploy the CA public key and intermediate certificates

    Before modifying anything on the server-side, deploy the CA public key and intermediate certificates in the certificate-stores used by your client browser and the SDK language.

    For example, here are steps to do that for Firefox and Chrome.

  3. Rotate certificates on the server
    1. Configure the new root CA certificate (newca.pem is the new root CA certificate).
      • Using CLI:
        couchbase-cli ssl-manage -c <node-name>:8091 -u Administrator \
        -p password --upload-cluster-ca=newca.pem  
      • Using REST:
        curl -X POST \
        --data-binary "@newca.pem" http://Administrator:password@
    2. Configure the new intermediate and node certificate.

      For each node, copy over new chain.pem file, and per node private key (new pkey.pem file, if the node certificate is rotated) to the inbox folder.

      • Using CLI:
        couchbase-cli ssl-manage -c <node-name>:8091 -u Administrator \
        -p password --set-node-certificate
      • Using REST:
        curl -X \
        POST http://Administrator:password@[node-name]:8091/node/controller/reloadCertificate 
  4. Test the server CA certificate
    You can also use OpenSSL's s_client by trying to connect to a server that you know is using a certificate signed by the CA that you just installed:
    openssl s_client \
    -connect https://<hostname>:8091 -CApath <root ca public key> 
  5. Troubleshoot X.509 certificates
    During the development process these external tools might come in handy for verifying and debugging SSL traffic:
    • openssl: OpenSSL command line tool
    • wireshark: Network traffic analyzer
    • nmap: Sophisticated security scanner
  6. Revert from the X.509 to the self-signed certificate

    If you configured Couchbase to use X.509 certificates, and you want to go back to the self-signed certificates, you can do this by regenerating the self-signed cluster certificate, using CLI or REST.

    Warning: Moving from CA certificates to self-signed certificates causes application-downtime, because you need to reconfigure the self-signed cluster-certificate on the client-machines after self-signed certificate's regeneration.
    • Using CLI:
      couchbase-cli ssl-manage -c <hostname>:8091 -u Administrator -p password \
    • Using REST:
      curl -X POST  http://Administrator:password@remoteHost:8091/controller/regenerateCertificate

Troubleshooting X.509

This section lists the error messages connected to the configuration of cluster and node certificates in Couchbase.

Cluster CA Certificates

Here are error messages you might encounter when configuring the cluster CA certificate, and suggested corrective actions:

Table 2. Error messages when configuring cluster CA certificates
Couchbase Error Message Description Suggested User Action
Certificate should not be empty This error message can occur if the request body of the certificate is empty. Open the certificate file, and verify whether it is empty or not. The certificate file should be readable using openssl or via online SSL tools such as sslchecker .
Certificate is not valid at this time This error message can occur if the certificate has expired, or is not yet valid. Verify whether the certificate validity-dates (begins on, and expires on) are currently valid corresponding to the server clock time.
Malformed certificate This error message can occur due to many reasons - an extra space in the certificate digest body, incorrect certificate format, and so on. Use a properly configured certificate, and make sure it’s readable, using openssl. It should look as follows: Certificate begins with


and ends with


on a new line with no spaces before or after.

Only one certificate per request is allowed Appears when the file contains more than one key or certificate. Open the .pem file, and make sure that it has only a single certificate digest (such as single BEGIN CERTIFICATE, END CERTIFICATE pragmas).
Encrypted certificates are not supported This error message can occur if you are trying to load a certificate that is encrypted. Verify by opening the certificate file. If you see something similar to the line shown below, you will know your certificate is encrypted.:


Couchbase does not support encrypted certificates. Decrypt the certificate with openssl before loading the certificate in Couchbase.
openssl rsa -in privateKey.pem -out newPrivateKey.pem 
Invalid certificate type: ~s Appears when a header other than BEGIN CERTIFICATE has been found. Open the certificate file, and verify whether it is a valid certificate. The certificate file should be readable using openssl or via online SSL tools such as sslchecker.

Node Certificates

Here are some error messages you might encounter when configuring the node certificate and the suggested corrective actions:

Table 3. Error messages when configuring node certificates
Couchbase Error Message Description Suggested User Action
Cluster CA needs to be set before setting node certificate This error can occur when your cluster is still using the self-generated certificate, and you are attempting to configure a node certificate. Set up the cluster CA certificate before configuring the per node certificate.
Incorrectly configured certificate chain. <Error> Denotes an invalid certificate in the chain file when configuring Couchbase. Chain file should contain a sequence of PEM (base64) encoded X.509 certificates ordered from leaf to and including the intermediate certificate authorities.
Unable to read private key file <Path>. <Error> <Error> is one of the file read errors. Make sure that you have copied an unencrypted version of the private key file to the inbox folder on the Couchbase node.
Unable to read certificate chain file <Path>. <Error> <Error> is one of the file read errors. Make sure that you have copied an unencrypted version of the chain file to the inbox folder on the Couchbase node.
Invalid private key type: <Type> The private key has an unsupported header. Make sure that you use a valid private key file.
Provided certificate doesn't match provided private key The certificate doesn't recognize the message signed with a private key. Be sure that you use a complete key pair
Encrypted keys are not supported The private key is encrypted. Couchbase does not support encrypted keys. You should decrypt the private key with OpenSSL before loading the certificate in Couchbase.
Provided private key contains incorrect number of entries The private key is a chain of entries. The private key file should contain a single key digest.
Malformed or unsupported private key format The private key cannot be used. Open the key file, and verify whether it is a valid private key. The certificate file should be readable using openssl.
File does not exist The file is missing, does not exist. Add the missing file.
Missing permission for reading the file, or for searching one of the parent directories You don't have the proper permissions to read the file or to search its parent directories. Change the permissions to allow you to read the file.