Using Logs

Using Logs

Logs allow you to identify activity and errors within the Couchbase cluster.

Collect Log Information

From the main Couchbase Web Console screen select the Log tab. Two sub-tabs are available: Logs and Collect Information.

The Logs tab displays available logs by the Event, Module Code, Server Node, and Time.

The Collection Information tab enables you to collect logs and diagnostic information from either all nodes or selected nodes in a cluster. You can also upload the collected logs directly to Couchbase Support with your data (customer name, ticket number) for easy tracking.

Log Types

Couchbase Server creates a number of different log files depending on the components of the system that produce the error and the level and severity of the reported problem.

Platform Location
Linux /opt/couchbase/var/lib/couchbase/logs
Windows C:\Program Files\Couchbase\Server\var\lib\couchbase\logs

Assumes default installation location

Mac OS X /Users/couchbase/Library/Application Support/Couchbase/var/lib/couchbase/logs

The table below summarizes the log files for the different components in Couchbase:

Table 1. Log files
File Log Contents
audit Security audit log for administrators.
babysitter Troubleshooting log for the babysitter process which is responsible for spawning all Couchbase Server processes and respawning them where necessary.
couchdb Troubleshooting log for the couchdb subsystem which underlies map-reduce and spatial views
crash-log.bin Used to pass service crash reports from the babysitter to the ns_server. For example, if the ns_server is available, any crash of the babysitter's child is passed directly to the special crash logger service within the ns_server. If the logger service is not attached to the babysiter, then the babysitter saves that crash report to the disk and the ns_server can later obtain and log it even if the babysitter is restarted. This is not a log file in itself.
debug Debug-level troubleshooting for the cluster management component.
error Error-level troubleshooting log for the cluster management component.
fts Troubleshooting logs for the full-text search service.
goxdcr Troubleshooting log for the cross datacenter replication (XDCR) component used in Couchbase Server versions after 4.0.
http_access The admin access log records server requests (including administrator logins) to the REST API or Couchbase Server web console. It is output in common log format and contains several important fields such as remote client IP, timestamp, GET/POST request and resource requested, HTTP status code, and so on.
http_access_internal The admin access log records internal server requests (including administrator logins) to the REST API or Couchbase Server web console. It is output in common log format and contains several important fields such as remote client IP, timestamp, GET/POST request and resource requested, HTTP status code, and so on.
indexer Troubleshooting log for the indexing and storage subsystem.
info Info-level troubleshooting log for the cluster management component.
mapreduce_errors JavaScript and other view-processing errors are reported in this file.
memcached Contains information relating to the core memcached component, including DCP stream requests and slow operations.
metakv Troubleshooting log for the metakv store, a cluster-wide metadata store.
ns_couchdb Contains information related to starting up the CouchDB subsystem.
projector Troubleshooting log for the projector process which is responsible for sending appropriate mutations from Data nodes to Index nodes.
reports Contains progress and crash reports for the Erlang processes. Due to the nature of Erlang, processes crash and restart upon an error.
ssl_proxy Troubleshooting log for the ssl proxy spawned by the cluster manager.
stats Contains periodic statistic dumps from the cluster management component.
views Troubleshooting log for the view engine, predominantly focussing on the changing of partition states.
xdcr Troubleshooting log for the cross datacenter replication (XDCR) component used in Couchbase Server versions prior to 4.0.
xdcr_errors Error-level troubleshooting log for the cross datacenter replication (XDCR) component used in Couchbase Server versions prior to 4.0.
xcdr_trace Trace-level troubleshooting log for the cross datacenter replication (XDCR) component used in Couchbase Server versions prior to 4.0. Unless trace-level logging is explicitly turned on this log is empty.

Some logs are automatically rotated after a certain fixed size. For example, individual log files are automatically numbered with the number suffix incremented for each new log, and with a maximum of 20 files per log. Individual log file sizes are limited to 10MB by default.

For other logs, when a log file reaches 40MB it will be rotated and compressed. The file will keep 5 rotations (the current rotation plus four compressed rotations). Here is an example list of log files:

-rw-rw---- 1 couchbase couchbase 12M Feb 2 16:15 couchdb.log 
        -rw-rw---- 1 couchbase couchbase 4.8M Feb 2 16:13 couchdb.log.1.gz 
        -rw-rw---- 1 couchbase couchbase 4.5M Jan 30 17:35 couchdb.log.2.gz 
        -rw-rw---- 1 couchbase couchbase 3.9M Jan 30 17:34 couchdb.log.3.gz 
        -rw-rw---- 1 couchbase couchbase 5.7M Jan 30 17:30 couchdb.log.4.gz 
      

In this list, the oldest file has the largest number.

To provide custom rotation settings for each component, add the following to your static_config:

{disk_sink_opts_disk_debug, 
        [{rotation, [{size, 10485760}, 
        {num_files, 10}]}]}.

This will rotate the debug.log at 10MB and keep 10 copies of the log (the current log and 9 compressed logs).

Managing Logs

Changing log file location

The default file log location is /opt/couchbase/var/lib/couchbase/logs, however, if you want to change the default log location to a different directory, change the log file configuration option.

Note: To implement a log file location change (from the default), you must be log in as either root or sudo and the Couchbase service must be restarted.

To change the log file configuration:

  1. Log in as root or sudo and navigate to the directory where you installed Couchbase. For example: /opt/couchbase/etc/couchbase/static_config
  2. Edit the static_config file and change the error_logger_mf_dir variable to a different directory. For example: {error_logger_mf_dir, "/home/user/cb/opt/couchbase/var/lib/couchbase/logs"}
  3. Restart the Couchbase service. After restarting the Couchbase service, all subsequent logs will be in the new directory.
Changing logging levels

The default logging level for all log files are set to debug except for couchdb, which is set to info. If you want to change the default logging level, modify the logging level configuration options.

The configuration change can be performed in one of the following ways:

  • persistent
  • dynamic (on the fly, without restarting).
Changing logging levels to be persistent

Logging levels can be changed so that the changes are persistent, that is, the changes continue to be implemented should a Couchbase Server reboot occur.

Note: To implement logging level changes, the Couchbase service must be restarted.

To change logging levels to be persistent:

  1. Log in as root or sudo and navigate to the directory where you installed Couchbase. For example: /opt/couchbase/etc/couchbase/static_config
  2. Edit the static_config file and change the desired log component. For example, parameters with the loglevel_ prefix set the logging level.
  3. Restart the Couchbase service.

After restarting the Couchbase service, logging levels for that component will be changed.

Changing logging levels dynamically

If logging levels are changed dynamically and if a Couchbase server reboot occurs, then the changed logging levels revert to the default.

To change logging levels dynamically, execute a curl POST command using the following syntax:

curl -X POST -u adminName:adminPassword HOST:PORT/diag/eval \ 
              -d ‘ale:set_loglevel(<log_component>,<logging_level>).’				 

Where:

Log_component
The default log level (except couchdb) is debug; for example ns_server. The available loggers are ns_server, couchdb, user, Menelaus, ns_doctor, stats, rebalance, cluster, views, mapreduce_errors , xdcr and error_logger.
Logging_level
The available log levels are debug, info, warning, and error.
curl -X POST -u Administrator:password http://127.0.0.1:8091/diag/eval \ 
                -d 'ale:set_loglevel(ns_server,error).				

Collect Logs with CLI or REST API

Using CLI commands

You can use the CLI command cbcollect_info, which is one of the most important diagnostic tools used by Couchbase technical support.

The other three CLI commands you can use to start and stop log collection and to read the log collection status:
Using REST API

The Logs REST API provides the endpoints for retrieving log and diagnostic information.

To retrieve log information use the /diag and /sasl_logs REST endpoints.