Clock Sync with NTP
Network Time Protocol (NTP) is used to synchronize time across multiple servers. The purpose of NTP is to keep an individual node's clock 'accurate'. This is done by having the node periodically synchronize its clock with a reference server. You can specify multiple servers to provide redundancy in case one or more time servers are unavailable. Correct time synchronisation is important in Couchbase Server for Expiry behaviour and to maintain accurate Hybrid Logical Clocks used in Time-Based Conflict Resolution.
Time Sync Configuration
The following figure illustrates the recommended topology for time synchronization using NTP.
You should install local NTP Servers and also have more than one NTP server for fault tolerance. Each Couchbase Server node should have NTP Client configured which synchronize time with local NTP Server.
NTP server on each cluster should ideally reference same NTP server on the Internet to have the same time.
Basic Setup and Configuration
NTP Server Configuration
For detailed information on how to setup, best practices, and advanced configuration about NTP, refer to http://www.ntp.org/.
Recommended NTP Server Configuration
- Using an external NTP server to synchronize each Couchbase node independently.
- Using a dedicated NTP server ( a core server, possibly running on a separate instance) within the Couchbase cluster. This server is synchronized with external NTP servers (over the Internet) while the rest of the cluster will synchronize with this one.
- Using a dedicated pool of NTP servers within the Couchbase cluster. These NTP servers within a pool synchronize with external NTP servers (over the Internet) while the rest of the cluster will synchronize with this one.
- In case of a bigger, scalable cluster, Couchbase recommends that you run your own NTP server pool that is externally synchronized. You should have at least two NTP sources on the same local network as each Couchbase RZA server group. This configuration will provide perfect relative and absolute clock synchronization, high availability, and easy maintenance.
- Couchbase recommends configuring NTP servers/pool as peers across clusters. This enables the NTP servers/pool on one cluster to have a relationship between the NTP servers/pool of the other cluster.
NTP Client Configuration
- Start NTP on each node.
Start NTP by starting the NTP service, or daemon, on your system. On most systems, starting the NTP daemon happens automatically on startup. You do not need to perform this action manually. However, if you need to make adjustments to the NTP configuration, it is useful to know how to stop and start the service.For example, the following command starts the daemon:
$ service ntp start -x
- Point each instance of NTP to the same set of reference servers.
Specify the time server(s) in the NTP configuration file (usually /etc/ntp.conf). You can specify multiple servers, one server per line.It is recommended to setup your own NTP server and sync NTP clients.
server ntp1.phx.xxx.com server ntp2.phx.xxx.com
The configuration file is read when the NTP service starts. So, if you change the configuration file after NTP is running, stop and restart the service to have the new configuration options take effect.Once ntp.conf is configured with correct settings, start the ntp daemon.
/etc/init.d/ntp startYou will see the NTP will slowly start to synchronize the time of your Linux machine with the NTP Server.Note: As noted in the NTP documentation, it is important to remember that ntpdate with contrived cron scripts is no substitute for the NTP daemon, which uses sophisticated algorithms to maximize accuracy and reliability while minimizing resource use.
Verifying Your NTP Configuration
To verify that your NTP configuration is working properly, run the following:
- Use the ntpstat command to view the status of the NTP service on the instance.
If your output resembles the output below, then NTP is working properly on the instance.
[ec2-user ~]$ ntpstat
synchronised to NTP server (126.96.36.199) at stratum 3 time correct to within 399 ms polling server every 64 sIf your output states " unsynchronised", wait for about a minute and try again. The first synchronization may take a minute to complete.Note: Do not start XDCR until every node in each cluster says synchronized.
If your output states "Unable to talk to NTP daemon. Is it running?", you probably need to start the NTP service and enable it to automatically start at boot time.
- (Optional) You can use the ntpq -p command to see a list of peers known to the NTP server and a summary of their state.
[ec2-user ~]$ ntpq -p remote refid st t when poll reach delay offset jitter ================================================================================ +lttleman.deekay 188.8.131.52 2 u 15 128 377 88.649 5.946 6.876 -bittorrent.tomh 184.108.40.206 3 u 133 128 377 182.673 8.001 1.278 *ntp3.junkemailf 220.127.116.11 2 u 68 128 377 29.377 4.726 11.887 +tesla.selinc.co 18.104.22.168 2 u 31 128 377 28.586 -1.215 1.435
If the output of this command shows no activity, check whether your security groups, network ACLs, or firewalls block access to the NTP port.