Data Service and Core Data Access

Data Service and Core Data Access

Data service in Couchbase Server provides the core data access with the database engine and incremental MapReduce view processing with the views engine.

Couchbase Server stores data as items. An item is made up of a key (also known as a document key or a document ID) and a document value, along with associated metadata. Couchbase Server organizes data into Buckets.

Couchbase Server provides simple to use and efficient GET and SET methods to mutate and retrieve items by keys, and a number of query methods to filter, group, and aggregate data. Data can be accessed concurrently from many applications and through a mix of these methods at any moment in time. The database engine can process these highly concurrent requests at a sub-millisecond latency at scale. It achieves this through a managed cache, a high throughput storage engine, and a memory based replication architecture. For more information, see Database engine architecture.

Couchbase Server database engine stores and retrieves information using memory first on a single node that carries the master vBucket. This is the fastest option for storing data. Depending on the bucket type, data gets stored on disk eventually. Couchbase Server also provides tunable durability options to store data in the RAM of multiple nodes (using the replicateTo flag) or on disk on one or more nodes (using the persistTo flag). Storing data in the RAM of multiple nodes protects against node failures and also provides a fast and durable way to store data. Storing data on disk can be slow as the operational latency depends on the disk subsystem and typically disk subsystems are much slower than memory access.
When retrieving data using a key, Couchbase Server database engine provides full consistency (sometimes referred to as read-your-own-write semantics) by ensuring access to the master vBucket or optionally allowing access to eventually consistent replica vBuckets for reads (also known as replica reads).

Items are organized into buckets, which provide grouping of items (keys and values). Buckets govern resource allocation and usage, high availability settings, and security for the group of items. Buckets use vBuckets to physically organize storage and distribution of items. Items in a bucket are indexed through Views and indexes created on the buckets’ data. Items in a bucket can also be replicated between clusters using cross datacenter replication (XDCR).

Creating items
Information is stored in the database using a variant of memcached binary protocol interface that stores a value against a specified key. As a developer, you typically use this key through one of the Couchbase client SDKs. Bulk operations to set the key-value pairs of a large number of documents at the same time are available, and are more efficient than multiple smaller requests.

The value stored can be a JSON document or any binary value, including structured and unstructured strings, serialized objects (from the native client language), and native binary data (for example, images or audio). Each bucket can mix the types of data stored with JavaScript Object Notation (JSON) or binary value types. However, using the JSON format gives access to a more powerful query functionality.

Updating items
You can update information in the database using the memcached protocol interface through a Couchbase client SDK. The protocol includes functions to directly update the entire content, and also to perform simple operations, such as appending information to an existing record, or incrementing and decrementing integer values.
Expiration flag
Each document stored in the database has an optional expiration flag (TTL: Time To Live) that is used to automatically delete items. This flag can be set when creating or updating an item. Use the expiration flag for data that has a limited life and needs to be deleted automatically. This flag is useful when storing temporary data such as session state next to your persisted data or when using Couchbase as a caching store in front of another database for high speed access.

You can specify the value of an expiration flag (TTL) in seconds or as Unix epoch time (number of seconds that have elapsed since January 01, 1970). However, if the TTL for an item is greater than 30 days, specify the value in Unix epoch time, for example 1451606399. By default, the expiration flag is set to infinite, that is, the information is stored indefinitely.

Deleting items
You can delete information from the Couchbase Server using a Couchbase client SDK which includes an explicit delete command to remove a key-value pair from the server.
Retrieving items
You can retrieve items from the Couchbase Server with or without referencing keys. You can retrieve values by key using the Data Service.

If you know the key used to store a particular value, then you can use the memcached protocol (or an appropriate memcached compatible client-library) to retrieve the value stored against a specific key. You can also perform bulk retrieve operations.

There are two main ways to retrieve data without referencing keys:
  • Use N1QL to submit a SQL-like query to retrieve your data.
    Note: You must enable Query Service to run N1QL queries.
  • Use views to define and query an alternate projection over the data in the bucket that provides an alternative key that you can use to query. Views are a part of the data service.