Data Structures
Edit this article in GitHub
Version 2.3

Data Structures

You can use complex data structures such as dictionaries and lists in Couchbase. These data structures may be manipulated with basic operations without retrieving and storing the entire document.

Data structures in Couchbase are similar in concept to data structures in C# (.NET):
  • Map is like .NET Dictionary<TKey, TValue>, and is a key-value structure, where a value is accessed by using a key string.
  • List is like a .NET List<TValue> and is a sequential data structure. Values can be placed in the beginning or end of a list, and can be accessed using numeric indexes.
  • Queue is like a IQueue implementation which offers FIFO (first-in-first-out) semantics, allowing it to be used as a lightweight job queue.
  • Set is a wrapper over a List<TValue> which provides the ability to handle unique values.

These data structures are stored as JSON documents in Couchbase, and can therefore be accessed using N1QL and normal key-value operations. Data structures can also be manipulated using the traditional sub-document and full-document KV APIs.

Using the data structures API may help your application in two ways:
  • Simplicity: Data structures provide high level operations by which you can deal with documents as if they were container data structures. Adding an item to a dictionary is expressed as MapAdd, rather than retrieving the entire document, modifying it locally, and then saving it back to the server.
  • Efficiency: Data structure operations do not transfer the entire document across the network. Only the relevant data is exchanged between client and server, allowing for less network overhead and lower latency.
Note: Besides the IBucket level data structure methods, there is a new namespace called Couchbase.Collections which contains implementations of some of the core interfaces in System.Collection.Generics such as ISet<TValue>, IList<TValue>, IDictionary<Tkey, TValue> and a CouchbaseQueue class. See section The Couchbase.Collections Namespace below called for more details.

Creating a Data Structure

Data structures can be created implicitly using the create=True. For example, to add an item to a map, specify the document ID of the map itself (i.e. the ID which uniquely identifies the map in the server), the key within the map, and the value to store under the key:
bucket.MapAdd("map_id", "name", "Mark Nunberg", create=True)
Likewise, to create a list, specify the document ID and the value to add:
bucket.ListAppend("list_id", "hello", create=True)
Note that the create=True will not overwrite an existing document with the same name if it already exists. It is therefore always safe to use, unless your application creates data structures independently.

Data structures can be explicitly created and reset using full-document methods, and initialize it with its JSON equivalent. To create a new empty list, set, or queue, use bucket.Upsert("list_id", "[]"). To create an empty map, use bucket.Upsert("map_id", "{}").

Accessing Data Structures

Data structures can be accessed using their appropriate methods. Most data access methods will return an ValueResult-like object with the actual returned value under the value property.
bucket.ListGet<string>(0).Value  # 'hello'
bucket.MapGet<string>("map_id", "name").Value  # 'mark nunberg'
Native .NET exceptions such as IndexOutOfRangeException will be returned if a given map key or list index is not found within the document. If the document itself does not exist, a Couchbase.DocumentDoesNotExistException will be returned instead.
Here is a list of common operations:
Table 1. Data Structure Operations
   
MapAdd Add a key to a map.
bucket.MapAdd("map_id", "some_key", "some value")
MapRemove Remove a key from a map.
bucket.MapRemove("map_id", "some_key")
MapGet Get an item from a map.
bucket.MapGet<TValue>("map_id", "some_key").Value #=> value
If the key is not found, an IndexError is raised.
ListAppend Add an item to the end of a list.
bucket.ListAppend("list_id", "some_value")
ListPrepend Add an item to the beginning of a list.
bucket.ListPrepend("list_id", "some_value")
ListSet Adds an item to a list at a given index.
bucket.ListSet("list_id", 0, "some_value")
ListRemove Removes an item from a list at a given index.
bucket.ListRemove("list_id", 0)
ListGet Get an item from a list by its index.
bucket.ListGet<TValue>("list_id", 0).Value #=> value
If the index is out of range, an IndexOutOfRangeException will be returned. Note that you can get the last array element by specifying -1 as the index.
SetAdd Add an item to a set, if the item does not yet exist in the set.
bucket.SetAdd("list_id", "some_value")
Note that a set is just a list. You can retrieve the entire set by simply using a full-document get operation:
set = bucket.Get("list_id").Value
Note: Currently, you can only store primitive values in sets, such as strings, ints, and booleans.
SetRemove Remove an item from a set, if it exists. An exception is not thrown if the item does not exist. You can determine if an item existed or not by the return value. If the item did not exist beforehand, None is returned. This operation uses CAS to ensure the correct item is removed.
bucket.SetRemove("list_id", "some_value")
SetContains Checks if a value exists in a set.
bucket.SetContains("set_id", "some_value").Value #=> True or False
QueuePush Add an item to the beginning of the queue.
bucket.QueuePush("a_queue", "job123", create=True)
Note that a queue is just a list. You can retrieve items from the middle of the queue by using ListGet
QueuePop Remove an item from the end queue and return it. This operation uses CAS to ensure no two processs can retrieve the same item.
item = bucket.QueuePop("a_queue").Value #=> 'job123'
If the queue is empty, then a couchbase.exceptions.QueueEmpty error is thrown.
MapSize, ListSize, SetSize, QueueSize These methods get the length of the data structure. For maps, this is the number of key-value pairs inside the map. For lists, queues, and sets, this is the number of elements in the structure.
len = bucket.ListSize('a_list').Value #=> 42

Note that there are only two basic types: map and list. Types such as queue and set are merely derivatives of list.

Data Structures and Key-Value APIs

Data structures can be accessed using key-value APIs as well. In fact, the data structure API is actually a client-side wrapper around the key-value and sub-document API. Most of the data structure APIs wrap the sub-document API directly.
Note: Because the data structure API is just a wrapper around the various key-value APIs, you are free to switch between them in your code.

The Couchbase.Collections Namespace

In addition to the IBucket level methods for working with Data Structures, the .NET SDK also has implementations of System.Collection.Generic interfaces for Sets, Lists, Queues and Dictionaries. Instead of maintaining in-memory storage, these implementations persist to Couchbase Server as a JSON document as the structure is modified.

Note: These classes are idiomatic to the .NET framework and thus are available to the SDK. Other SDKs may or may not support equivalents and/or have implementations that are platform idiomatic.

CouchbaseSet<TValue>

This class is an implementation of ISet<TValue> and represents a set, which is a data structure that can store certain values in any order without duplicates. The underlying storage medium is a JSON document with a single array: "[]". The CouchbaseSet supports the following operations:
Method Description
GetEnumerator Returns an enumerator that iterates through the collection.
Add(TValue) Adds a value to the set.
Clear() Removes all values from the set.
Contains(TValue) Checks if an items exists within the set.
CopyTo(TArray, startIndex) Copies the set to another array at a given index.
Remove(TValue) Removes an item from the set.
Count Returns the total number of items int the set.
SetEquals(IEnumerable<Tvalue>) Compares two sets.
Overlaps(IEnumerable<Tvalue>) Determines whether the current set overlaps with the specified collection.
IsSupersetOf(IEnumerable<TValue>) Determines whether the current set is a superset of a specified collection.
IsSubsetOf(IEnumerable<TValue> other) Determines whether a set is a subset of a specified collection.
SymmetricExceptWith(IEnumerable<TValue>) Modifies the current set so that it contains only elements that are present either in the current set or in the specified collection, but not both.
ExceptWith(IEnumerable<TValue>) Removes all elements in the specified collection from the current set.
IntersectWith(IEnumerable<TValue>) Modifies the current set so that it contains only elements that are also in a specified collection.
UnionWith(IEnumerable<TValue>) Modifies the current set so that it contains all elements that are present in the current set, in the specified collection, or in both.
Here is an example of using the CouchbaseSet:
public class Poco
{
    public string Key { get; set; }
    public string Name { get; set; }
    protected bool Equals(Poco other)
    {
        return string.Equals(Key, other.Key, StringComparison.OrdinalIgnoreCase);
    }
    public override bool Equals(object obj)
    {
        if (ReferenceEquals(null, obj)) return false;
        if (ReferenceEquals(this, obj)) return true;
        if (obj.GetType() != this.GetType()) return false;
        return Equals((Poco) obj);
    }
    public override int GetHashCode()
    {
        return (Key != null ? StringComparer.OrdinalIgnoreCase.GetHashCode(Key) : 0);
    }
}

var collection = new CouchbaseSet<Poco>(_bucket, "pocos");
collection.Add(new Poco { Key = "poco1", Name = "Poco-pica" });
collection.Remove(new Poco {Key = "poco1", Name = "Poco-pica"});

var exists = collection.Contains(new Poco {Key = "poco1", Name = "Poco-pica"});
foreach(var poco in collection){ Console.WriteLine(poco);}

The Poco class overrides Equals() and GetHashCode() will be used henceforth in each example.

CouchbaseDictionary<TKey, TValue>

The CouchbaseDictionary class is an implementation of IDictionary<TKey, TValue> which stores its values as a hash map in a JSON document in Couchbase. It supports the following operations:

Method Description
Key Gets the key for this document.
GetEnumerator() Returns an enumerator that iterates through the collection.
Add(KeyValuePair<TKey, TValue>) Adds an item to the hash map.
Clear() Removes all items from the hash map.
Contains(KeyValuePair<TKey, TValue>) Determines if an item exists within the hash map.
CopyTo(KeyValuePair<TKey, TValue>[], arrayIndex) Copies the values to an array starting at an index.
Remove(KeyValuePair<TKey, TValue> item) Removes an item from the hash map.
Count Returns the total number of items into the hash map.
Add(TKey key, TValue value) Adds an item to the hash map.
Remove(TKey key) Removes an item from the hash map.
TryGetValue(TKey key, out TValue value) Gets a value from the hash map at a given key.
this[TKey key] Indexer - gets a value from the hash map at a given key.
Keys Get all of the keys in the document.
Values Get all of the values in the document.
Here is an example using the CouchbaseDictionary<TKey, TValue>:
var dictionary = new CouchbaseDictionary<string, Poco>(_bucket, key);
dictionary.Add("somekey1", new Poco { Name = "poco1" });
dictionary.Add("somekey2", new Poco { Name = "poco2" });
var removed = dictionary.Remove("somekey2");

foreach(var kv in dictionary)
{
    Console.WriteLine("{0}=>{1}", kv.Key, kv.Value);
}            

CouchbaseList<TValue>

The CouchbaseList<TValue> is an implementation of IList<TValue> and represents a list of items including duplicates. It supports the following operations:

Method Description
Key Gets the key for this document.
Add(T item) Adds an item to the list.
GetEnumerator() Returns an enumerator that iterates through the collection.
Contains(T item) Determines if an item exists in the list.
Remove(T item) Removes an item from the list.
IndexOf(T item) Gets the index of an item in the list.
Insert(int index, T item) Inserts an item into the list at a given index.
RemoveAt(int index) Removes an item from the list at a given index.
this[int index] Indexer - gets an item in the list at a given index.
Get(int index) Gets an item in the list at a given index.
Clear() Removes all items from the list.
Count Returns the total number of items into the list.
Here is an example using the CouchbaseList<TValue>:
var collection = new CouchbaseList<Poco>(_bucket, "BucketListTests_Test_Enumeration");

var numItems = 5;
for (var i = 0; i < numItems; i++)
{
    collection.Add(new Poco {Key = "poco"+i, Name = "Poco-pica"+i});
}

foreach (var poco in collection)
{
    Console.WriteLine(poco);
}

CouchbaseQueue<TValue>

The CouchbaseQueue<TValue> is implementation of a Queue data structure and offers similar functionality to System.Collections.Generic.Queue<TValue>. It offers the following operations:

Method Description
Key Gets the key for this document.
GetEnumerator() Returns an enumerator that iterates through the collection.
Clear() Removes all items from the queue.
CopyTo(KeyValuePair<TKey, TValue>[], arrayIndex) Copies the values to an array starting at an index.
Dequeue() Removes and returns the object at the beginning of the queue. This operation uses CAS to ensure no two calls gets the same item.
Enqueue(T item) Adds an object to the end of the queue.
Peek() Returns the object at the beginning of the queue without removing it.

Here is an example of using the CouchbaseQueue<TValue>:

var queue = new CouchbaseQueue<Poco>(_bucket, key);
queue.Enqueue(new Poco { Name = "pcoco1" });
queue.Enqueue(new Poco { Name = "pcoco2" });
queue.Enqueue(new Poco { Name = "pcoco3" });

var item = queue.Peek();
Console.WriteLine(item);

Concurrency

Where appropriate, all of operations on the IBucket level and the implementations in Couchbase.Collections utilise the CAS (Compare and Swap) value when manipulating data to ensure that two different processes do not manipulate the same data at the same time. For example: when using QueuePop, two threads could not receive the same item.