What's New?

What's New?

Couchbase Server is a comprehensive, general purpose database that supports broad set of web, mobile, and IoT applications.

Release 4.6.2

N1QL Enhancements

N1QL introduces a number of new functions and performance optimizations in the Couchbase Server 4.6.2 Query engine.
Expression Support in FROM clause
Earlier releases allow only keyspace identifier or subquery in SELECT statement’s FROM clause. Couchbase Server 4.6.2 adds support for generic N1QL expressions in SELECT statement’s FROM clause. This is very powerful functionality, as it enables usage of various N1QL functions, operators, path expressions, and language constructs on constant expressions, variables, and subqueries. Further, it improves performance of correlated Subqueries that access nested arrays and sub-documents of the same document (as in its parent query). For details, see FROM statement syntax and Subqueries.
Array Indexing
Couchbase Server 4.6.2 release adds the following enhancements to Array Indexing feature:
  • Allow arbitrary variable names in array index selections. That is, a SELECT query that needs to use an array index can use different variable names in the query from those used in the array index definition.

  • Support simpler syntax for array indexing when all array elements are indexed, without requiring the use of the ARRAY operator in the index definition.

  • In earlier releases, it is required to provide the whole-array as an additional index-key to create a covering array index. But now in 4.6.2, the explicit array index key is not required when the query predicate can be exactly and completely pushed to the indexer. For details, see Array Indexing.
Token Functions
The following new token functions are added. For more details and examples, see Token Functions.
  • HAS_TOKEN() - This function checks if input_obj contains the token token_expr.
  • CONTAINS_TOKEN() - This function is an alias of HAS_TOKEN().
  • CONTAINS_TOKEN_LIKE() - Checks if input_obj contains the token token_expr, which can have wildcards in the same way as the LIKE operator.
  • CONTAINS_TOKEN_REGEXP - Checks if input_obj contains the token token_expr, which can have regular expressions.
Object Functions
The following new object functions are added. For more details and examples, see Object Functions.
  • OBJECT_RENAME() - Renames a field name old_field in the JSON object obj to new_field.
  • OBJECT_REPLACE() - Replaces all occurrences of the value old_val to new_val for any of the fields in the JSON object obj.
Date Functions
The following new date functions are added. For more details and examples, see Date Functions.
  • DATE_RANGE_STR - This function is enabled to accept different supported date formats for different parameters. This removes the earlier limitation to require all parameters to be in the same format.
  • WEEKDAY_STR() - Returns the weekday name from the input date in string format.
  • WEEKDAY_MILLIS() - Returns the weekday name from the input date in Unix timestamp.
Performance Improvements
Couchbase Server 4.6.2 includes the following improvements:
  • The following functions are enabled to automatically use available functional indexes created with SUFFIXES() or TOKENS() respectively called suffixes-index and tokens-index.
    • LIKE operator to use suffixes-index

    • CONTAINS() can use suffixes-index

    • CONTAINS_TOKEN_LIKE() can use suffixes-index

    • HAS_TOKEN() and CONTAINS_TOKEN() can use tokens-index

    • CONTAINS_TOKEN_REGEXP() can use tokens-index
  • When covering UNNEST index is used, the operators LIMIT and ORDER can be push down to indexer and exploit index order.
  • Push down LIMIT operation to Index during IntersectScan, UnionScan, and DistinctScan.
  • New query execution operator OrderedIntersectScan is introduced to leverage the index order during intersect scans. For example, when two index scans are being intersected, and the SELECT query has ORDER BY clause on an attribute that is the leading index keys of one of the index, then the order of results obtained from that index are preserved through the query processing to avoid redundant sorting.
  • Optimize processing of queries to use qualified secondary index when the WHERE clause has an IN operator with a parameterized right-side value. Earlier releases used to do full-index scan.
  • Queries having WHERE clause with NOT IN or NOT WITHIN operators are enabled to use qualified secondary indexes. In earlier releases, they used primary indexes.
  • Queries having a WHERE clause with an OR predicate:
    • With disjoint OR conditions (on different attributes) are enabled to use qualified secondary indexes for individual conditions. For example, if the predicate is (a = 5) OR (b = 10), where a has one index and b has another index, then both indexes are used for the query. Earlier releases used to use primary index.
    • Create variable spans for the Index Scans when using OR predicates to the same index. For example, a predicate like (a = 5) OR ((a = 10) and (b = 10)), can use just one index with both a and b as index keys. In such case, N1QL will generate more optimized spans for the index scan.

Release 4.6.0

Couchbase Server 4.6 delivers some exciting capabilities for cross datacenter replication, security, query, tools, and application development. This release also adds support for Microsoft Windows 10 Anniversary Edition and Mac OS Sierra. For more information, see Supported Platforms.

Cross Datacenter Replication

Cross Datacenter Replication with Timestamp-based Conflict Resolution
This release introduces a new option to resolve conflicts with XDCR using timestamps. With this option, conflicts are resolved by comparing timestamps of conflicting documents. The timestamp-based conflict resolution provides a new option for applications which want users to continue seeing the latest change or version, no matter when conflicts are resolved in the background. Server set timestamp on every document combines the best of logical and physical clocks and captures the causality relationship like logical clocks. For more information, see Timestamp-based Conflict Resolution.

Security

Hardened Security with Pluggable Authentication Module
By adding support for Pluggable Authentication Modules (PAM), this release of Couchbase Server enables you to centralize and synchronize password management across servers. You can use existing password management services such as Linux /etc/shadow for a Couchbase cluster. You can also control password expiration rules and other password policies. PAM Authentication in Couchbase is available only on Linux platform and is an Enterprise only feature. For more information, see Pluggable Authentication Modules.
Secret Management
With the Secret Management functionality, Couchbase Server provides you a way to securely manage server secrets which helps hardening of Couchbase Server. This feature allows businesses to fulfill important requirements around their server secrets needed for compliance. For details, see Secret Management and Hardening.

N1QL Enhancements

N1QL introduces a number of new functions and performance optimizations in the Couchbase Server 4.6 Query engine.
String Functions
The following new string functions are added. For details and examples, see String Functions.
  • TOKENS() - Tokenizes given string or JSON object based on specified delimiter and options. When used with GSI functional indexes, this function can bring huge performance improvement to certain queries on text fields.
  • REVERSE() - This function reverses the input string.
Date Functions
The following new date functions are added. For details and examples, see Date Functions.
  • ARRAY_DATE_RANGE(expression1, expression2, part [,n]) - Returns an array of dates from the start date until the end date, incrementing the specified ‘part’ of the date/time by ‘n’.
  • CLOCK_LOCAL() - Returns the local time at the server.
  • CLOCK_UTC() - Returns the Coordinated Universal Time.
  • CLOCK_TZ() - Returns the time in the specified timezone
  • DATE_FORMAT_STR(expression,fmt) - Converts a given date string parameter to the specified format.
  • MILLIS_TO_LOCAL(millis,fmt) - Converts the UNIX milliseconds into local time in the specified format.
  • MILLIS_TO_TZ(millis,zone) - Converts the UNIX milliseconds into time in the specified timezone. This is alias of MILLIS_TO_ZONE_NAME(millis,zone) that is available in earlier versions.
  • NOW_LOCAL(void) - Returns the current local time at the server. This is same as CLOCK_LOCAL() function.
  • NOW_TZ(zone) - Returns the current time in specified timezone.
  • NOW_UTC() - Returns the current time in UTC.
  • STR_TO_TZ(strdate,zone)- Returns the specified date string in the specified timezone. This is alias of STR_TO_ZONE_NAME(strdate,zone) available in earlier versions.
  • DATE_PART_MILLIS(expression, part [,timezone])- A new timezone parameter is added to this existing function. This function first converts the date in UNIX milliseconds into a date string in the specified timezone, and then returns the corresponding part.
Array Functions
The following new Array functions are added. For details and examples, see Array Functions.
  • ARRAY_UNION(arr1, arr2, ...) - Returns set union of the input arrays. It retains only distinct array elements in unspecified order.
  • ARRAY_SYMDIFF(arr1, arr2) - This function returns disjunctive union of two arrays, that is ARRAY_UNION() minus ARRAY_INTERSECTION(). Result includes values that are in only one of the arrays.
  • ARRAY_SYMDIFFN(arr1, arr2) - This function returns symmetric difference of multiple input arrays. Result includes values that are in odd number of the input arrays.
The following Array functions are updated to take variable number of arguments:
  • ARRAY_APPEND()
  • ARRAY_CONCAT()
  • ARRAY_INSERT()
  • ARRAY_PREPEND()
  • ARRAY_PUT()
  • ARRAY_REMOVE()
Object Functions
The following new object functions are added. For details and examples, see Object Functions.
  • OBJECT_CONCAT() - Concatenates two JSON objects and returns an object that includes fields from all input objects.
  • OBJECT_REMOVE() - Removes specified fields from the input object.
Performance Improvements
Couchbase Server 4.6 includes over 35 optimizations in the N1QL query engine spread across query planning, better index selection, more efficient query processing algorithms, operator pushdown to index, and efficient resource management. These optimizations are described below:
  • Block Nested Loop JOINs: The JOIN algorithm is improved to perform batch processing through various stages of the JOIN, such as fetching documents from data-service and the nested loop join of documents from the left side and right side keyspaces/buckets.
  • Covering Index Optimizations:
    • Queries using BETWEEN are optimized to leverage covering indexes.
    • Query planning is optimized to pick a covering non-array index over an array index when both indexes are available and qualifying for a query.
    • Query planner is optimized to pick the shortest covering index when multiple covering indexes are available. In earlier version, first encountered index was used.
    • Improved right side covering of INDEX JOIN.
  • IntersectScan Optimizations: This is a query operator to use multiple qualifying indexes for a query with conjunctive predicates. Multiple enhancements are made to optimally use IntersectScans (that is, to avoid or leverage them appropriately).
    • Avoid IntersectScan on duplicate/replica indexes with same definition, or when the indexes have overlapping definitions (in which case pick the minimal index). In such case, query planner is optimized to pick a single index.
    • Use IntersectScan on multiple array-indexes for a query with UNNEST operation.
    • Use IntersectScan on non-array index and an array-index, when an UNNEST-query has no qualified covering index. Earlier versions use the secondary scan in such case.
  • IndexCountScan Optimizations: This is a query operator that makes COUNT() operations efficient by pushing down the operation to Index whenever possible. Following scenarios are enabled to pushdown COUNT().
    • Queries with conjunctive/AND predicates with IN/WITHIN clause having all static values. Note that, IndexCountScan pushdown doesn’t happen when where-clause contains an OR predicate or uses query-parameters in the IN/WITHIN clause.
    • Queries using BETWEEN, <, and > operators with parameters in the WHERE clause.
  • Improved HTTP performance between client and Query service for transferring query results.
  • Improved performance of queries with ORDER BY ASC clause, by leveraging index order in more scenarios, such as when leading index keys are not present in ORDER BY but have fixed/constant values in the query WHERE clause.

Full Text Search [Developer Preview]

Faster Full Text Indexing and Queries
Full text search (FTS) in 4.6 is noticeably snappier due to many performance enhancements, small and large. Many improvements are due to enhancements made in bleve, the full-text search and indexing Go library that powers FTS.

The biggest single contributor to performance improvements is MossStore, the new default KV store underlying full text indexes. FTS has for some time used Moss to improve query and especially indexing performance. Moss, which stands for “Memory-oriented sorted segments”, is a simple, fast, persistable, ordered key value collection implemented as a pure Golang library.

MossStore extends Moss so that it efficiently persists sorted segments to disk when necessary. MossStore is recommended for all use cases, but advanced users can still change back to ForestDB by setting the “store” “kvStoreName” property to “forestdb”.

Index Type Mapping by Keys
You can now create custom index mappings by document type when the type is specified in the document key. Previously, you could create custom index mappings for different types of objects, but only when the type was indicated by an attribute in the JSON document body. (By default, FTS looks for an attribute named “type”). With this enhancement, it’s easier to support the common data modeling style in which the document type is indicated by a portion of the key, for example, “user::will.gardella”. For details, see Index Type Mapping By Keys.
Sorting
You can now sort search results by any indexed field. In the previous releases, search results are always sorted by descending score so that highest scoring results are listed first. This is still the default sort order, so if you don’t specify a sort order, you’re unlikely to notice any difference. For details, see Sorting Query Results.

CBImport and CBExport Tools [Developer Preview]

This release introduces tools to import and export data to and from Couchbase Server. cbimport imports data from a CSV file or a JSON document, and cbexport exports data as a JSON document. For details, see cbimport and cbexport.

Data Structures for Simplified Application Development

Couchbase is extending the programming model with the new Datastructure SDK feature, further simplifies application development. Building on Couchbase Server 4.5’s support for sub-document level changes, Couchbase has now added support for lists, maps, sets, and queues to libraries for Java, .NET, Node.js, and PHP among other platforms.

Datastructure is built on standard JSON giving developers access to the complete set of Couchbase’s best-in-class data access options, including N1QL and Full Text Search. The new structures and other services work seamlessly with the same underlying data representation. For example, a Java developer can use the Java Collections Framework with the list interface and still use SQL standard query to access the same data from N1QL.

Big Data Connectors for Spark and Kafka

Spark Connector 2.0

Support for Spark 2.0 is now available, including the Structured Streaming API that enables continuous analysis of operational data.

Dynamic topology is now automatically supported, making it easier to manage analytics processing in a changing production environment. Stream performance has been significantly improved.

For details, see Spark Connector 2.0.

Kafka Connector 3.1
The Couchbase Kafka Connector makes it easier to build scalable and reliable streaming data services between Apache Kafka and other systems. As of version 3.0, it has been re-written to leverage Kafka Connect, which standardizes the management of the connector, enables end-to-end monitoring, and supports dashboard tools such as Confluent Control Center.
Kafka Streams make it easier to write stream-based applications. You can now use Couchbase as a Kafka Source or Kafka Sink based on the new Kafka Connect protocol (supported on Kafka 0.9 or newer). With filtering capability you can build an intelligent stream processing environment easily.
Configuration and management of the Connector is made easier with a couple powerful changes, including Dynamic Topology support for rebalance and failover scenarios. SSL support can now easily be enabled simply by setting a configuration property.
Various important efficiency improvements make better use of connections and resources with the Connector.
  • The Connector now has node partition distribution awareness, allowing the cluster map to send DCP stream partitions to Kafka tasks, reducing socket connections.
  • The Connector now maintains replication state and can resume streaming from where it left off after a temporary disconnection.
  • Faster serialization is now possible as Couchbase sequence numbers are persisted as Kafka Connect offsets instead of as Zookeeper nodes.
  • Bulk mode for DCP Snapshots is a new option, making offsets only committed in Kafka once the entire snapshot is received - avoiding duplicate retransmissions of mutations.
For details, see Kafka Connector 3.1.