What's New in Version 4.6.0?
Couchbase Server is a comprehensive, general purpose database that supports broad set of web, mobile, and IoT applications.
Couchbase Server 4.6 delivers some exciting capabilities for cross datacenter replication, security, query, tools, and application development. This release also adds support for Microsoft Windows 10 Anniversary Edition and Mac OS Sierra. For more information, see Supported Platforms.
Cross Datacenter Replication
- Cross Datacenter Replication with Timestamp-based Conflict Resolution
- This release introduces a new option to resolve conflicts with XDCR using timestamps. With this option, conflicts are resolved by comparing timestamps of conflicting documents. The timestamp-based conflict resolution provides a new option for applications which want users to continue seeing the latest change or version, no matter when conflicts are resolved in the background. Server set timestamp on every document combines the best of logical and physical clocks and captures the causality relationship like logical clocks. For more information, see Timestamp-based Conflict Resolution.
- Hardened Security with Pluggable Authentication Module
- By adding support for Pluggable Authentication Modules (PAM), this release of Couchbase Server enables you to centralize and synchronize password management across servers. You can use existing password management services such as Linux /etc/shadow for a Couchbase cluster. You can also control password expiration rules and other password policies. PAM Authentication in Couchbase is available only on Linux platform and is an Enterprise only feature. For more information, see Pluggable Authentication Modules.
- Secret Management
- With the Secret Management functionality, Couchbase Server provides you a way to securely manage server secrets which helps hardening of Couchbase Server. This feature allows businesses to fulfill important requirements around their server secrets needed for compliance. For more information, see Secret Management and Hardening.
- String Functions
The following new string functions are added. For more details and examples, see
- TOKENS() - Tokenizes given string or JSON object based on specified delimiter and options. When used with GSI functional indexes, this function can bring huge performance improvement to certain queries on text fields.
- REVERSE() - This function reverses the input string.
- Date Functions
The following new date functions are added. For more details and examples, see
- ARRAY_DATE_RANGE(expression1, expression2, part [,n]) - Returns an array of dates from the start date until the end date, incrementing the specified ‘part’ of the date/time by ‘n’.
- CLOCK_LOCAL() - Returns the local time at the server.
- CLOCK_UTC() - Returns the Coordinated Universal Time.
- CLOCK_TZ() - Returns the time in the specified timezone
- DATE_FORMAT_STR(expression,fmt) - Converts a given date string parameter to the specified format.
- MILLIS_TO_LOCAL(millis,fmt) - Converts the UNIX milliseconds into local time in the specified format.
- MILLIS_TO_TZ(millis,zone) - Converts the UNIX milliseconds into time in the specified timezone. This is alias of MILLIS_TO_ZONE_NAME(millis,zone) that is available in earlier versions.
- NOW_LOCAL(void) - Returns the current local time at the server. This is same as CLOCK_LOCAL() function.
- NOW_TZ(zone) - Returns the current time in specified timezone.
- NOW_UTC() - Returns the current time in UTC.
- STR_TO_TZ(strdate,zone)- Returns the specified date string in the specified timezone. This is alias of STR_TO_ZONE_NAME(strdate,zone) available in earlier versions.
- DATE_PART_MILLIS(expression, part [,timezone])- A new timezone parameter is added to this existing function. This function first converts the date in UNIX milliseconds into a date string in the specified timezone, and then returns the corresponding part.
- Array Functions
The following new Array functions are added. For more details and examples, see
- ARRAY_UNION(arr1, arr2, ...) - Returns set union of the input arrays. It retains only distinct array elements in unspecified order.
- ARRAY_SYMDIFF(arr1, arr2) - This function returns disjunctive union of two arrays, that is ARRAY_UNION() minus ARRAY_INTERSECTION(). Result includes values that are in only one of the arrays.
- ARRAY_SYMDIFFN(arr1, arr2) - This function returns symmetric difference of multiple input arrays. Result includes values that are in odd number of the input arrays.
The following Array functions are updated to take variable number of arguments:
- Object Functions
The following new object functions are added. For more details and examples, see
- OBJECT_CONCAT() - Concatenates two JSON objects and returns an object that includes fields from all input objects.
- OBJECT_REMOVE() - Removes specified fields from the input object.
- Performance Improvements
Couchbase Server 4.6 includes over 35 optimizations in the N1QL query engine spread across query planning, better index selection, more efficient query processing algorithms, operator pushdown to index, and efficient resource management. These optimizations are described below:
- Block Nested Loop JOINs: The JOIN algorithm is improved to perform batch processing through various stages of the JOIN, such as fetching documents from data-service and the nested loop join of documents from the left side and right side keyspaces/buckets.
- Covering Index Optimizations:
- Queries using BETWEEN are optimized to leverage covering indexes.
- Query planning is optimized to pick a covering non-array index over an array index when both indexes are available and qualifying for a query.
- Query planner is optimized to pick the shortest covering index when multiple covering indexes are available. In earlier version, first encountered index was used.
- Improved right side covering of INDEX JOIN.
- IntersectScan Optimizations: This is a query operator to use multiple qualifying indexes for a query with conjunctive predicates. Multiple enhancements are made to optimally use IntersectScans (that is, to avoid or leverage them appropriately).
- Avoid IntersectScan on duplicate/replica indexes with same definition, or when the indexes have overlapping definitions (in which case pick the minimal index). In such case, query planner is optimized to pick a single index.
- Use IntersectScan on multiple array-indexes for a query with UNNEST operation.
- Use IntersectScan on non-array index and an array-index, when an UNNEST-query has no qualified covering index. Earlier versions use the secondary scan in such case.
- IndexCountScan Optimizations: This is a query operator that makes COUNT() operations efficient by pushing down the operation to Index whenever possible. Following scenarios are enabled to pushdown COUNT().
- Queries with conjunctive/AND predicates with IN/WITHIN clause having all static values. Note that, IndexCountScan pushdown doesn’t happen when where-clause contains an OR predicate or uses query-parameters in the IN/WITHIN clause.
- Queries using BETWEEN, <, and > operators with parameters in the WHERE clause.
- Improved HTTP performance between client and Query service for transferring query results.
- Improved performance of queries with ORDER BY ASC clause, by leveraging index order in more scenarios, such as when leading index keys are not present in ORDER BY but have fixed/constant values in the query WHERE clause.
Full Text Search [Developer Preview]
- Faster Full Text Indexing and Queries
Full text search (FTS) in 4.6 is noticeably snappier due to many performance enhancements, small and large. Many improvements are due to enhancements made in
bleve, the full-text search and indexing Go library that powers FTS.
The biggest single contributor to performance improvements is MossStore, the new default KV store underlying full text indexes. FTS has for some time used Moss to improve query and especially indexing performance. Moss, which stands for “Memory-oriented sorted segments”, is a simple, fast, persistable, ordered key value collection implemented as a pure Golang library.
MossStore extends Moss so that it efficiently persists sorted segments to disk when necessary. MossStore is recommended for all use cases, but advanced users can still change back to ForestDB by setting the “store” “kvStoreName” property to “forestdb”.
- Index Type Mapping by Keys
- You can now create custom index mappings by document type when the type is specified in the document key. Previously, you could create custom index mappings for different types of objects, but only when the type was indicated by an attribute in the JSON document body. (By default, FTS looks for an attribute named “type”). With this enhancement, it’s easier to support the common data modeling style in which the document type is indicated by a portion of the key, for example, “user::will.gardella”. See Index Type Mapping By Keys for more information.
- You can now sort search results by any indexed field. In the previous releases, search results are always sorted by descending score so that highest scoring results are listed first. This is still the default sort order, so if you don’t specify a sort order, you’re unlikely to notice any difference. See Sorting Query Results for more information.
CBImport and CBExport Tools [Developer Preview]
This release introduces tools to import and export data to and from Couchbase Server. cbimport imports data from a CSV file or a JSON document, and cbexport exports data as a JSON document. For more information, see cbimport and cbexport.
Data Structures for Simplified Application Development
Couchbase is extending the programming model with the new Datastructure SDK feature, further simplifies application development. Building on Couchbase Server 4.5’s support for sub-document level changes, Couchbase has now added support for lists, maps, sets, and queues to libraries for Java, .NET, Node.js, and PHP among other platforms.Datastructure is built on standard JSON giving developers access to the complete set of Couchbase’s best-in-class data access options, including N1QL and Full Text Search. The new structures and other services work seamlessly with the same underlying data representation. For example, a Java developer can use the Java Collections Framework with the list interface and still use SQL standard query to access the same data from N1QL.
Big Data Connectors for Spark and Kafka
- Spark Connector 2.0
Support for Spark 2.0 is now available, including the Structured Streaming API that enables continuous analysis of operational data.Dynamic topology is now automatically supported, making it easier to manage analytics processing in a changing production environment. Stream performance has been significantly improved.
For more information, see Spark Connector 2.0.
- Kafka Connector 3.1
- The Couchbase Kafka Connector makes it easier to build scalable and reliable streaming data services between Apache Kafka and other systems. As of version 3.0, it has been re-written to leverage Kafka Connect, which standardizes the management of the connector, enables end-to-end monitoring, and supports dashboard tools such as Confluent Control Center.
- Kafka Streams make it easier to write stream-based applications. You can now use Couchbase as a Kafka Source or Kafka Sink based on the new Kafka Connect protocol (supported on Kafka 0.9 or newer). With filtering capability you can build an intelligent stream processing environment easily.
- Configuration and management of the Connector is made easier with a couple powerful changes, including Dynamic Topology support for rebalance and failover scenarios. SSL support can now easily be enabled simply by setting a configuration property.
Various important efficiency improvements make better use of connections and resources with the Connector.
- The Connector now has node partition distribution awareness, allowing the cluster map to send DCP stream partitions to Kafka tasks, reducing socket connections.
- The Connector now maintains replication state and can resume streaming from where it left off after a temporary disconnection.
- Faster serialization is now possible as Couchbase sequence numbers are persisted as Kafka Connect offsets instead of as Zookeeper nodes.
- Bulk mode for DCP Snapshots is a new option, making offsets only committed in Kafka once the entire snapshot is received - avoiding duplicate retransmissions of mutations.