Release notes for the 3.0 version of the Kafka Connector.
Couchbase Kafka Connector 3.0.0 GA (14 December 2016)
Version 3.0.0 is GA release. It brings documentation update, feature enhancements, and bug fixes.
What's new in Kafka Connector 3.0.0
- Kafka Connect - This release is based on Kafka Connect, a mechanism in Apache Kafka 0.9 and higher versions for scalably and reliably streaming data between Apache Kafka and other systems.
Here are some advantages for Ops teams:
Here are some advantages for application developers and ETL engineers:
- Part of Kafka
- One service to manage with connector configuration handled internally.
- Unified connector control, monitoring, and metrics.
- Easy to set up as a self-service system for developers and ETL team.
- Integrated with any configuration management or container system.
- Configuration-only solution for copying large scale data to/from Kafka.
- Built for real-time ETL.
- Out of the box tools such as Confluent Control Center dashboards can visualize the complete data pipeline.
- Dynamic topology - The Kafka Connector now handles rebalance and failover scenarios. Adding and removing nodes to an active Couchbase Server cluster no longer requires restarting the Kafka Connector. (KAFKAC-7)
- Node-aware partition distribution - For efficiency reasons, the Couchbase Kafka Connector 3.0 uses the Couchbase Server cluster map to send DCP stream partitions to Kafka tasks so that they use the smallest number of DCP socket connections.
- Connector maintains replication state and can resume streaming from where it left off after a temporary disconnection. (KAFKAC-47)
- Faster serialization - Couchbase sequence numbers are now persisted as Kafka Connect offsets instead of Zookeeper nodes, thus making the serializer faster. (KAFKAC-44)
- SSL support - You can enable SSL support by simply setting a configuration property. (KAFKAC-52)
- Bulk mode for snapshot transmission - You can now enable bulk mode for DCP Snapshots, so that the offsets are only committed in Kafka once the entire snapshot is received. This avoids creating duplicates caused by retransmission of mutations in more than one DCP snapshot. (KAFKAC-50)
Couchbase Kafka Connector 3.0.0 BETA (22 November 2016)
Version 3.0.0-BETA is pre-release version of the 3.0.0. It brings documentation update, feature enhancements and bug fixes
- KAFKAC-52: Support for SSL connections
- Update dependencies: dcp-client to 0.7.0, and confluent libraries up to versions shipped with 3.1.1
- Cleanup various configuration workarounds for platform 3.0
Couchbase Kafka Connector 3.0.0 DP4 (5 November 2016)
Version 3.0.0-DP4 is the fourth developer preview of the 3.0.x series.
- KAFKAC-54: Create example of using in Kafka Stream to process events from Couchbase
- Rename internal classes, and make configuration more consistent with other connectors (e.g. instead of timeout_ms, use timeout.ms)
- Allow to override internal convertor into SourceRecord, and allow to inject Filter class to skip events before writing into Kafka
Couchbase Kafka Connector 3.0.0 DP3 (20 October 2016)
Version 3.0.0-DP3 is the third developer preview of the 3.0.x series. It implements new features and also includes bug fixes to previous release.
- KAFKAC-50: Allow to buffer DCP snapshots for consistent writes.
- KAFKAC-51: Specify key for SourceRecord. Allows to use multiple Kafka partitions.
- KAFKAC-53: Node-aware distribution of partitions for Tasks. Reduces amount of resources allocated on the server.
Couchbase Kafka Connector 3.0.0 DP2 (24 September 2016)
Version 3.0.0-DP2 is the second developer preview of the 3.0.x series. It improves configuration. And now can maintain replication state, which allow to resume transmission.
Couchbase Kafka Connector 3.0.0 DP1 (6 September 2016)
Version 3.0.0-DP1 is the first developer preview of the 3.0.x series.