download the new version Sticky Previews 2.8

The new KRaft consensus and quorum controller enables Kafka clusters to scale to millions of partitions through improved control plane performance with the new metadata management improves stability, and makes it easier to monitor, administer, and support Kafka allows Kafka to have a single security model for the whole system and makes controller failover near-instantaneous. The image below shows a much faster shutdown of a Kafka cluster with two million partitions using the new quorum controller versus ZooKeeper. This significantly decreases the unavailability window, improving the worst-case recovery time of the system. In case of issues, unlike the ZooKeeper-based controller, the quorum controller does not need to load state from ZooKeeper since the internal state of the cluster is already distributed in the metadata topic. The event log is periodically snapshotted to guarantee that the log cannot grow indefinitely.

The active controller stores the metadata in an event-sourced log topic while the other controllers within the quorum follow the active controller by responding to the events that it creates. The new quorum controller introduced with KRaft ensures that metadata is accurately replicated across the quorum. KRaft makes use of an event-based variant of the Raft consensus algorithm, hence its name.

This new KRaft mode improves partition scalability and resiliency while simplifying deployments of Apache Kafka that now can be deployed standalone. This greatly simplifies Kafka’s architecture by consolidating responsibility for metadata into Kafka itself without the requirement of a third-party tool like Apache ZooKeeper. KRaft is the consensus protocol developed to allow metadata management directly in Apache Kafka.

In development for several years, it was released in early access in Kafka 2.8, then in preview in Kafka 3.0. In particular, this is the first release that marks KRaft (Kafka Raft) consensus protocol as production ready. The Apache Software Foundation has released Apache Kafka 3.3.1 with many new features and improvements.

Download the new version Sticky Previews 2.8