By Shreyas Kaundinya
Kafka with KRaft Architecture
Kafka with Zookeeper Architecture
What is Zookeeper?
- Zookeeper is a centralised coordination service for maintaining
configuration that provides distributed synchronisation using Zookeeper
Atomic Broadcast - ZooKeeper provided the authoritative store of metadata holding the system’s
most important facts: where partitions live, which replica is the leader,
and so forth. - It served as a critical controller for the entire system
- Zookeeper is an additional entity in the system
- This required the users of Kafka having to maintain two systems, two network
layers, two security implementations - They need individual tuning to be done
- Monitoring also becomes an issue as they are two separate entities
What’s KRaft?
- An alternative to Zookeeper, that is introduced in the Kafka Improvement
Proposal - 500 (KIP-500) that is being brought in through the version 2.8 - Uses the Raft Consensus Protocol to replicate metadata among different
Broker nodes - Uses event sourcing variant of Raft
- Improves stability, simplifies the software
- Easier to monitor, administer, and support Kafka
- Single security model instead of having to manage for Zookeeper and Kafka
separately - Controller failover near real time
How the new Quorum Controller works?
- Uses the new KRaft consensus algorithm to replicate metadata amongst the
nodes in the network - Here’s a visualization of how raft works :
https://thesecretlivesofdata.com/raft/ - Uses event sourcing, which essentially works on the model of state/event
based storage model in an append only mode. - Logs are periodically abridged snapshots to guarantee that they are not
growing indefinitely - The leader maintains the log of events and the passive nodes follow the
leader to get the updates - If a leader fails,
- leader election occurs and the new leader takes over
- The availability of the system is greatly increased
(Source :
https://developer.confluent.io/learn/kraft/)