Kafka Needs No Keeper Colin McCabe 2 Introduction Kafka has - PowerPoint PPT Presentation

94 Instant Failover - Low-latency failover via Raft election - Standbys contain all data in memory Controller - Brokers do not need to re-fetch Controller Broker offset=1 Controller Broker offset=2

95 Metadata Caching - Brokers can persist metadata to disk - Only fetch what they need Controller - Use snapshots if we’re too far behind Controller Broker /mnt/logs/kafka/metadata Controller offset=1 Broker /mnt/logs/kafka/metadata offset=2

96 Broker Registration - Building a map of the cluster Controller - What brokers exist in the cluster? - How can they be reached?

97 Broker Registration - Brokers send heartbeats to the active Controller controller - The controller uses this to build a map of the cluster

98 Broker Registration - Brokers send heartbeats to the active Controller controller - The controller uses this to build a map of the cluster - The controller also tells brokers if they should be fenced or shut down

99 Fencing - Brokers need to be fenced if they’re Controller partitioned from the controller, or can’t keep up - Brokers self-fence if they can’t talk to the controller

100 Handling network partitions