Brokers and Clusters

• Brokers are nothing but a Kafka service running node in a cluster.
• Brokers are can act as incharge of the cluster and make sure data is replicated across the cluster to provide resiliency to Kafka Cluster.
• Upon Kafka cluster created ( Kafka with multiple brokers ), a broker will be marked as controller node
• Controller is like a master node, responsible for all the client interactions, cluster administration.
• For example while creating topic, we create partitions, this partition creation will be taken care by Controller
• As mentioned above, for a cluster we need a leader node and will be elected one as a leader by Zookeeper.
• Producers/consumers, they interact with cluster and all their requests are processed by Leader node.
• Remaining brokers in the cluster are in sync with leader node
• The partition distribution of a topic is also managed by leader node and remember this distribution is completely random.
• Zookeeper will act as backbone for Kafka cluster, it takes care of replication, cluster failover etc.
• Zookeeper will maintain the consensus in the cluster so that each Kafka node know/aware of the cluster