KRaft in Action New Broker joining Kafka Cluster

​

Quick overview

• It contacts the controller leader to register.
• The controller proposes the registration change to the Raft metadata log.
• A quorum commit makes the registration authoritative and durable.
• The controller publishes updated metadata and may trigger rebalancing.

​

Step-by-step broker join flow

​

1) New broker process starts

• Broker ID (or request for automatic assignment)
• Advertised listeners
• Rack or availability-zone information (optional)
• Any additional broker-level metadata

​

2) Controller leader check and election

• If a controller leader exists, the broker will send its registration to that leader.
• If there is no leader, the controller quorum runs a Raft leader election. The elected controller leader will handle metadata changes (including this registration).

​

3) Broker sends registration to the controller leader

• Broker ID uniqueness and ranges
• Listener and advertised address validation
• Compatibility with cluster-level policies (e.g., rack-awareness constraints)

{
  "brokerId": -1,
  "host": "broker-9.example.com",
  "port": 9092,
  "advertisedListeners": ["PLAINTEXT://broker-9.example.com:9092"],
  "rack": "rack-az-3"
}

​

4) Controller proposes the registration to the Raft metadata log

• Proposes a metadata change entry (broker-registration) to the Raft metadata log.
• Sends AppendEntries to the controller quorum to replicate the proposed change.
• Waits for acknowledgements from a majority (quorum) of controllers.

​

5) Controller acknowledges the broker (post-commit)

• The controller leader responds to the broker confirming registration and the assigned broker ID (if it was assigned).
• The controller broadcasts updated cluster metadata (or ensures brokers can read it from the metadata log), so other brokers learn about the newly joined broker.
• Brokers update their local metadata cache and may perform any necessary connection setup.

​

6) Optional: Controller triggers rebalancing and partition reassignment

• Each reassignment or leader movement is a metadata change that must be proposed and committed to the Raft log.
• Once committed, brokers receive the updated partition assignments and either fetch new replicas or truncate existing replicas as needed.
• Reassignments are applied incrementally and are visible only after the metadata commit.

​

7) Final commit and new cluster state

• The metadata log persists the cluster state transitions.
• The cluster transitions to the updated state where the new broker can participate in serving replicas, accepting produce/consume traffic per the committed metadata.

​

Component responsibilities (at a glance)

​

Important notes and guarantees

• A leader-coordinated, quorum-committed metadata change ensures consistency and durability across controller failures.
• Any control-plane operation (broker join, topic/partition creation, config change) follows the same propose-and-commit Raft lifecycle.
• Rebalancing only becomes authoritative after a commit to the metadata log; until committed, changes are not visible cluster-wide.

​

Summary

• It registers with the controller leader.
• The registration is proposed to the Raft metadata log and must be committed by a quorum.
• Once committed, the broker is acknowledged and cluster metadata is updated; rebalancing follows the same commit lifecycle.

​

Links and references

• Raft consensus algorithm
• Apache Kafka KRaft documentation
• ZooKeeper project (for comparison)