Skip to main content
This guide explains how to design a highly available (HA) Kubernetes control plane, why HA matters, and practical topology options for etcd. It keeps the original diagrams and examples while clarifying component behavior, failure scenarios, and recommended configurations for production clusters. Why HA matters
  • If the control plane (master) is lost, pods already running on worker nodes continue to serve traffic. However, cluster operations that require control-plane intervention—such as scheduling new pods, replacing failed replicas, or applying configuration changes—will fail until a control-plane component becomes available.
  • In production you should avoid single points of failure by making the control plane redundant. An HA control plane improves cluster availability for control operations, not only for running workloads.
How HA works at a glance
  • Run the same control-plane components (kube-apiserver, kube-controller-manager, kube-scheduler, and etcd) on multiple master nodes.
  • Put a load balancer in front of the kube-apiserver instances so clients (kubectl, controllers, etc.) can reach any healthy API server.
  • Use leader election for components that must be single-active (controller-manager and scheduler) to prevent duplicate actions.
  • Deploy etcd either as stacked members on control-plane nodes or on a separate external etcd cluster depending on reliability and operational requirements.
Control plane components and HA behavior kube-apiserver — active/active
  • The API server can scale horizontally because it is essentially stateless; cluster state is persisted in etcd.
  • Put a load balancer (for example HAProxy, NGINX, or a cloud load-balancer) in front of API servers. Point clients’ kubeconfigs to the load balancer address so requests route to any healthy API server. The API server listens on port 6443 by default.
Example kubeconfig cluster entry (points to the load balancer):
Controller-manager and scheduler — active/standby (leader election)
  • Controllers and the scheduler must avoid performing duplicate work. They therefore use leader election: one instance is the active leader while others remain standby. If the leader fails, another instance takes over.
  • Leader election is enabled by default and is controlled by flags like --leader-elect, --leader-elect-lease-duration, --leader-elect-renew-deadline, and --leader-elect-retry-period.
Example flags for kube-controller-manager:
The scheduler uses the same leader-election mechanism with identical flags. Leader election is typically implemented via Kubernetes API objects (leases) so that at any time only one instance is active.
A stylized diagram of two mirrored control-plane nodes labeled "Active" and "Standby," each containing Controller-Manager, Scheduler, API Server, and etcd components. It illustrates a high-availability Kubernetes control plane setup.
etcd topologies: stacked vs external etcd is the single source of truth for cluster state. Choosing the right etcd topology affects availability, operational complexity, and recovery procedures. Two common deployment patterns:
  1. Stacked control plane topology (etcd on each control plane node)
  • Each master hosts an etcd member alongside kube-apiserver, kube-controller-manager, and kube-scheduler.
  • Simpler deployment footprint and fewer machines to manage, but a failed master removes an etcd member which may reduce cluster quorum, depending on cluster size.
A slide titled "Stacked Topology" showing two master-node diagrams, each containing etcd, API Server, Controller-Manager, and Scheduler components. A bullet list at right notes pros (easier to set up/manage, fewer servers) and a con (risk during failures).
  1. External etcd topology (separate etcd cluster)
  • etcd runs on dedicated servers separate from control-plane nodes.
  • This reduces coupling: a control-plane node failure does not remove etcd members and does not directly affect etcd quorum.
  • Requires more servers and a separate operational procedure for etcd backups, restores, and scaling.
etcd configuration and API server integration
  • The kube-apiserver is the only control-plane component that communicates directly with etcd. Provide it with the list of etcd server endpoints via the --etcd-servers flag.
Example kube-apiserver flag:
  • etcd uses Raft consensus. Clients may read from and write to any member, but quorum (majority) is required for commits. For production clusters, use an odd number of etcd members (typically 3 or 5) to tolerate failures while minimizing split-brain risk.
For production etcd clusters, follow quorum best practices: use an odd number of members (3 or 5 typically) to tolerate failures while minimizing split-brain risk. Also implement regular backups and test restores as part of your operational runbook.
Putting the pieces together: converting a single-master cluster to HA
  • Add additional master nodes that run the same control-plane components.
  • Deploy a load balancer in front of kube-apiserver instances and update kubeconfigs to point at the load balancer.
  • Ensure controller-manager and scheduler are configured for leader election (default behavior).
  • Choose an etcd topology (stacked or external) based on your reliability goals and available infrastructure:
    • Stacked: fewer machines, simpler setup, but consider quorum impact when masters fail.
    • External: better isolation, more resilient etcd cluster, but more machines and operational complexity.
  • Monitor control-plane components, etcd quorum status, and load-balancer health. Test failover scenarios and restore procedures before promoting the cluster to production.
Further reading and references This enhanced explanation preserves the original diagrams and examples while clarifying behaviors and configuration options for an HA Kubernetes control plane.

Watch Video