Skip to main content
In this lesson we cover running etcd in a high‑availability (HA) configuration for Kubernetes. Topics include a quick recap of what etcd is, how a distributed etcd cluster works (Raft leader/follower model), quorum and fault tolerance, installation notes, a minimal systemd example, and basic etcdctl usage. What is etcd? etcd is a distributed, reliable key‑value store that is simple, secure, and fast. Instead of rows and columns, etcd stores documents (values) keyed by unique keys. Each document may contain structured data (JSON, YAML) and updates to one key do not affect others.
A slide titled "Key-Value Store" showing several two-column tables labeled "Key" and "Value" with entries for Name, Age, Location and Salary for different example records.
Example records (JSON documents):
Because etcd stores critical cluster state for Kubernetes, it is built to run distributed across multiple servers so data remains available even when nodes fail.
A dark blue slide showing a glowing rounded rectangle that reads "ETCD is a distributed, reliable key-value store that is Simple, Secure, and Fast." To the right is a large blue gear-shaped icon with a simple face and a small "© Copyright KodeKloud" notice in the bottom-left.
How a distributed etcd cluster works When etcd runs on multiple servers, each member keeps a copy of the data. Reads can be served from any member, but followers may lag behind the leader — so reads from followers can be stale. For the strongest consistency (linearizable reads) query the leader or use etcd’s linearizable read mode. Writes use the Raft consensus algorithm with a leader/follower model:
  • A single leader is elected; all writes go through the leader.
  • If a client writes to a follower, the follower forwards the request to the leader.
  • The leader appends the write as a log entry and replicates it to followers.
  • The leader considers the write committed only after a majority (quorum) acknowledges persistence.
  • Once committed, the change is applied to the state machine on each member.
This design produces a single, consistent sequence of writes and prevents split‑brain scenarios.
A dark-themed infographic titled "Write" showing three mobile-style server boxes with a stacked-database icon and the number "2379." Arrows from name tabs labeled "John" and "Joe" point to the left and right servers, illustrating write operations to database replicas.
Leader election and Raft (3‑node example) Raft handles leader election using randomized election timeouts:
  • Each node starts with a randomized election timeout.
  • The first node whose timeout expires becomes a candidate and requests votes.
  • If it receives votes from a majority of members, it becomes leader.
  • Leader sends periodic heartbeats (AppendEntries RPCs) to prevent new elections.
  • If heartbeats stop (leader crash or partition), followers trigger a new election.
If the leader fails or is partitioned, the remaining quorum elects a new leader automatically. Quorum and fault tolerance Quorum is the minimum number of members required to make progress (commit writes). For N members: quorum = floor(N / 2) + 1 Examples: Key takeaways:
  • A 3‑node cluster tolerates 1 node failure.
  • A 5‑node cluster tolerates 2 node failures.
  • A 2‑node cluster is not recommended: quorum is 2, so losing one node prevents commits.
Use an odd number of members (3, 5, 7 …) to minimize the chance of a partition leaving no majority.
Use an odd number of etcd members (minimum 3) for HA. Three is a good starting point; five increases fault tolerance to two node failures.
A slide titled "Number of Nodes" showing a table of Instances, Quorum, and Fault Tolerance for 1–7 nodes. To the right is a diagram of a Kubernetes control plane with etcd instances and master nodes containing API Server, Controller-Manager, and Scheduler.
Deployment notes and ports
  • Client communication: port 2379
  • Peer (cluster) communication: port 2380
  • When creating members, provide the initial cluster membership so each member knows the other peers and the cluster token/state.
  • Ensure certificate SANs include the advertise/listen IPs and hostnames for TLS.
Ports quick reference: Installing etcd (binary-based example) A straightforward installation approach is to download the official release tarball, extract it, and move the binaries to a system path. Replace the version below with the latest supported release for production.
Place TLS certificates under /etc/etcd/ (paths in the systemd example below assume this). Make sure certificates are generated using your PKI process and that SANs match the advertise/listen URLs. Example systemd unit (etcd.service) A minimal systemd unit demonstrates important flags. The --initial-cluster option lists all peers in the bootstrap. Update names/IPs and file paths to match your environment.
Notes:
  • Replace peer-1/peer-2/peer-3 and IP addresses with your hostnames/IPs.
  • Ensure TLS certificate paths, auth flags, and advertise/listen URLs match your certificates and network.
  • Use --initial-cluster-state new for a fresh cluster. Use existing when re‑joining an existing cluster.
Certificates must include correct SANs that match advertise/listen hostnames and IPs. Mismatched SANs will cause peer/client TLS handshakes to fail and prevent cluster formation.
Using etcdctl for basic operations Use etcdctl v3 API to interact with the cluster. If needed, set ETCDCTL_API=3. For secured clusters pass TLS flags or set environment variables. Examples:
Useful troubleshooting commands:
Design guidance recap
  • Minimum recommended HA cluster: 3 etcd members (odd number).
  • 3 members tolerate 1 node failure; 5 members tolerate 2.
  • Avoid 2‑node clusters — they do not improve availability.
  • Choose between stacked topology (etcd co‑located on control-plane nodes) or an external dedicated etcd cluster based on operational needs, fault tolerance, and cost.
Practical note For demos or local testing you may run fewer nodes for resource reasons, but for production HA deploy at least three etcd members on separate, resilient control-plane hosts. Further reading and references That concludes this lesson on etcd in high availability mode.

Watch Video