Skip to main content
In this lesson we’ll cover etcd in a high-availability (HA) configuration — a critical part of running a resilient Kubernetes control plane. We’ll briefly recap what etcd is, then focus on cluster configuration, Raft consensus, leader election, quorum, deployment topology, and practical installation steps.

What is etcd?

etcd is a distributed, reliable key-value store designed for simplicity, security, and speed. It stores structured data (JSON, YAML, etc.) as individual objects (documents) keyed by a path. Each key holds an independent value, so updates to one key do not affect others.
A slide titled "Key-Value Store" showing several small tables of key–value pairs (Name, Age, Location, Salary) with example entries such as John Doe, Dave Smith, Aryan Kumar, Lauren Rob, and Lily Oliver. The layout illustrates storing person attributes as keys and values.
Examples of stored documents (JSON):
As datasets grow in complexity, structured formats like JSON/YAML make it easy to store and query hierarchical information — which etcd handles efficiently.

etcd is distributed

etcd is commonly deployed as a cluster across multiple nodes to provide durability and availability of critical control-plane state (for example, Kubernetes metadata). Each member maintains a full copy of the data, so reads can be served locally and writes are replicated across members to ensure consistency.
A dark-themed diagram titled "Consistent" showing three identical server boxes, each labeled "2379" with a stacked-database icon and up/down "READ/WRITE" arrows above them. It visually represents three consistent nodes handling read/write operations.
Read behavior:
  • Followers can serve local reads for lower latency, but those can be stale.
  • For strongly consistent (linearizable) reads, the follower must confirm with the leader.
Write behavior:
  • All writes are serialized by the leader and replicated to followers.
  • A write is considered committed only after a majority of members acknowledge it.

Write handling and leader election (Raft)

When a client issues a write, the request is handled by the cluster leader:
  • If a follower receives the client request, it forwards it to the leader.
  • The leader appends the write to its log and replicates the entry to followers.
  • The leader marks the write committed only after a majority of members confirm the log entry.
A dark-themed diagram titled "Write" showing three server/device cards each with a "2379" label and a stacked-database icon. Two labeled name inputs ("John" and "Joe") have arrows pointing down to the left and right servers, indicating write operations.
etcd implements the Raft consensus algorithm to provide:
  • Leader election: a single leader serializes writes.
  • Log replication: the leader replicates entries to followers.
  • Heartbeats: the leader periodically sends heartbeats; missing heartbeats trigger new elections.
Typical three-node election flow:
  • Nodes start without a leader and run randomized election timeouts.
  • The first node whose timeout expires becomes a candidate and requests votes.
  • If it receives votes from a majority, it becomes leader and begins sending heartbeats.
  • If the leader fails or is partitioned, followers trigger a new election.

Quorum and fault tolerance

etcd requires a quorum (majority) to commit writes. For n members: quorum = floor(n / 2) + 1 Notes:
  • A write commits only when the quorum has persisted the entry.
  • Followers that were offline will catch up when they rejoin.
  • Even-sized clusters are more vulnerable to split-brain scenarios because they can split into equal halves without a quorum.
Prefer odd-sized etcd clusters (3 or 5 members). Odd sizes maximize the chance one partition contains a quorum, reducing downtime risk during network partitions.
Do not run production etcd on a single node. For HA, use at least 3 members and ensure you have a tested backup and recovery plan for etcd data.
Practical guidance:
  • Minimum HA cluster: 3 members (tolerates one failure).
  • Typical production choices: 3 or 5 members depending on fault-tolerance needs.
  • Avoid more than 5 members unless required — larger clusters increase coordination overhead.

Design choices and topology

Two common topologies:
  • Stacked topology: etcd runs on the same machines as the Kubernetes control-plane components (API server, controller-manager, scheduler). Simple and common for small/medium clusters.
  • Dedicated topology: etcd runs on separate, dedicated machines for higher isolation, security, and performance in larger deployments.
Choose topology based on scale, security, and operational complexity.
A slide titled "Number of Nodes" showing a table of Instances, Quorum, and Fault Tolerance on the left, and a Kubernetes control-plane diagram on the right with etcd replicas and master nodes containing API Server, Controller‑Manager, and Scheduler.

Installing etcd (binaries and certificates)

Install etcd by downloading the release tarball, extracting the binaries, and installing them on each member node:
Create directories and place TLS certificates (generate these beforehand — CA, server cert/key, peer cert/key):
Important: etcd should be configured to use mutual TLS (mTLS) for client and peer communication to secure cluster traffic.

systemd service example

Below is a typical systemd unit ExecStart snippet for an etcd member. Replace the environment placeholders (ETCD_NAME, INTERNAL_IP, PEER1_IP, PEER2_IP) with real values on each node.
The --initial-cluster parameter tells each member which peers belong to the initial cluster so they can discover each other during bootstrap.

Using etcdctl

Use the etcdctl CLI for key operations. Ensure the etcdctl version matches the server version, and configure endpoints and TLS environment variables appropriately. Examples:
Tip: For automation, set environment variables such as ETCDCTL_API, ETCDCTL_ENDPOINTS, and TLS variables to avoid repeating flags.

Design summary

  • Minimum recommended members for HA: 3 (tolerates one failure).
  • Prefer odd-sized clusters (3 or 5) to reduce the chance that partitions will leave every partition without a quorum.
  • For small deployments, running etcd on control-plane nodes (stacked topology) is common; in larger or more secure deployments, run etcd on dedicated nodes.
  • Choose cluster size based on required fault tolerance and operational cost. 3 nodes = basic HA, 5 nodes = higher tolerance; more than 5 is rarely needed.
In this lesson we recommend a three-member etcd cluster for HA. For the lab/demo we may use fewer nodes due to resource constraints, but for production always use at least three members and test failure/recovery scenarios. That’s it for this lesson.

Watch Video