What is etcd?
etcd is a distributed, reliable key-value store designed for simplicity, security, and speed. It stores structured data (JSON, YAML, etc.) as individual objects (documents) keyed by a path. Each key holds an independent value, so updates to one key do not affect others.
etcd is distributed
etcd is commonly deployed as a cluster across multiple nodes to provide durability and availability of critical control-plane state (for example, Kubernetes metadata). Each member maintains a full copy of the data, so reads can be served locally and writes are replicated across members to ensure consistency.
- Followers can serve local reads for lower latency, but those can be stale.
- For strongly consistent (linearizable) reads, the follower must confirm with the leader.
- All writes are serialized by the leader and replicated to followers.
- A write is considered committed only after a majority of members acknowledge it.
Write handling and leader election (Raft)
When a client issues a write, the request is handled by the cluster leader:- If a follower receives the client request, it forwards it to the leader.
- The leader appends the write to its log and replicates the entry to followers.
- The leader marks the write committed only after a majority of members confirm the log entry.

- Leader election: a single leader serializes writes.
- Log replication: the leader replicates entries to followers.
- Heartbeats: the leader periodically sends heartbeats; missing heartbeats trigger new elections.
- Nodes start without a leader and run randomized election timeouts.
- The first node whose timeout expires becomes a candidate and requests votes.
- If it receives votes from a majority, it becomes leader and begins sending heartbeats.
- If the leader fails or is partitioned, followers trigger a new election.
Quorum and fault tolerance
etcd requires a quorum (majority) to commit writes. For n members: quorum = floor(n / 2) + 1
Notes:
- A write commits only when the quorum has persisted the entry.
- Followers that were offline will catch up when they rejoin.
- Even-sized clusters are more vulnerable to split-brain scenarios because they can split into equal halves without a quorum.
Prefer odd-sized etcd clusters (3 or 5 members). Odd sizes maximize the chance one partition contains a quorum, reducing downtime risk during network partitions.
Do not run production etcd on a single node. For HA, use at least 3 members and ensure you have a tested backup and recovery plan for etcd data.
- Minimum HA cluster: 3 members (tolerates one failure).
- Typical production choices: 3 or 5 members depending on fault-tolerance needs.
- Avoid more than 5 members unless required — larger clusters increase coordination overhead.
Design choices and topology
Two common topologies:- Stacked topology: etcd runs on the same machines as the Kubernetes control-plane components (API server, controller-manager, scheduler). Simple and common for small/medium clusters.
- Dedicated topology: etcd runs on separate, dedicated machines for higher isolation, security, and performance in larger deployments.

Installing etcd (binaries and certificates)
Install etcd by downloading the release tarball, extracting the binaries, and installing them on each member node:systemd service example
Below is a typical systemd unit ExecStart snippet for an etcd member. Replace the environment placeholders (ETCD_NAME, INTERNAL_IP, PEER1_IP, PEER2_IP) with real values on each node.
--initial-cluster parameter tells each member which peers belong to the initial cluster so they can discover each other during bootstrap.
Using etcdctl
Use the etcdctl CLI for key operations. Ensure the etcdctl version matches the server version, and configure endpoints and TLS environment variables appropriately. Examples:ETCDCTL_API, ETCDCTL_ENDPOINTS, and TLS variables to avoid repeating flags.
Design summary
- Minimum recommended members for HA: 3 (tolerates one failure).
- Prefer odd-sized clusters (3 or 5) to reduce the chance that partitions will leave every partition without a quorum.
- For small deployments, running etcd on control-plane nodes (stacked topology) is common; in larger or more secure deployments, run etcd on dedicated nodes.
- Choose cluster size based on required fault tolerance and operational cost. 3 nodes = basic HA, 5 nodes = higher tolerance; more than 5 is rarely needed.
Links and references
- etcd project: https://etcd.io/
- etcd on GitHub: https://github.com/etcd-io/etcd
- Raft consensus algorithm: https://raft.github.io/
- Kubernetes etcd considerations: https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/