Skip to main content
This lesson explains how to safely take nodes out of a Kubernetes cluster for maintenance—OS upgrades, package updates, security patches—and the cluster behaviors and commands you should use to avoid downtime. When a node goes down, the pods scheduled on it become inaccessible. How users are impacted depends on how those pods were deployed:
  • If a Deployment or ReplicaSet manages multiple replicas (e.g., the blue app), other replicas often continue serving traffic so users are unaffected.
  • If an application runs as a single standalone pod (e.g., the green app), users will observe downtime when that pod becomes unavailable.
Kubernetes uses node health signals, taints, and tolerations to decide what happens next.

​
Node health, taints, and eviction behavior

  • If a node briefly becomes unreachable and the kubelet resumes quickly, pods simply continue running.
  • If the node remains unreachable longer than the eviction timeout, the control plane marks pods as failed and evicts them.
  • When the node controller detects a node is NotReady/unreachable it applies a NoExecute taint (for example, node.kubernetes.io/unreachable). Pods that do not tolerate this taint will be evicted after a grace period.
A pod can tolerate this taint and limit how long it can stay on an unhealthy node using tolerationSeconds. Example:
The node controller eviction timeout is controlled by kube-controller-manager’s --pod-eviction-timeout (default 5m0s). If the node becomes available again before this timeout, pods remain where they are; if not, pods are evicted and controllers recreate replacements on other nodes. Important distinctions:
  • Pods managed by controllers (Deployment, ReplicaSet, StatefulSet, DaemonSet, etc.) will be recreated on other nodes when evicted.
  • Standalone pods (not managed by a controller) will not be recreated after eviction.
  • Static pods (managed by kubelet) are recreated on the original node when the kubelet returns.
Typical sequence when a node goes offline:
  1. Node becomes NotReady.
  2. Node controller applies a NoExecute taint.
  3. The controller waits up to --pod-eviction-timeout (default 5 minutes).
  4. Pods without the toleration are evicted; controller-managed pods are recreated on other nodes.

​
Planned maintenance: cordon, drain, and uncordon

For planned maintenance you should proactively move workloads off a node so downtime is controlled and predictable. Use kubectl drain to do this safely. Drain the node:
Key behaviors of kubectl drain:
  • Cordons the node (marks it unschedulable).
  • Evicts pods safely, subject to PodDisruptionBudgets (PDBs).
  • By default, it:
    • Ignores DaemonSet-managed pods.
    • Ignores mirror pods for static pods.
    • Respects PDBs and will not evict pods if that would violate the PDB.
  • Refuses to evict unmanaged (standalone) pods unless you use --force.
You can cordon a node without evicting pods:
Uncordon a node to allow scheduling again:
Notes on behavior:
  • Pods are not “moved” between nodes. Draining causes pods on the node to terminate; controllers notice missing replicas and create new pods elsewhere.
  • After maintenance and uncordon, pods recreated elsewhere will not automatically move back to the original node. Scheduling depends on the cluster scheduler and constraints.
Before draining, check for pods protected by PodDisruptionBudgets (PDBs), or critical unmanaged pods. Draining may be blocked until the cluster can maintain the availability guarantees expressed by PDBs.
Use --force and --delete-local-data with caution. Forcing eviction of unmanaged pods or deleting local data (like emptyDir) can cause data loss or application disruption. Prefer controlled replication and graceful eviction.

​
Common kubectl drain flags

​
Practical recommendations

  • For planned OS upgrades or reboots, prefer kubectl drain to ensure replicas are recreated on other nodes before you restart the node.
  • If workloads are highly available (sufficient replicas) and you expect the node to return quickly, a short reboot without draining may be acceptable—but it risks eviction if the node exceeds the eviction timeout.
  • Verify PodDisruptionBudgets before draining to avoid unexpected blocking.
  • Remember that static pods and mirror pods behave differently—static pods are re-created by kubelet on the same node when it returns.
  • Test draining and uncordoning in a staging environment to validate application behavior and PDB interactions.

​
Quick reference commands

  • Drain a node:
    • kubectl drain node-1
  • Cordon a node (prevent new pods from being scheduled):
    • kubectl cordon node-1
  • Uncordon a node (allow scheduling again):
    • kubectl uncordon node-1
Practice draining, cordoning, and uncordoning nodes in a test cluster or lab environment to get comfortable with how your apps and PDBs behave during maintenance.

Watch Video

Practice Lab