Skip to main content
This lesson walks through safe cluster maintenance operations: inspecting cluster state, cordoning and draining nodes for maintenance, handling special cases (DaemonSets and pods not managed by controllers), and restoring nodes after work is complete. It explains why some pods cannot be evicted automatically and shows safer alternatives for critical workloads. Prerequisites
  • kubectl configured to talk to the target cluster.
  • Basic familiarity with Pods, Deployments, DaemonSets, and node taints.
  • Optional: set a short alias for kubectl to speed commands.
Optional helper
Inspect the cluster and current placements
  • Check nodes:
Example output:
  • Check deployments:
Example output:
  • Inspect pods and their node assignments:
Example output showing three pods scheduled on node01:
Drain node01 for maintenance (DaemonSets present)
  • To prepare node01 for maintenance, use kubectl drain. The first attempt may fail when DaemonSets are present because kubectl drain does not delete DaemonSet-managed pods by default:
Example error:
  • Re-run drain with --ignore-daemonsets to evict normal pods while leaving DaemonSet-managed pods running on the node:
Example output:
  • Verify the pods were rescheduled (in this environment, they move to the control plane node):
Example output:
  • The node is cordoned (marked SchedulingDisabled):
Example:
Bring the node back after maintenance
  • When maintenance is complete, make the node schedulable again:
  • Verify node status:
You should no longer see SchedulingDisabled for node01. Why pods did not move back to node01 automatically
  • Kubernetes will not migrate already-running pods back to their original node after you uncordon it. The scheduler only places new pods; existing pods remain where they are until they are deleted or evicted and recreated.
  • Example: after uncordon, pods may continue to run on the control plane:
Example:
Why pods were allowed on the control plane node
  • Control plane (master) nodes are commonly tainted to prevent scheduling of regular workloads. In this environment the control plane node has no taints, so regular pods were allowed to be scheduled there.
  • Example excerpt from kubectl describe node controlplane:
Annotations (excerpt):
(Note: the JSON-like annotation above is wrapped in backticks to avoid MDX parsing.) Second maintenance window: a non-controller-managed pod blocks drain
  • If a Pod on the node is not managed by a controller (for example, it was created via kubectl run or a plain Pod manifest), kubectl drain will refuse to delete it because deleting an unmanaged Pod can result in permanent loss of local state:
Example error:
  • Inspect the pods on the node to find the unmanaged pod:
Example:
Why drain failed and the implications of forcing
  • kubectl drain refuses to delete pods not managed by a controller to avoid accidental deletion of stateful or locally-stored data. Controller-managed Pods (ReplicaSet, Deployment, StatefulSet, Job, DaemonSet) can safely be evicted because the controller will recreate them elsewhere.
  • You can override this behavior with --force, but this will delete the unmanaged pod and may cause permanent data loss:
Example output (forced eviction):
Safer approach: make critical apps controller-managed and cordon the node
  • Best practice: deploy critical applications using controllers (Deployment, StatefulSet, etc.) so they can be safely evicted and rescheduled elsewhere during maintenance.
  • After converting hr-app to a Deployment, verify controllers and pods:
Example:
  • If you want to keep the existing critical hr-app running on node01 but prevent any new pods from being scheduled there, use cordon (do not drain). Cordon marks the node unschedulable without evicting running pods:
Example:
  • Confirm the node is cordoned:
Example:
Quick reference: common maintenance commands Why convert single pods to controller-managed workloads?
  • Controller-managed workloads (Deployments, ReplicaSets, StatefulSets) are recreated automatically if evicted or deleted. This enables safer maintenance and avoids manual intervention.
  • If your workload stores local state that cannot be replicated, consider alternatives: StatefulSets with persistent volumes, relocating state to networked storage, or planning maintenance windows with careful backups.
Summary
  • Use kubectl drain to prepare a node for maintenance. Add --ignore-daemonsets to skip DaemonSet-managed pods.
  • kubectl drain will refuse to delete Pods not managed by a controller; --force will override but is risky.
  • Convert critical single Pods into controller-managed workloads so they can be safely evicted and recreated.
  • Use kubectl cordon to prevent scheduling of new pods on a node while leaving existing pods running; use kubectl uncordon to restore scheduling.
Links and References
Always prefer controller-managed workloads for production-critical applications. This enables safe node maintenance without risking permanent loss of pods or local data.

Watch Video