> ## Documentation Index > Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt > Use this file to discover all available pages before exploring further. # Lab Solution OS Upgrades optional > Guide to safely cordoning, draining, and restoring Kubernetes nodes, handling DaemonSets and unmanaged pods, and best practices for controller-managed workloads during maintenance This lesson walks through safe cluster maintenance operations: inspecting cluster state, cordoning and draining nodes for maintenance, handling special cases (DaemonSets and pods not managed by controllers), and restoring nodes after work is complete. It explains why some pods cannot be evicted automatically and shows safer alternatives for critical workloads. Prerequisites * kubectl configured to talk to the target cluster. * Basic familiarity with Pods, Deployments, DaemonSets, and node taints. * Optional: set a short alias for kubectl to speed commands. Optional helper ```bash theme={null} # Optional helper alias alias k=kubectl ``` Inspect the cluster and current placements * Check nodes: ```bash theme={null} k get nodes ``` Example output: ```console theme={null} NAME STATUS ROLES AGE VERSION controlplane Ready control-plane,master 15m v1.20.0 node01 Ready,SchedulingDisabled 14m v1.20.0 ``` * Check deployments: ```bash theme={null} k get deploy ``` Example output: ```console theme={null} NAME READY UP-TO-DATE AVAILABLE AGE blue 3/3 3 3 10m ``` * Inspect pods and their node assignments: ```bash theme={null} k get pods -o wide ``` Example output showing three pods scheduled on node01: ```console theme={null} NAME READY STATUS RESTARTS AGE IP NODE blue-746c87566d-aaa 1/1 Running 0 5m 10.244.1.2 node01 blue-746c87566d-bbb 1/1 Running 0 5m 10.244.1.3 node01 blue-746c87566d-ccc 1/1 Running 0 5m 10.244.1.4 node01 ``` Drain node01 for maintenance (DaemonSets present) * To prepare node01 for maintenance, use `kubectl drain`. The first attempt may fail when DaemonSets are present because `kubectl drain` does not delete DaemonSet-managed pods by default: ```bash theme={null} k drain node01 ``` Example error: ```console theme={null} There are pending nodes to be drained: node01 error: cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-system/kube-flannel-ds-4k7hj, kube-system/kube-proxy-5hcwr ``` * Re-run drain with `--ignore-daemonsets` to evict normal pods while leaving DaemonSet-managed pods running on the node: ```bash theme={null} k drain node01 --ignore-daemonsets ``` Example output: ```console theme={null} node/node01 already cordoned WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-flannel-ds-4k7hj, kube-system/kube-proxy-5hcwr evicting pod default/blue-746c87566d-aaa evicting pod default/blue-746c87566d-bbb evicting pod default/blue-746c87566d-ccc pod/blue-746c87566d-aaa evicted pod/blue-746c87566d-bbb evicted pod/blue-746c87566d-ccc evicted node/node01 evicted ``` * Verify the pods were rescheduled (in this environment, they move to the control plane node): ```bash theme={null} k get pods -o wide ``` Example output: ```console theme={null} NAME READY STATUS RESTARTS AGE IP NODE blue-746c87566d-xxx 1/1 Running 0 63s 10.244.0.5 controlplane blue-746c87566d-yyy 1/1 Running 0 63s 10.244.0.6 controlplane blue-746c87566d-zzz 1/1 Running 0 63s 10.244.0.4 controlplane ``` * The node is cordoned (marked `SchedulingDisabled`): ```bash theme={null} k get nodes ``` Example: ```console theme={null} NAME STATUS ROLES AGE VERSION controlplane Ready control-plane,master 15m v1.20.0 node01 Ready,SchedulingDisabled 14m v1.20.0 ``` Bring the node back after maintenance * When maintenance is complete, make the node schedulable again: ```bash theme={null} k uncordon node01 ``` * Verify node status: ```bash theme={null} k get nodes ``` You should no longer see `SchedulingDisabled` for node01. Why pods did not move back to node01 automatically * Kubernetes will not migrate already-running pods back to their original node after you uncordon it. The scheduler only places new pods; existing pods remain where they are until they are deleted or evicted and recreated. * Example: after uncordon, pods may continue to run on the control plane: ```bash theme={null} k get pods -o wide ``` Example: ```console theme={null} NAME READY STATUS RESTARTS AGE IP NODE blue-746c87566d-xxx 1/1 Running 0 10m 10.244.0.5 controlplane ``` Why pods were allowed on the control plane node * Control plane (master) nodes are commonly tainted to prevent scheduling of regular workloads. In this environment the control plane node has no taints, so regular pods were allowed to be scheduled there. * Example excerpt from `kubectl describe node controlplane`: ```console theme={null} Taints: Unschedulable: false ``` Annotations (excerpt): ```console theme={null} flannel.alpha.coreos.com/backend-data: `{"VNI":1,"VtepMAC":"1e:4c:dc:69:ad:e5"}` flannel.alpha.coreos.com/backend-type: vxlan flannel.alpha.coreos.com/kube-subnet-manager: true flannel.alpha.coreos.com/public-ip: 10.42.115.9 kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock ``` (Note: the JSON-like annotation above is wrapped in backticks to avoid MDX parsing.) Second maintenance window: a non-controller-managed pod blocks drain * If a Pod on the node is not managed by a controller (for example, it was created via `kubectl run` or a plain Pod manifest), `kubectl drain` will refuse to delete it because deleting an unmanaged Pod can result in permanent loss of local state: ```bash theme={null} k drain node01 --ignore-daemonsets ``` Example error: ```console theme={null} node/node01 cordoned error: unable to drain node "node01", aborting command... There are pending nodes to be drained: node01 error: cannot delete Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet (use --force to override): default/hr-app ``` * Inspect the pods on the node to find the unmanaged pod: ```bash theme={null} k get pods -o wide ``` Example: ```console theme={null} NAME READY STATUS RESTARTS AGE IP NODE blue-... 1/1 Running 0 5m34s 10.244.0.5 controlplane hr-app 1/1 Running 0 78s 10.244.1.5 node01 ``` Why drain failed and the implications of forcing * `kubectl drain` refuses to delete pods not managed by a controller to avoid accidental deletion of stateful or locally-stored data. Controller-managed Pods (ReplicaSet, Deployment, StatefulSet, Job, DaemonSet) can safely be evicted because the controller will recreate them elsewhere. * You can override this behavior with `--force`, but this will delete the unmanaged pod and may cause permanent data loss: ```bash theme={null} k drain node01 --ignore-daemonsets --force ``` Example output (forced eviction): ```console theme={null} node/node01 already cordoned WARNING: deleting Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: default/hr-app; ignoring DaemonSet-managed Pods: kube-system/kube-flannel-ds-4k7hj, kube-system/kube-proxy-5hcwr evicting pod default/hr-app pod/hr-app evicted node/node01 evicted ``` Safer approach: make critical apps controller-managed and cordon the node * Best practice: deploy critical applications using controllers (Deployment, StatefulSet, etc.) so they can be safely evicted and rescheduled elsewhere during maintenance. * After converting `hr-app` to a Deployment, verify controllers and pods: ```bash theme={null} k get deployments k get pods -o wide ``` Example: ```console theme={null} # deployment present NAME READY UP-TO-DATE AVAILABLE AGE hr-app 1/1 1 1 2m # pods NAME READY STATUS RESTARTS AGE IP NODE hr-app-abcdef 1/1 Running 0 2m 10.244.1.6 node01 ``` * If you want to keep the existing critical `hr-app` running on node01 but prevent any new pods from being scheduled there, use `cordon` (do not drain). Cordon marks the node unschedulable without evicting running pods: ```bash theme={null} k cordon node01 ``` Example: ```console theme={null} node/node01 cordoned ``` * Confirm the node is cordoned: ```bash theme={null} k get nodes ``` Example: ```console theme={null} NAME STATUS ROLES AGE VERSION controlplane Ready control-plane,master 20m v1.20.0 node01 Ready,SchedulingDisabled 19m v1.20.0 ``` Quick reference: common maintenance commands | Command | Purpose | | - | - | | `k cordon ` | Mark a node unschedulable to prevent new pods being scheduled. | | `k uncordon ` | Make a node schedulable again. | | `k drain ` | Evict pods from a node in preparation for maintenance. Refuses to delete DaemonSets and unmanaged Pods by default. | | `k drain --ignore-daemonsets` | Evict regular pods while leaving DaemonSet pods. | | `k drain --ignore-daemonsets --force` | Force deletion of unmanaged Pods (risky—may lose local data). | | `k get pods -o wide` | See pod-to-node assignments. | | `kubectl describe node ` | Inspect taints and other node details. | Why convert single pods to controller-managed workloads? * Controller-managed workloads (Deployments, ReplicaSets, StatefulSets) are recreated automatically if evicted or deleted. This enables safer maintenance and avoids manual intervention. * If your workload stores local state that cannot be replicated, consider alternatives: StatefulSets with persistent volumes, relocating state to networked storage, or planning maintenance windows with careful backups. Summary * Use `kubectl drain` to prepare a node for maintenance. Add `--ignore-daemonsets` to skip DaemonSet-managed pods. * `kubectl drain` will refuse to delete Pods not managed by a controller; `--force` will override but is risky. * Convert critical single Pods into controller-managed workloads so they can be safely evicted and recreated. * Use `kubectl cordon` to prevent scheduling of new pods on a node while leaving existing pods running; use `kubectl uncordon` to restore scheduling. Links and References * [Kubernetes: kubectl drain docs](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#drain) * [Kubernetes: Managing pods and controllers](https://kubernetes.io/docs/concepts/workloads/controllers/) * [Kubernetes: Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) Always prefer controller-managed workloads for production-critical applications. This enables safe node maintenance without risking permanent loss of pods or local data. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.