> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# K8s Cluster Upgrade
> Staged Kubernetes cluster upgrade plan prioritizing control plane first, then sequential worker node upgrades using cordon, drain and PodDisruptionBudgets to avoid downtime.
This is a common Kubernetes interview / operational scenario:
You manage a 40-node Kubernetes production cluster that must stay online 24/7. The cluster is running an older Kubernetes version that is nearing end-of-life. You need to upgrade the entire cluster without causing downtime. The safest approach is a staged upgrade.
A Kubernetes cluster has two logical parts:
* Control plane — the cluster “brain” (API server, scheduler, controller-manager, etcd). It makes scheduling and cluster-control decisions.
* Worker nodes — the cluster “muscle” (kubelet, kube-proxy, and the pods that serve traffic).
Key compatibility rule: worker node kubelets must never be newer than the control plane. However, the control plane can run a newer minor version than kubelets. That allows you to upgrade the control plane first while worker nodes continue serving traffic.
Table — Quick comparison
| Component | Role | Notes |
| - | - | - |
| Control plane | Cluster API, scheduler, controller-manager, etcd | Upgrade first; coordinates the cluster |
| Worker nodes | Run pods and serve traffic | Upgrade after control plane; kubelet must not be newer than control plane |
Recommended staged plan
1. Upgrade the control plane.
2. Upgrade worker nodes one at a time (or in small safe batches).
3. Use PodDisruptionBudgets (PDBs) and cordon/drain to protect availability.
Stage 1 — Upgrade the control plane
Upgrade control plane nodes first. Pods on worker nodes keep serving traffic even if control plane components are temporarily being upgraded. This minimizes risk to running workloads.
Stage 2 — Upgrade worker nodes, one node at a time
After the control plane upgrade completes, upgrade worker nodes with the following sequence for each node:
1. Cordon the node so the scheduler stops placing new pods on it.
2. Drain the node so existing pods are gracefully evicted and rescheduled elsewhere.
3. Perform the node upgrade (OS/kernel, kubelet/kube-proxy; or replace the VM/instance).
4. Uncordon the node so it can accept new pods again.
Common commands:
```bash theme={null}
kubectl cordon
kubectl drain --ignore-daemonsets --delete-local-data
# perform the node upgrade (varies by environment)
kubectl uncordon
```
Summary table — Node maintenance commands
| Action | Command | Notes |
| - | - | - |
| Prevent scheduling | `kubectl cordon ` | Marks node unschedulable |
| Evict pods and drain | `kubectl drain --ignore-daemonsets --delete-local-data` | `--ignore-daemonsets` required; `--delete-local-data` only if local data can be discarded |
| Re-enable scheduling | `kubectl uncordon ` | Node becomes schedulable again |
Protecting application availability with PodDisruptionBudget (PDB)
Before draining a node, ensure applications remain available. Example: a checkout service has three replica pods and two of them are on the node you plan to drain. If Kubernetes evicts both replicas simultaneously, capacity could drop to a single pod and impact users.
Create a PodDisruptionBudget to guarantee a minimum number of available replicas during voluntary disruptions:
```yaml theme={null}
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: checkout-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: checkout
```
With this PDB, Kubernetes will block evicting more than one checkout pod at a time, forcing the drain process to wait until replacements are ready on other nodes. That preserves availability at the cost of a longer upgrade window.
When running `kubectl drain`, include `--ignore-daemonsets` (daemonset pods are not evicted) and `--delete-local-data` only if you are certain local data can be discarded. If pods are not managed by a controller (Deployment, ReplicaSet, StatefulSet, etc.), drains may be blocked — consider adding or converting pods to be controller-managed.
Best practices and operational tips
* Always upgrade the control plane first, then worker nodes.
* Upgrade nodes one-by-one, or in small batches if you’ve validated capacity and PDBs.
* Use PDBs to prevent simultaneous eviction of critical replicas.
* Prefer replacing nodes (create new, join to cluster, drain and remove old) when using cloud/managed environments — this reduces risk and is commonly used by managed Kubernetes providers.
* Test the upgrade process in a staging environment that mirrors production.
Compatibility warning: kubelets (node agents) must never be newer than the control plane. Upgrading worker nodes before the control plane can cause version skew issues. Always verify Kubernetes version skew policy for your target versions.
Further reading
* Kubernetes official upgrade guide: [https://kubernetes.io/docs/tasks/administer-cluster/cluster-upgrade/](https://kubernetes.io/docs/tasks/administer-cluster/cluster-upgrade/)
* PodDisruptionBudget docs: [https://kubernetes.io/docs/concepts/workloads/pods/disruptions/](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/)
* Draining nodes: [https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/](https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/)
Following this staged approach — control plane first, then worker nodes using cordon/drain with appropriate PDBs — is a common, safe method to upgrade a busy production Kubernetes cluster without causing downtime.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.