Cluster Maintenance Section Introduction

In this lesson we cover the essential cluster maintenance practices required to run Kubernetes reliably in production. You will learn:

How to manage operating system upgrades and their availability implications for cluster nodes.
The safe, recommended sequence to evacuate workloads from a node (planned maintenance) and how to recover nodes so live applications remain healthy.
Kubernetes upgrade planning: how releases and versioning work, compatibility constraints between control plane and kubelet versions, and best practices for selecting which versions to target.
A hands‑on, end‑to‑end cluster upgrade that demonstrates coordinating control plane and node upgrades while minimizing disruption.
Backup and restore strategies for both cluster state and application data, followed by a disaster‑recovery exercise to validate your procedures.

This lesson combines conceptual guidance, prescriptive best practices, and practical procedures so you can maintain and upgrade Kubernetes clusters with minimal downtime and predictable outcomes.

Plan maintenance windows and backups before any OS or Kubernetes upgrade. Always test upgrades in a staging environment that mirrors production to validate your upgrade and rollback procedures.

1. Node maintenance: unplanned vs. planned events

Understanding different node maintenance scenarios helps you choose the right mitigation steps.

Scenario	Impact	Recommended action
Unplanned node failure (hardware, crash)	Node leaves cluster unexpectedly, pods are rescheduled if capacity exists	Monitor & alert. Replace or repair node. Verify pod rescheduling and data consistency.
Planned maintenance (OS patch, firmware, kernel upgrade)	Node intentionally taken out of service	Drain the node, perform maintenance, verify node health, then bring it back.
Capacity rebalancing / scaling down	Pods need to migrate	Use pod disruption budgets and `kubectl drain` to coordinate safe eviction.

2. Safe sequence for draining and restoring nodes

Follow this sequence to minimize disruption:

Cordon the node to prevent new pods:
- kubectl cordon <node-name>
Drain the node to evict pods safely:
- kubectl drain <node-name> --ignore-daemonsets --delete-local-data --force
  - Use --force only when necessary; avoid deleting data for stateful workloads unless they are replicated.
  - Respect PodDisruptionBudgets (PDBs) — if a PDB blocks eviction, either wait or adjust the budget for maintenance windows.
Perform OS or firmware maintenance, or reboot the node.
After maintenance, ensure node readiness and kubelet health:
- Check node status: kubectl get nodes
- Inspect system and kubelet logs: journalctl -u kubelet
Uncordon the node to allow scheduling:
- kubectl uncordon <node-name>
Verify pods are scheduling and application health is intact:
- kubectl get pods -o wide
- Run application-level health checks and monitor metrics.

Use PodDisruptionBudgets and readiness probes to control how many replicas can be evicted at once and to prevent downtime during maintenance.

3. Kubernetes upgrade planning and version compatibility

Before upgrading, review these principles:

Kubernetes releases follow a regular cadence. Each minor release includes features, deprecations, and bug fixes—review release notes for breaking changes.
Version skew policy: Kubernetes supports a limited skew between control plane (API server, controller-manager, scheduler) and kubelet/node components. In practice, upgrade the control plane first, then the nodes. Always consult official release notes and the Kubernetes compatibility matrix for the precise skew policy for your versions.
Choose a target version that:
- Is a stable, non-deprecated release.
- Is supported by your cloud provider or distribution.
- Has been validated in staging for your workloads and third‑party controllers.

Recommended upgrade planning checklist:

Backup cluster state (etcd snapshot) and persistent application data.
Ensure monitoring, logging, and alerting are functional.
Confirm third‑party controllers, CRDs, and operators support the target version.
Create a rollback plan and test it in staging.

References:

4. Typical upgrade procedure (high-level)

A safe, common upgrade flow:

Backup etcd and application data (see Section 6).
Upgrade control plane components first (API server, controller-manager, scheduler).
- For kubeadm clusters: kubeadm upgrade apply <version>
- For managed control planes (GKE/EKS/AKS), use the provider’s upgrade workflow.
Verify control plane health and API compatibility.
Upgrade kube-proxy, CoreDNS/Core components if required.
Upgrade worker nodes (kubelet and kube-proxy) in a rolling fashion:
- Cordon and drain each node, upgrade kubelet and container runtime, then uncordon.
Monitor workloads and metrics after each node upgrade; roll back if issues are detected.

Important: Always upgrade the control plane before kubelets to avoid API incompatibilities.

5. Performing an end-to-end cluster upgrade (hands-on)

This exercise demonstrates upgrading a live cluster while minimizing application disruption.

Pre-upgrade:
- Run a full etcd snapshot.
- Validate application backups and test restores in staging.
- Ensure all nodes are Ready and no long-running pod restarts exist.
Control plane upgrade:
- Follow your cluster type’s documented steps (kubeadm, hosted control plane, or cloud-provided workflow).
- Confirm kubectl get componentstatuses and kubectl get cs (or equivalent) show healthy components.
Node upgrades:
- Upgrade nodes one at a time with kubectl cordon → kubectl drain → apply OS/kernel/runtime patches → restart kubelet → validate → kubectl uncordon.
Post-upgrade:
- Run integration tests, smoke tests, and performance checks.
- Monitor metrics and logs for abnormal behavior.

6. Backup and restore strategies

Backups need to cover both cluster control plane state (etcd) and application data (persistent volumes). Recommended tools and approaches:

Area	Tool / Method	Notes
Control plane state	`etcdctl snapshot save` or vendor snapshot APIs	Ensure consistent snapshot, encrypt backups, and store off-site.
Application-level backups	Velero (`https://velero.io/`)	Backs up PV snapshots, manifests, and cluster resources. Supports cloud object stores.
Database/application data	Native DB backup utilities (mysqldump, pg_dump, database snapshots)	Ensure consistent backups (quiesce DB or use transaction-consistent snapshots).

Links:

Disaster recovery exercise:

Take and verify backups of etcd and application data.
Simulate a disaster (e.g., corrupt control plane or wipe cluster nodes).
Restore etcd from snapshot or rebuild the control plane using provider docs.
Recreate resources and restore PVs/application data from Velero or database backups.
Validate application state, connectivity, and data integrity.

7. Best practices summary

Always test upgrades and restores in staging that mirrors production.
Automate backups and validate restore processes regularly.
Upgrade control plane first, then worker nodes.
Use kubectl cordon/kubectl drain and follow PodDisruptionBudget constraints.
Monitor during and after maintenance — metrics and alerts are your early warning system.
Keep an up-to-date rollback plan and practice it periodically.

This lesson provides the theory and practical steps needed to perform safe node maintenance, plan and execute Kubernetes upgrades, and recover from disasters with confidence.

Introduction

Core Concepts

Scheduling

Logging & Monitoring

Application Lifecycle Management