
Step-by-step troubleshooting
- Validate cluster nodes
- Verify nodes are Ready before focusing on control plane services:
- If nodes are NotReady, check
kubeletand node-level networking/firewall first.
- Inspect control plane components (pod-managed)
- If your control plane runs as pods (typical with kubeadm), list system pods:
If you see control plane pods in
CrashLoopBackOff, ImagePullBackOff, or NotReady status, run kubectl describe pod <pod-name> -n kube-system to inspect events and kubectl logs <pod-name> -n kube-system for container output. These commands often show configuration, permission, or startup errors.- Inspect control plane components (host-managed)
- On clusters where control plane binaries run as host services (not as pods), check the service status on the master node. Many distributions provide both a service wrapper and systemd unit; prefer systemd where available.
kubelet and kube-proxy on worker nodes:
- Inspect logs for control plane components
- Pod-managed components: fetch logs with
kubectl logs:
- Host-managed components: use the host journal (
journalctl) to view unit logs:
- etcd health: run
etcdctl(or check the etcd pod’s readiness) and confirm cluster quorum. - Network: verify connectivity between masters and workers (iptables, firewall, routing, CNI plugin status).
- Certificates and clock: check certificate expiry and NTP / clock skew between nodes.
- Recent changes: review config changes, upgrades, or package installs that preceded the failure.
- Resource pressure: CPU, memory, or disk pressure can cause control plane instability.
If the API server is down, kubelets and kube-proxy might still appear healthy locally, but cluster-level operations will fail. Prioritize restoring the API server and etcd (if applicable) before investigating higher-level application issues.