> ## Documentation Index > Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt > Use this file to discover all available pages before exploring further. # Lab Solutions Worker Node Failure > Guide to diagnosing and fixing kubelet-related worker node failures by inspecting services, logs, and configuration to restore node readiness In this lesson you'll practice diagnosing and repairing worker node failures by inspecting the cluster from the control plane, SSHing into the worker, and resolving kubelet-related issues until the node returns to `Ready`. Prerequisites: * Access to the control plane and worker node via SSH. * Root or sudo on the nodes. * Basic familiarity with systemd, `journalctl`, and kubelet configuration files. Quick alias for convenience: ```bash theme={null} root@controlplane:~# alias k=kubectl ``` ## 1) First symptom: node01 is NotReady From the control plane, check node status: ```bash theme={null} root@controlplane:~# k get nodes NAME STATUS ROLES AGE VERSION controlplane Ready control-plane,master 36m v1.20.0 node01 NotReady ` ` 36m v1.20.0 ``` node01 is `NotReady`. The kubelet is the primary node agent to investigate. SSH to the worker and inspect the kubelet service: ```bash theme={null} root@controlplane:~# ssh node01 Last login: Fri Apr 22 22:23:00 2022 from 10.54.130.4 root@node01:~# service kubelet status ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: inactive (dead) since Fri 2022-04-22 22:58:51 UTC; 1min 58s ago Docs: https://kubernetes.io/docs/home/ Process: 1770 ExecStart=/usr/bin/kubelet ... (code=exited, status=0/SUCCESS) Main PID: 1770 (code=exited, status=0/SUCCESS) ``` Here the kubelet is inactive. Start it and verify the service becomes active: ```bash theme={null} root@node01:~# service kubelet start root@node01:~# service kubelet status ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: active (running) since Fri 2022-04-22 23:01:07 UTC; 2s ago Main PID: 18420 (kubelet) ``` Back on the control plane, confirm the node is now Ready: ```bash theme={null} root@controlplane:~# k get nodes NAME STATUS ROLES AGE VERSION controlplane Ready control-plane,master 38m v1.20.0 node01 Ready ` ` 37m v1.20.0 ``` This resolved the first occurrence. The lab will now present new failure scenarios for further troubleshooting. ## 2) Second symptom: kubelet repeatedly fails (exit code 255) Later, node01 becomes `NotReady` again: ```bash theme={null} root@controlplane:~# kubectl get nodes NAME STATUS ROLES AGE VERSION controlplane Ready control-plane,master 39m v1.20.0 node01 NotReady ` ` 39m v1.20.0 ``` SSH to node01 and inspect kubelet status — it is auto-restarting with an exit code: ```bash theme={null} root@controlplane:~# ssh node01 root@node01:~# service kubelet status ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: activating (auto-restart) (Result: exit-code) since Fri 2022-04-22 23:02:58 UTC Process: 19364 ExecStart=/usr/bin/kubelet ... (code=exited, status=255) Main PID: 19364 (code=exited, status=255) ``` Gather kubelet logs to see why it's failing: ```bash theme={null} root@node01:~# journalctl -u kubelet -n 200 ... Apr 22 23:03:19 node01 kubelet[19450]: F0422 23:03:19.266538 19450 server.go:257] unable to load client CA file /etc/kubernetes/pki/WRONG-CA-FILE.crt: open /etc/kubernetes/pki/WRONG-CA-FILE.crt: no such file or directory ... ``` The log shows the kubelet configuration points to a non-existent client CA file. Inspect the kubelet config file where additional options are read: ```bash theme={null} root@node01:~# ls /var/lib/kubelet/ config.yaml cpu_manager_state device-plugins kubeadm-flags.env pki plugins plugins_registry pod-resources pods root@node01:~# cat /var/lib/kubelet/config.yaml apiVersion: kubelet.config.k8s.io/v1beta1 authentication: anonymous: enabled: false webhook: cacheTTL: 0s enabled: true x509: clientCAFile: /etc/kubernetes/pki/WRONG-CA-FILE.crt authorization: mode: Webhook webhook: cacheAuthorizedTTL: 0s cacheUnauthorizedTTL: 0s ... ``` The `clientCAFile` points to `/etc/kubernetes/pki/WRONG-CA-FILE.crt`. The correct CA on kubeadm clusters is typically `/etc/kubernetes/pki/ca.crt`. Update the config and restart kubelet: ```bash theme={null} root@node01:~# sed -i 's|/etc/kubernetes/pki/WRONG-CA-FILE.crt|/etc/kubernetes/pki/ca.crt|' /var/lib/kubelet/config.yaml root@node01:~# service kubelet restart root@node01:~# service kubelet status ● kubelet.service - kubelet: The Kubernetes Node Agent Active: active (running) since Fri 2022-04-22 23:01:07 UTC; 2s ago ``` Verify the control plane — the node may become `Ready` again until the next failure. ## 3) Third symptom: kubelet cannot register with API server (connection refused) When the node becomes `NotReady` again, check the kubelet logs: ```bash theme={null} root@controlplane:~# kubectl get nodes NAME STATUS ROLES AGE VERSION controlplane Ready control-plane,master 44m v1.20.0 node01 NotReady ` ` 44m v1.20.0 ``` On node01, view recent kubelet logs: ```bash theme={null} root@node01:~# journalctl -u kubelet -n 200 ... Apr 22 23:12:26 node01 kubelet[24060]: I0422 23:12:26.498317 24060 kubelet_node_status.go:71] Attempting to register node node01 Apr 22 23:12:26 node01 kubelet[24060]: E0422 23:12:26.499538 24060 kubelet_node_status.go:93] Unable to register node "node01" with API server: Post "https://controlplane:6553/api/v1/nodes": dial tcp 10.54.130.2:6553: connect: connection refused ``` The kubelet is failing to connect to the API server at port `6553`. The kubelet's kubeconfig (`/etc/kubernetes/kubelet.conf`) defines the API server `server` URL. Inspect the kubeconfig: ```bash theme={null} root@node01:~# cat /etc/kubernetes/kubelet.conf apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CR... server: https://controlplane:6553 name: default-cluster contexts: - context: cluster: default-cluster namespace: default user: default-auth name: default-context current-context: default-context users: - name: default-auth user: client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem client-key: /var/lib/kubelet/pki/kubelet-client-current.pem ``` The `server` is set to `https://controlplane:6553` which is incorrect — the kube-apiserver listens on port `6443` by default. Update the kubeconfig to the correct port and restart kubelet: ```bash theme={null} root@node01:~# sed -i 's|https://controlplane:6553|https://controlplane:6443|' /etc/kubernetes/kubelet.conf root@node01:~# service kubelet restart root@node01:~# service kubelet status ● kubelet.service - kubelet: The Kubernetes Node Agent Active: active (running) since Fri 2022-04-22 23:13:48 UTC; 6s ago ``` Confirm the node re-registers on the control plane: ```bash theme={null} root@controlplane:~# kubectl get nodes NAME STATUS ROLES AGE VERSION controlplane Ready control-plane,master 51m v1.20.0 node01 Ready ` ` 50m v1.20.0 ``` ## Common causes and quick fixes | Symptom | Likely cause | Example fix | | -: | - | - | | Kubelet inactive / dead | Service stopped or crashed | `service kubelet start` and check `journalctl -u kubelet -xe` | | Kubelet exit with file not found | Misconfigured path in `/var/lib/kubelet/config.yaml` (e.g. `clientCAFile`) | Edit `/var/lib/kubelet/config.yaml` to use `/etc/kubernetes/pki/ca.crt` and restart kubelet | | Cannot reach API server | Incorrect `server` URL/port in `/etc/kubernetes/kubelet.conf` or network connectivity | Update `/etc/kubernetes/kubelet.conf` to `https://controlplane:6443` and restart kubelet; verify network/DNS/iptables | | TLS / certificate errors | Missing or expired certificates | Verify files under `/etc/kubernetes/pki` and `/var/lib/kubelet/pki` and rotate/regenerate if required | | API server connection refused | API server down or wrong port | Check API server on control plane and kubeconfig `server` field | Useful references: * [Kubernetes kubelet troubleshooting](https://kubernetes.io/docs/tasks/administer-cluster/kubelet/) * [kubeadm kubelet configuration](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#kubelet-configuration) When a worker node shows `NotReady`, follow this prioritized checklist: 1. From the control plane, identify the affected node: `kubectl get nodes`. 2. SSH into the worker and check the kubelet service: `systemctl status kubelet` or `service kubelet status`. 3. Inspect logs for immediate clues: `journalctl -u kubelet -xe` or `journalctl -u kubelet -n 200`. 4. Verify kubelet configuration files: * `/var/lib/kubelet/config.yaml` (e.g. `x509.clientCAFile`) * `/etc/kubernetes/kubelet.conf` (API server `server` URL) 5. Fix obvious misconfigurations (correct file paths, ports, or certificates), restart kubelet, and re-check node status: `kubectl get nodes`. 6. If the issue persists, confirm network connectivity to the API server and validate certificate validity. That's the end of this lesson. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.