Lab Solutions Worker Node Failure

In this lesson you’ll practice diagnosing and repairing worker node failures by inspecting the cluster from the control plane, SSHing into the worker, and resolving kubelet-related issues until the node returns to Ready. Prerequisites:

Access to the control plane and worker node via SSH.
Root or sudo on the nodes.
Basic familiarity with systemd, journalctl, and kubelet configuration files.

Quick alias for convenience:

root@controlplane:~# alias k=kubectl

1) First symptom: node01 is NotReady

From the control plane, check node status:

root@controlplane:~# k get nodes
NAME           STATUS     ROLES                  AGE   VERSION
controlplane   Ready      control-plane,master   36m   v1.20.0
node01         NotReady   ` <none> `                 36m   v1.20.0

node01 is NotReady. The kubelet is the primary node agent to investigate. SSH to the worker and inspect the kubelet service:

root@controlplane:~# ssh node01
Last login: Fri Apr 22 22:23:00 2022 from 10.54.130.4
root@node01:~# service kubelet status
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
   Drop-In: /etc/systemd/system/kubelet.service.d
            └─10-kubeadm.conf
   Active: inactive (dead) since Fri 2022-04-22 22:58:51 UTC; 1min 58s ago
     Docs: https://kubernetes.io/docs/home/
  Process: 1770 ExecStart=/usr/bin/kubelet ... (code=exited, status=0/SUCCESS)
 Main PID: 1770 (code=exited, status=0/SUCCESS)

Here the kubelet is inactive. Start it and verify the service becomes active:

root@node01:~# service kubelet start
root@node01:~# service kubelet status
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
   Drop-In: /etc/systemd/system/kubelet.service.d
            └─10-kubeadm.conf
   Active: active (running) since Fri 2022-04-22 23:01:07 UTC; 2s ago
 Main PID: 18420 (kubelet)

Back on the control plane, confirm the node is now Ready:

root@controlplane:~# k get nodes
NAME           STATUS   ROLES                 AGE   VERSION
controlplane   Ready    control-plane,master  38m   v1.20.0
node01         Ready    ` <none> `                37m   v1.20.0

This resolved the first occurrence. The lab will now present new failure scenarios for further troubleshooting.

2) Second symptom: kubelet repeatedly fails (exit code 255)

Later, node01 becomes NotReady again:

root@controlplane:~# kubectl get nodes
NAME           STATUS     ROLES                 AGE   VERSION
controlplane   Ready      control-plane,master  39m   v1.20.0
node01         NotReady   ` <none> `                 39m   v1.20.0

SSH to node01 and inspect kubelet status — it is auto-restarting with an exit code:

root@controlplane:~# ssh node01
root@node01:~# service kubelet status
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
   Drop-In: /etc/systemd/system/kubelet.service.d
            └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Fri 2022-04-22 23:02:58 UTC
  Process: 19364 ExecStart=/usr/bin/kubelet ... (code=exited, status=255)
 Main PID: 19364 (code=exited, status=255)

Gather kubelet logs to see why it’s failing:

root@node01:~# journalctl -u kubelet -n 200
...
Apr 22 23:03:19 node01 kubelet[19450]: F0422 23:03:19.266538 19450 server.go:257] unable to load client CA file /etc/kubernetes/pki/WRONG-CA-FILE.crt: open /etc/kubernetes/pki/WRONG-CA-FILE.crt: no such file or directory
...

The log shows the kubelet configuration points to a non-existent client CA file. Inspect the kubelet config file where additional options are read:

root@node01:~# ls /var/lib/kubelet/
config.yaml  cpu_manager_state  device-plugins  kubeadm-flags.env  pki  plugins  plugins_registry  pod-resources  pods

root@node01:~# cat /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
x509:
  clientCAFile: /etc/kubernetes/pki/WRONG-CA-FILE.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
...

The clientCAFile points to /etc/kubernetes/pki/WRONG-CA-FILE.crt. The correct CA on kubeadm clusters is typically /etc/kubernetes/pki/ca.crt. Update the config and restart kubelet:

root@node01:~# sed -i 's|/etc/kubernetes/pki/WRONG-CA-FILE.crt|/etc/kubernetes/pki/ca.crt|' /var/lib/kubelet/config.yaml
root@node01:~# service kubelet restart
root@node01:~# service kubelet status
● kubelet.service - kubelet: The Kubernetes Node Agent
   Active: active (running) since Fri 2022-04-22 23:01:07 UTC; 2s ago

Verify the control plane — the node may become Ready again until the next failure.

3) Third symptom: kubelet cannot register with API server (connection refused)

When the node becomes NotReady again, check the kubelet logs:

root@controlplane:~# kubectl get nodes
NAME           STATUS     ROLES                 AGE   VERSION
controlplane   Ready      control-plane,master  44m   v1.20.0
node01         NotReady   ` <none> `                 44m   v1.20.0

On node01, view recent kubelet logs:

root@node01:~# journalctl -u kubelet -n 200
...
Apr 22 23:12:26 node01 kubelet[24060]: I0422 23:12:26.498317 24060 kubelet_node_status.go:71] Attempting to register node node01
Apr 22 23:12:26 node01 kubelet[24060]: E0422 23:12:26.499538 24060 kubelet_node_status.go:93] Unable to register node "node01" with API server: Post "https://controlplane:6553/api/v1/nodes": dial tcp 10.54.130.2:6553: connect: connection refused

The kubelet is failing to connect to the API server at port 6553. The kubelet’s kubeconfig (/etc/kubernetes/kubelet.conf) defines the API server server URL. Inspect the kubeconfig:

root@node01:~# cat /etc/kubernetes/kubelet.conf
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CR...
    server: https://controlplane:6553
  name: default-cluster
contexts:
- context:
    cluster: default-cluster
    namespace: default
    user: default-auth
  name: default-context
current-context: default-context
users:
- name: default-auth
  user:
    client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
    client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

The server is set to https://controlplane:6553 which is incorrect — the kube-apiserver listens on port 6443 by default. Update the kubeconfig to the correct port and restart kubelet:

root@node01:~# sed -i 's|https://controlplane:6553|https://controlplane:6443|' /etc/kubernetes/kubelet.conf
root@node01:~# service kubelet restart
root@node01:~# service kubelet status
● kubelet.service - kubelet: The Kubernetes Node Agent
   Active: active (running) since Fri 2022-04-22 23:13:48 UTC; 6s ago

Confirm the node re-registers on the control plane:

root@controlplane:~# kubectl get nodes
NAME           STATUS   ROLES                 AGE   VERSION
controlplane   Ready    control-plane,master  51m   v1.20.0
node01         Ready    ` <none> `                50m   v1.20.0

Common causes and quick fixes

Symptom	Likely cause	Example fix
Kubelet inactive / dead	Service stopped or crashed	`service kubelet start` and check `journalctl -u kubelet -xe`
Kubelet exit with file not found	Misconfigured path in `/var/lib/kubelet/config.yaml` (e.g. `clientCAFile`)	Edit `/var/lib/kubelet/config.yaml` to use `/etc/kubernetes/pki/ca.crt` and restart kubelet
Cannot reach API server	Incorrect `server` URL/port in `/etc/kubernetes/kubelet.conf` or network connectivity	Update `/etc/kubernetes/kubelet.conf` to `https://controlplane:6443` and restart kubelet; verify network/DNS/iptables
TLS / certificate errors	Missing or expired certificates	Verify files under `/etc/kubernetes/pki` and `/var/lib/kubelet/pki` and rotate/regenerate if required
API server connection refused	API server down or wrong port	Check API server on control plane and kubeconfig `server` field

Useful references:

When a worker node shows NotReady, follow this prioritized checklist:

From the control plane, identify the affected node: kubectl get nodes.
SSH into the worker and check the kubelet service: systemctl status kubelet or service kubelet status.
Inspect logs for immediate clues: journalctl -u kubelet -xe or journalctl -u kubelet -n 200.
Verify kubelet configuration files:
- /var/lib/kubelet/config.yaml (e.g. x509.clientCAFile)
- /etc/kubernetes/kubelet.conf (API server server URL)
Fix obvious misconfigurations (correct file paths, ports, or certificates), restart kubelet, and re-check node status: kubectl get nodes.
If the issue persists, confirm network connectivity to the API server and validate certificate validity.

That’s the end of this lesson.

Introduction

Core Concepts

Scheduling

Logging & Monitoring

Application Lifecycle Management

Cluster Maintenance

Security

Storage

Networking

Design and Install a Kubernetes Cluster

Install "Kubernetes the kubeadm way"

Troubleshooting

Helm Basics (2025 Updates)

Kustomize Basics (2025 Updates)

Other Topics

Mock Exams

Lab Solutions Worker Node Failure

1) First symptom: node01 is NotReady

2) Second symptom: kubelet repeatedly fails (exit code 255)

3) Third symptom: kubelet cannot register with API server (connection refused)

Common causes and quick fixes

Watch Video

​1) First symptom: node01 is NotReady

​2) Second symptom: kubelet repeatedly fails (exit code 255)

​3) Third symptom: kubelet cannot register with API server (connection refused)

​Common causes and quick fixes

Watch Video

1) First symptom: node01 is NotReady

2) Second symptom: kubelet repeatedly fails (exit code 255)

3) Third symptom: kubelet cannot register with API server (connection refused)

Common causes and quick fixes