Skip to main content
In this guide, we walk through troubleshooting and resolving worker node failures in a Kubernetes cluster. Before you begin, ensure that your lab environment is properly set up. Below is a step-by-step procedure to diagnose and fix issues on a worker node (node01).

​
Step 1: Verify Node Status from the Control Plane

Start by checking the status of all the nodes in the cluster. Run the following command on the control plane:
In this output, you can clearly see that node01 is in a NotReady state.
A quick look at the node status can help you identify which node is experiencing issues before diving into detailed troubleshooting.

​
Step 2: Examine Detailed Node Information

Investigate further by describing the details and events related to the problematic node:
Review the output carefully. Look for information such as:
  • Kubelet Version: v1.20.0
  • PodCIDR: 10.244.1.0/24
Even though many components are functioning correctly, the node remains in a NotReady state, indicating that the root cause might lie elsewhere.

​
Step 3: Check the Kubelet Service on node01

Next, SSH into the worker node to examine the status of the kubelet service:
You might see output similar to this:
Since the kubelet service is not active, start it by executing:
Then, verify that it is running:
Return to the control plane and run:
You should now see both nodes reported with a Ready status.

​
Step 4: Simulate the Next Failure and Investigate Further

After the initial fix, the cluster shows issues again with node01 reverting to a NotReady state. Verify the node status:
SSH into node01 once more and inspect the kubelet service status:
You may now observe that the service is in an “activating (auto-restart)” state, repeatedly exiting with code 255:
Check the logs to understand the problem:
Look for error messages such as:
The logs suggest the kubelet is failing to load its configuration due to an incorrect certificate authority (CA) file.

​
Step 5: Correct the Kubelet Configuration

Examine the kubelet configuration file by executing:
You might see an entry like this:
This file points to the wrong CA certificate. Identify the correct CA file (for example, /etc/kubernetes/pki/ca.cert) and update the configuration accordingly. After making the change, restart the kubelet service:
Then, confirm the service is active:
Return to the control plane and check that both nodes are now Ready:
Always back up configuration files before making any changes.

​
Step 6: Fix the Incorrect Control Plane Port in kubelet.conf

Even after the configuration fix, if node01 goes NotReady again, inspect the logs on node01. You might see an error like:
This indicates that the kubelet is attempting to connect to the control plane on an incorrect port (6553). To resolve this, inspect the kubelet configuration file:
You might see:
Update the port from 6533 to 6443 (the correct control plane port):
After saving the changes, restart the kubelet service:
Monitor its status to ensure that it is actively running:
Finally, verify from the control plane that both nodes are in a Ready state:

​
Conclusion

When troubleshooting worker node failures in a Kubernetes cluster, follow these steps:
  1. Check the node status using kubectl get nodes.
  2. SSH into the affected worker node and verify that the kubelet service is running.
  3. If the kubelet service fails, review the logs using journalctl -u kubelet to identify any misconfigurations.
  4. In this example, the issues included:
    • An incorrect client CA file in /var/lib/kubelet/config.yaml
    • An incorrect control plane port in /etc/kubernetes/kubelet.conf
  5. Correct the misconfigurations and restart the kubelet service, then verify that the node status returns to Ready.
Following this systematic approach will help you quickly pinpoint and resolve issues during daily operations in your Kubernetes cluster. Happy troubleshooting! For further reading:

Watch Video