Skip to main content
In this lesson we inspect and manipulate static pods on Kubernetes nodes. We’ll cover how to:
  • Identify static pods in the cluster
  • Confirm a pod is static by inspecting ownerReferences
  • Find the kubelet staticPodPath
  • Count static pod manifest files
  • Inspect the image used by a static pod (kube-apiserver example)
  • Create a static pod by dropping a manifest into the staticPodPath
  • Fix a misconfigured static pod image
  • Permanently remove a static pod (delete its manifest on the node that hosts it)
Static pods are created and managed by the kubelet on each node. The kubelet watches a directory (the staticPodPath) for manifest files and creates pods from them.
Static pods are created directly by the kubelet from manifest files on a node. Deleting a static pod via kubectl delete pod only removes the API-visible Pod object — the kubelet will recreate it if the manifest file still exists. To permanently remove a static pod, delete its manifest from the node’s staticPodPath.

​
Quick reference: static pods vs. controller-managed pods

References:

​
1) List all pods and spot static pods

List all pods across namespaces to locate candidates that look like static pods:
Heuristic: many kubelet-created static pod names include the node name as a suffix (for example, kube-apiserver-controlplane indicates the pod runs on node controlplane). In this example, etcd, kube-apiserver, kube-controller-manager, and kube-scheduler are static pods; CoreDNS is not.

​
2) Confirm a pod is static by inspecting ownerReferences

A definitive way to tell if a pod is static is to inspect its ownerReferences in the pod YAML. A static pod will list its owner as a Node. Example: inspect the kube-apiserver static pod:
Relevant excerpt (ownerReferences indicates the Node is the owner):
Compare with a controller-managed pod (CoreDNS):
Excerpt (ownerReferences points to a ReplicaSet):

​
3) Find the kubelet staticPodPath

The kubelet configuration file contains the staticPodPath setting — the directory kubelet watches for manifest files. On the node you want to inspect (example: controlplane), check the kubelet config:
Look for this excerpt:
Then list the manifest files in that directory:
The number of manifest files should match the static pods observed earlier (4 in this example).

​
4) Inspect the kube-apiserver static pod image

You can either open the manifest or inspect the API-visible pod YAML to find the container image used by a static pod. Example: view the manifest file:
Excerpt showing the image:
Note: image tags and registries vary by cluster. Use the image present on your node.

​
5) Create a static pod named static-busybox (busybox, sleep 1000)

Create a Pod manifest with kubectl run using --dry-run=client -o yaml and then place that manifest into the kubelet staticPodPath. Important: when you use --command -- <cmd>, any kubectl global options must come before the --. Keep --dry-run=client -o yaml before --command --. Generate the manifest and save it:
(Optionally pin the image: --image=busybox:1.28.4.) Move the manifest into the staticPodPath (requires root privileges):
Kubelet will detect the new manifest and create the static pod automatically. Verify with:
Common mistakes and troubleshooting:
  • If you put kubectl options after the --, they become part of the container command instead of kubectl options.
  • If the image cannot be pulled, pod status will show ErrImagePull or ImagePullBackOff. Verify the image name/tag and network access to the registry.
Example of failure due to bad image tag:
Fix such issues by editing the manifest at /etc/kubernetes/manifests/static-busybox.yaml and correcting the image: value. Kubelet will automatically apply the change.

​
6) Delete a static pod permanently

A kubectl delete pod will remove the API-visible Pod object, but kubelet will recreate it if the manifest remains on the node. To permanently remove a static pod, remove its manifest from the node’s staticPodPath. Steps:
  1. Determine which node hosts the static pod (pod name suffix, kubectl get pods -o wide, or kubectl get pod <name> -o yaml).
  2. SSH to that node (find node IPs with kubectl get nodes -o wide).
  3. Inspect /var/lib/kubelet/config.yaml on that node to confirm staticPodPath.
  4. Remove the manifest file from the staticPodPath.
Example: remove static-greenbox which runs on node01. View pods:
Find node IP and SSH:
On node01, inspect kubelet config and remove manifest:
Back on the control plane, watch the pod terminate and disappear:
When removing static pod manifests on a node, be careful: deleting critical control-plane manifests (for example, apiserver/etcd) will disrupt cluster control services on that node. Only remove manifests you intend to permanently delete, and ensure you have access to recover the node if needed.
That covers typical operations for static pods: identifying them, confirming ownership, locating kubelet’s staticPodPath, creating static pods by placing manifests into that path, troubleshooting image issues, and permanently removing static pods by deleting the manifest on the node that hosts them. Links and references:

Watch Video