> ## Documentation Index > Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt > Use this file to discover all available pages before exploring further. # Lab Solution Static Pods Optional > Guide to identifying, inspecting, creating, troubleshooting, and permanently removing Kubernetes static pods by working with kubelet staticPodPath, manifest files, and ownerReferences In this lesson we inspect and manipulate static pods on Kubernetes nodes. We'll cover how to: * Identify static pods in the cluster * Confirm a pod is static by inspecting `ownerReferences` * Find the kubelet `staticPodPath` * Count static pod manifest files * Inspect the image used by a static pod (kube-apiserver example) * Create a static pod by dropping a manifest into the `staticPodPath` * Fix a misconfigured static pod image * Permanently remove a static pod (delete its manifest on the node that hosts it) Static pods are created and managed by the kubelet on each node. The kubelet watches a directory (the `staticPodPath`) for manifest files and creates pods from them. Static pods are created directly by the kubelet from manifest files on a node. Deleting a static pod via `kubectl delete pod` only removes the API-visible Pod object — the kubelet will recreate it if the manifest file still exists. To permanently remove a static pod, delete its manifest from the node's `staticPodPath`. *** ## Quick reference: static pods vs. controller-managed pods | Characteristic | Static pod | Controller-managed pod | | - | -: | - | | Created by | Kubelet on node (manifest file) | Kubernetes control plane (Deployment/ReplicaSet, etc.) | | Owner in `ownerReferences` | `Node` | `ReplicaSet`, `DaemonSet`, `StatefulSet`, etc. | | Managed via `kubectl delete pod` | Kubelet will recreate if manifest exists | Deletion is honored; controller may recreate depending on controller | | Manifest location | `staticPodPath` (kubelet config) | API objects stored in etcd | References: * [Kubernetes documentation — Static Pods](https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/) *** ## 1) List all pods and spot static pods List all pods across namespaces to locate candidates that look like static pods: ```bash theme={null} root@controlplane:~# kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-74ff55c5b-brwnd 1/1 Running 0 9m44s kube-system coredns-74ff55c5b-nfg4f 1/1 Running 0 9m43s kube-system etcd-controlplane 1/1 Running 0 9m54s kube-system kube-apiserver-controlplane 1/1 Running 0 9m54s kube-system kube-controller-manager-controlplane 1/1 Running 0 9m54s kube-system kube-flannel-ds-95zmd 1/1 Running 0 9m37s kube-system kube-flannel-ds-ct7qn 1/1 Running 0 9m14s kube-system kube-proxy-c5zpk 1/1 Running 0 9m44s kube-system kube-proxy-wclgr 1/1 Running 0 9m14s kube-system kube-scheduler-controlplane 1/1 Running 0 9m54s root@controlplane:~# ``` Heuristic: many kubelet-created static pod names include the node name as a suffix (for example, `kube-apiserver-controlplane` indicates the pod runs on node `controlplane`). In this example, `etcd`, `kube-apiserver`, `kube-controller-manager`, and `kube-scheduler` are static pods; CoreDNS is not. *** ## 2) Confirm a pod is static by inspecting ownerReferences A definitive way to tell if a pod is static is to inspect its `ownerReferences` in the pod YAML. A static pod will list its owner as a `Node`. Example: inspect the kube-apiserver static pod: ```bash theme={null} root@controlplane:~# kubectl get pod kube-apiserver-controlplane -n kube-system -o yaml ``` Relevant excerpt (ownerReferences indicates the Node is the owner): ```yaml theme={null} metadata: name: kube-apiserver-controlplane namespace: kube-system ... ownerReferences: - apiVersion: v1 controller: true kind: Node name: controlplane uid: fbab9414-1ae8-4f64-bc94-456dd8569e6c ``` Compare with a controller-managed pod (CoreDNS): ```bash theme={null} root@controlplane:~# kubectl get pod coredns-74ff55c5b-brwnd -n kube-system -o yaml ``` Excerpt (ownerReferences points to a ReplicaSet): ```yaml theme={null} metadata: generateName: coredns-74ff55c5b- labels: k8s-app: kube-dns pod-template-hash: 74ff55c5b ... ownerReferences: - apiVersion: apps/v1 controller: true kind: ReplicaSet name: coredns-74ff55c5b uid: c2935e74-2f0f-44b4-b023-57e228a19b6d ``` *** ## 3) Find the kubelet staticPodPath The kubelet configuration file contains the `staticPodPath` setting — the directory kubelet watches for manifest files. On the node you want to inspect (example: `controlplane`), check the kubelet config: ```bash theme={null} root@controlplane:~# cat /var/lib/kubelet/config.yaml ``` Look for this excerpt: ```yaml theme={null} kind: KubeletConfiguration ... staticPodPath: /etc/kubernetes/manifests ... ``` Then list the manifest files in that directory: ```bash theme={null} root@controlplane:~# ls /etc/kubernetes/manifests etcd.yaml kube-apiserver.yaml kube-controller-manager.yaml kube-scheduler.yaml ``` The number of manifest files should match the static pods observed earlier (4 in this example). *** ## 4) Inspect the kube-apiserver static pod image You can either open the manifest or inspect the API-visible pod YAML to find the container image used by a static pod. Example: view the manifest file: ```bash theme={null} root@controlplane:~# cat /etc/kubernetes/manifests/kube-apiserver.yaml ``` Excerpt showing the image: ```yaml theme={null} - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key image: k8s.gcr.io/kube-apiserver:v1.20.0 imagePullPolicy: IfNotPresent name: kube-apiserver ... ``` Note: image tags and registries vary by cluster. Use the image present on your node. *** ## 5) Create a static pod named `static-busybox` (busybox, sleep 1000) Create a Pod manifest with `kubectl run` using `--dry-run=client -o yaml` and then place that manifest into the kubelet `staticPodPath`. Important: when you use `--command -- `, any `kubectl` global options must come before the `--`. Keep `--dry-run=client -o yaml` before `--command --`. Generate the manifest and save it: ```bash theme={null} root@controlplane:~# kubectl run static-busybox --image=busybox --dry-run=client -o yaml --command -- sleep 1000 > static-busybox.yaml ``` (Optionally pin the image: `--image=busybox:1.28.4`.) Move the manifest into the staticPodPath (requires root privileges): ```bash theme={null} root@controlplane:~# sudo mv static-busybox.yaml /etc/kubernetes/manifests/ ``` Kubelet will detect the new manifest and create the static pod automatically. Verify with: ```bash theme={null} root@controlplane:~# kubectl get pods NAME READY STATUS RESTARTS AGE static-busybox-controlplane 1/1 Running 0 30s ``` Common mistakes and troubleshooting: * If you put `kubectl` options after the `--`, they become part of the container command instead of `kubectl` options. * If the image cannot be pulled, pod status will show `ErrImagePull` or `ImagePullBackOff`. Verify the image name/tag and network access to the registry. Example of failure due to bad image tag: ```bash theme={null} root@controlplane:~# kubectl get pods --watch static-busybox-controlplane 0/1 Pending static-busybox-controlplane 0/1 ErrImagePull static-busybox-controlplane 0/1 ImagePullBackOff ``` Fix such issues by editing the manifest at `/etc/kubernetes/manifests/static-busybox.yaml` and correcting the `image:` value. Kubelet will automatically apply the change. *** ## 6) Delete a static pod permanently A `kubectl delete pod` will remove the API-visible Pod object, but kubelet will recreate it if the manifest remains on the node. To permanently remove a static pod, remove its manifest from the node's `staticPodPath`. Steps: 1. Determine which node hosts the static pod (pod name suffix, `kubectl get pods -o wide`, or `kubectl get pod -o yaml`). 2. SSH to that node (find node IPs with `kubectl get nodes -o wide`). 3. Inspect `/var/lib/kubelet/config.yaml` on that node to confirm `staticPodPath`. 4. Remove the manifest file from the `staticPodPath`. Example: remove `static-greenbox` which runs on `node01`. View pods: ```bash theme={null} root@controlplane:~# kubectl get pods NAME READY STATUS RESTARTS AGE static-busybox-controlplane 1/1 Running 0 4m23s static-greenbox-node01 1/1 Running 0 3m17s ``` Find node IP and SSH: ```bash theme={null} root@controlplane:~# kubectl get nodes -o wide # note node01 internal IP (e.g., 10.38.102.8) root@controlplane:~# ssh root@10.38.102.8 ``` On node01, inspect kubelet config and remove manifest: ```bash theme={null} root@node01:~# cat /var/lib/kubelet/config.yaml ... staticPodPath: /etc/just-to-mess-with-you ... root@node01:~# cd /etc/just-to-mess-with-you root@node01:/etc/just-to-mess-with-you# ls greenbox.yaml root@node01:/etc/just-to-mess-with-you# rm greenbox.yaml ``` Back on the control plane, watch the pod terminate and disappear: ```bash theme={null} root@controlplane:~# kubectl get pods --watch static-greenbox-node01 1/1 Terminating 0 3m31s # eventually it will be gone root@controlplane:~# kubectl get pods NAME READY STATUS RESTARTS AGE static-busybox-controlplane 1/1 Running 0 4m50s ``` When removing static pod manifests on a node, be careful: deleting critical control-plane manifests (for example, apiserver/etcd) will disrupt cluster control services on that node. Only remove manifests you intend to permanently delete, and ensure you have access to recover the node if needed. *** That covers typical operations for static pods: identifying them, confirming ownership, locating kubelet's `staticPodPath`, creating static pods by placing manifests into that path, troubleshooting image issues, and permanently removing static pods by deleting the manifest on the node that hosts them. Links and references: * [Kubernetes — Static Pods](https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/) * [Kubernetes — Kubelet configuration](https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/) * [kubectl run documentation](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#run) This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.