> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Network Troubleshooting

> Guide to diagnosing Kubernetes service connectivity issues by checking Pods, Services, DNS, CNI and kube-proxy with practical ordered troubleshooting steps.

Kubernetes networking lets your workloads (Pods) communicate with each other, with external services, and with users. Because every Pod gets a unique IP, application design is simplified — Pods can talk to each other without SNAT — but troubleshooting requires checking several layers: the application Pod, Service abstraction, DNS, CNI, and kube-proxy.

This guide explains the core components to inspect and a practical, ordered troubleshooting sequence to follow when service connectivity breaks.

## Core components at a glance

| Component | Role | Quick checks |
| - | - | - |
| Pod | Basic unit running one or more containers; each Pod gets an IP | `kubectl get pods -l app=<name>` |
| Service | Stable access (ClusterIP/NodePort/LoadBalancer) to a set of Pods | `kubectl get svc <name> -o yaml` |
| DNS (CoreDNS) | Resolves service and pod names to IPs inside the cluster | `kubectl get pods -n kube-system -l k8s-app=kube-dns` |
| CNI plugin | Assigns Pod IPs and sets up Pod networking | Check CNI DaemonSet pods and logs (`kubectl get ds -n <cni-namespace>`) |
| kube-proxy | Programs node-level packet-forwarding (iptables or IPVS) for Services | `kubectl get pods -n kube-system -l k8s-app=kube-proxy` |

Start troubleshooting at the application Pod and work outward: Pod health → direct Pod connectivity → Service definition/endpoints → DNS → CNI → kube-proxy.

***

## 1) Check Pod health

A Service only sends traffic to Pods that are Ready. Begin by verifying the Pods backing the Service are Running and Ready.

Example:

```bash theme={null}
kubectl get pods -l app=hostnames
```

Sample output:

```text theme={null}
NAME                         READY   STATUS    RESTARTS   AGE
hostnames-6f78f87b8f-abcdef 1/1     Running   0          5m
hostnames-6f78f87b8f-ghijkl 1/1     Running   0          5m
```

What to look for:

* `READY 1/1` — all containers are ready.
* `STATUS` should be `Running`.
* `RESTARTS > 0` — may indicate container instability.
* `AGE` — how long the Pod has been running.

If Pods are not healthy, inspect logs and Pod details:

```bash theme={null}
kubectl logs <pod-name>
kubectl describe pod <pod-name>
```

***

## 2) Reach Pods directly (bypass the Service)

If Pods appear healthy, test direct connectivity to confirm the application is listening on the expected port. This removes the Service layer from the equation.

Get Pod IPs:

```bash theme={null}
kubectl get pods -o wide
```

Sample output:

```text theme={null}
NAME           READY   STATUS    IP           NODE
webapp-pod1    1/1     Running   10.244.1.5   node-1
webapp-pod2    1/1     Running   10.244.2.3   node-2
```

Or pull Pod IPs using jsonpath:

```bash theme={null}
kubectl get pods -l app=hostnames -o=jsonpath='{.items[*].status.podIP}'
# Example output:
# 10.244.0.5 10.244.0.6 10.244.0.7
```

From a temporary pod (BusyBox or similar) curl/wget the Pod IPs:

```bash theme={null}
kubectl run -it --rm --restart=Never busybox --image=busybox -- sh
# Inside the busybox shell:
wget -qO- http://10.244.0.5:9376
```

Interpretation:

* If direct requests succeed, the app is serving traffic; investigate Service, kube-proxy, or DNS next.
* If direct requests fail, the issue is likely the application itself or Pod-level networking (CNI).

***

## 3) Validate Service definition and endpoints

A Service must select the correct Pods and map `port` → `targetPort` correctly.

Inspect the Service:

```bash theme={null}
kubectl get svc hostnames -o yaml
```

Typical Service spec snippet:

```yaml theme={null}
spec:
  selector:
    app: hostnames
  ports:
  - port: 80
    targetPort: 9376
```

Check the Endpoints or EndpointSlices that the control plane exposes:

```bash theme={null}
kubectl get endpointslices -l kubernetes.io/service-name=hostnames -n default
```

Sample output:

```text theme={null}
NAME                ADDRESSTYPE   PORTS   ENDPOINTS                               AGE
hostnames-abc       IPv4          9376    10.244.0.5,10.244.0.6,10.244.0.7        1h
```

If the `ENDPOINTS` column is empty:

* The Service selector is not matching any Pods — re-check Pod labels vs Service selector.
* The targeted port may be incorrect — ensure `targetPort` matches the container port.

***

## 4) CoreDNS (cluster DNS)

CoreDNS resolves service names to ClusterIPs and Pod names to IPs. DNS failures prevent services from being reachable by name.

CoreDNS runs in the `kube-system` namespace and is exposed via the `kube-dns` Service. Key resources: CoreDNS Deployment, Corefile (ConfigMap), Service, and RBAC.

<Frame>
  <img src="https://mintcdn.com/kodekloud-c4ac6d9a/iXPD-mlT91ZNL1uK/images/Certified-Kubernetes-Administrator-CKA/Troubleshooting/Network-Troubleshooting/kubernetes-service-endpoints-dns-debugging-diagram.jpg?fit=max&auto=format&n=iXPD-mlT91ZNL1uK&q=85&s=a5dc5cc2e8943a497aee349daac3e540" alt="A labeled Kubernetes architecture diagram titled &#x22;Debugging Service Issues: Check Service Definition and Endpoints,&#x22; showing components like kube-system, ConfigMap, CoreDNS pods, Services, Control Plane, Worker Node, kube-proxy and client/worker pods. It visualizes how services, endpoints and DNS (kube-dns/CoreDNS) interact for debugging service connectivity." width="1920" height="1080" data-path="images/Certified-Kubernetes-Administrator-CKA/Troubleshooting/Network-Troubleshooting/kubernetes-service-endpoints-dns-debugging-diagram.jpg" />
</Frame>

Corefile example:

```text theme={null}
kubernetes cluster.local in-addr.arpa ip6.arpa {
    pods insecure
    fallthrough in-addr.arpa ip6.arpa
    ttl 30
}
forward . /etc/resolv.conf
```

DNS troubleshooting checklist:

1. Check CoreDNS Pod status:

```bash theme={null}
kubectl get pods -n kube-system -l k8s-app=kube-dns
```

* `Pending` Pods may indicate CNI issues.
* `CrashLoopBackOff` — inspect logs:

```bash theme={null}
kubectl logs -n kube-system <coredns-pod-name>
```

2. Verify kube-dns endpoints (EndpointSlices):

```bash theme={null}
kubectl get endpointslices -l kubernetes.io/service-name=kube-dns -n kube-system
```

If no endpoints are present, the Service selector might not match CoreDNS Pods — check labels and selectors.

3. Inspect `/etc/resolv.conf` inside a Pod:

```bash theme={null}
kubectl exec -it <pod-name> -- cat /etc/resolv.conf
```

* `nameserver` should be the kube-dns Service ClusterIP.
* `search` manages short-name resolution.
* `ndots` influences how unqualified names are searched.

4. Test DNS resolution from a Pod:

```bash theme={null}
kubectl exec -it busybox -- nslookup kubernetes.default.svc.cluster.local
kubectl exec -it busybox -- nslookup my-app-service.default.svc.cluster.local
```

If these fail, focus on CoreDNS (Deployment, Corefile, Service) and the network path to CoreDNS Pods.

***

## 5) CNI plugins

CNI plugins assign Pod IPs and set up Pod-to-Pod networking. A broken or misconfigured CNI commonly results in Pods stuck in `Pending`, or Pods running but lacking network connectivity.

Popular CNIs:

* Flannel — simple overlay (no default NetworkPolicy enforcement). Install example:

```bash theme={null}
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
```

* Calico — supports NetworkPolicy, can run with or without overlay. Install by applying the Calico manifest from the project.

Important: kubelet uses the first CNI config file it finds alphabetically in the CNI config directory. Multiple CNI config files can cause unexpected plugin selection.

<Callout icon="lightbulb" color="#1CB2FE">
  If multiple CNI config files exist, remove or rename the unwanted ones so the intended plugin is the alphabetically first file.
</Callout>

CNI troubleshooting steps:

* Ensure the CNI DaemonSet Pods are Running (namespace depends on plugin, e.g., `kube-flannel`, `calico-system`).
* Check the CNI Pods' logs for allocation errors.
* If Pods remain `Pending`, inspect kubelet logs and CNI plugin logs for network allocation failures.

***

## 6) kube-proxy

kube-proxy runs on every node and programs packet-forwarding rules (iptables or IPVS) so Services reach Pod backends. It watches Services and Endpoints and keeps the node’s rules in sync.

<Frame>
  <img src="https://mintcdn.com/kodekloud-c4ac6d9a/iXPD-mlT91ZNL1uK/images/Certified-Kubernetes-Administrator-CKA/Troubleshooting/Network-Troubleshooting/cni-kube-proxy-troubleshooting-network-issue.jpg?fit=max&auto=format&n=iXPD-mlT91ZNL1uK&q=85&s=5f7afe6a5094f9b4a63ddc441c9668ae" alt="An infographic titled &#x22;Troubleshooting CNI and kube-proxy&#x22; showing two panels: one for CNI troubleshooting (check DaemonSet pod status, analyze CNI pod logs) and one for kube-proxy rules (service-to-pod proxying). A warning triangle labeled &#x22;Network Issue&#x22; is shown beneath the panels." width="1920" height="1080" data-path="images/Certified-Kubernetes-Administrator-CKA/Troubleshooting/Network-Troubleshooting/cni-kube-proxy-troubleshooting-network-issue.jpg" />
</Frame>

kube-proxy troubleshooting steps:

1. Check kube-proxy Pods:

```bash theme={null}
kubectl get pods -n kube-system -l k8s-app=kube-proxy
```

2. Inspect kube-proxy logs:

```bash theme={null}
kubectl logs -n kube-system <kube-proxy-pod-name>
# Example:
# I1207 10:30:15.123 1 server.go:653] Successfully retrieved node IPv4 address
# I1207 10:30:15.456 1 proxier.go:683] Syncing iptables rules
```

3. Check kube-proxy configuration:

```bash theme={null}
kubectl get configmap kube-proxy -n kube-system -o yaml
```

Example snippet:

```yaml theme={null}
data:
  config.conf: |-
    mode: "ipvs"
    clusterCIDR: "10.244.0.0/16"
```

4. Verify rules on the node:

* For `iptables` mode: SSH to the node and inspect iptables rules.
* For `ipvs` mode: use `ipvsadm -Ln` to list IPVS virtual servers and destinations to confirm that Service VIPs map to Pod backends.

***

## Recap — recommended troubleshooting sequence

1. Pod health: `kubectl get pods` → logs and describe if unhealthy.
2. Direct Pod reachability: curl/wget Pod IPs from a debug Pod.
3. Service validation: verify Service YAML (`selector`, `port`, `targetPort`). Check EndpointSlices/Endpoints.
4. DNS: check CoreDNS Pods, Corefile, Service, and test resolution from a Pod.
5. CNI: ensure CNI DaemonSet Pods are running and logs show successful Pod IP allocation.
6. kube-proxy: check Pod status, logs, ConfigMap, and node-level iptables/IPVS rules.

Following this ordered sequence narrows the root cause quickly — whether it’s an application issue, Service misconfiguration, DNS problem, CNI allocation failure, or kube-proxy rule mismatch.

## Links and references

* [Kubernetes Networking Concepts](https://kubernetes.io/docs/concepts/cluster-administration/networking/)
* [CoreDNS Documentation](https://coredns.io/)
* [Flannel GitHub](https://github.com/flannel-io/flannel)
* [Calico Documentation](https://docs.projectcalico.org/)

<CardGroup>
  <Card title="Watch Video" icon="video" cta="Learn more" href="https://learn.kodekloud.com/user/courses/cka-certification-course-certified-kubernetes-administrator/module/aff99955-65fd-443e-950f-3b25d3311bc2/lesson/1dad8b2b-92d7-48d7-9544-91ee4f1e551c" />

  <Card title="Practice Lab" icon="flask-conical" cta="Learn more" href="https://learn.kodekloud.com/user/courses/cka-certification-course-certified-kubernetes-administrator/module/aff99955-65fd-443e-950f-3b25d3311bc2/lesson/6e76bbe5-b34b-4516-9bc6-7b9e248d53cd" />
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.