Application Failure

​

Troubleshooting checklist (ordered)

​

1) Test the user-facing endpoint

• From outside the cluster test Node IP + NodePort (NodePort Service) or an external load balancer URL:

# Example: curl the NodePort
curl http://<node-ip>:<node-port>

• From inside the cluster, curl the Service ClusterIP or Service DNS name:

# Example: curl the Service ClusterIP from another pod
kubectl exec -it debug-pod -- curl http://10.96.0.156:8080

$ curl http://<node-ip>:31672
curl: (7) Failed to connect to <node-ip> port 31672: Connection timed out

​

2) Inspect the Service

• Confirm the Service has endpoints (that it discovered pods matching its selector):

kubectl describe service web-service

Name:                   web-service
Namespace:              default
Selector:               name=webapp-mysql
Type:                   NodePort
IP:                     10.96.0.156
Port:                   <unset> 8080/TCP
TargetPort:             8080/TCP
NodePort:               <unset> 31672/TCP
Endpoints:              10.32.0.6:8080

• If Endpoints: is empty, verify the Service selector exactly matches the Pod labels. A label-selector mismatch is a common cause of missing endpoints.

​

3) Check the backing Pod(s)

• Ensure the pod is Running and not repeatedly restarting. Use:

kubectl get pods
kubectl describe pod webapp-mysql

• Look for CrashLoopBackOff, restart counts, and recent events in kubectl describe.
• Example Pod manifest for reference:

apiVersion: v1
kind: Pod
metadata:
  name: webapp-mysql
  labels:
    app: example-app
    name: webapp-mysql
spec:
  containers:
    - name: webapp-mysql
      image: simple-webapp-mysql
      ports:
        - containerPort: 8080

​

4) Read application logs

• Use kubectl logs to examine container output:
  • Follow logs in real time:
  kubectl logs -f webapp-mysql
  
  • View logs from the previous crashed instance:
  kubectl logs -p webapp-mysql
  
  • For multi-container pods:
  kubectl logs webapp-mysql -c <container-name>
  

10.32.0.1 - - [01/Apr/2019 12:51:55] "GET / HTTP/1.1" 200 -
10.32.0.1 - - [01/Apr/2019 12:51:55] "GET /static/img/success.jpg HTTP/1.1" 200 -
10.32.0.1 - - [01/Apr/2019 12:51:55] "GET /favicon.ico HTTP/1.1" 404 -
10.32.0.1 - - [01/Apr/2019 12:51:57] "GET / HTTP/1.1" 200 -

​

5) Move to the database layer (if web layer looks healthy)

• Repeat the Service → Pod → Logs checks for the DB Service and DB Pod.
• Look for messages indicating the DB cannot accept connections (authentication errors, missing schema migrations, out-of-memory kills, etc.).

​

6) Use describe for deeper diagnostics

• kubectl describe surfaces pod events, image pull errors, failed probes, and other useful messages:

kubectl describe pods <pod-name>

​

Quick command reference

​

Additional diagnostic tips

• If Services report endpoints but traffic still fails, check:
  • NetworkPolicies (restricting Pod-to-Pod traffic),
  • Node firewall rules and security groups,
  • kube-proxy status and mode on nodes.
• To verify connectivity from within the cluster, create a temporary debug pod (busybox, curl images) and exec into it to curl Service IPs and pod IPs.
• For troubleshooting liveness/readiness probe failures, inspect probe definitions in the Pod spec and pod events for probe failure messages.

​

Links and references

• Kubernetes: Troubleshooting Applications — https://kubernetes.io/docs/tasks/debug-application-cluster/debug-application/
• Kubernetes Concepts: Services — https://kubernetes.io/docs/concepts/services-networking/service/
• kubectl documentation — https://kubernetes.io/docs/reference/kubectl/