> ## Documentation Index > Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt > Use this file to discover all available pages before exploring further. # Demo Debugging a Production Incident in Skyline Air > Walkthrough diagnosing a production payment service incident on Kubernetes, covering namespace checks, pods, services, logs, probes, endpoints, and debugging commands Welcome back. This walkthrough reproduces the investigation Elena performed after being paged at 2 a.m. for a production incident involving the payment gateway. We follow the same sequence of checks you would use during a real production troubleshooting session, focusing on Kubernetes-level verification and service-level diagnostics. Before you run anything in production, confirm you have access to the production Kubernetes cluster and the appropriate approvals. Production kubeconfigs and contexts are typically audited. Before you start running commands in a production cluster, ensure you have the appropriate approvals and that your kubeconfig/context points to the correct cluster. ## Step 1 — Verify cluster access and find the namespace Start by listing namespaces to confirm you can reach the cluster and to identify where the payment components live. ```bash theme={null} # Verify cluster access and see available namespaces kubectl get namespaces ``` Example output: ```bash theme={null} controlplane ~ » kubectl get namespaces NAME STATUS AGE airline Active 7m50s booking-api Active 7m51s default Active 10h flight-search Active 7m51s inventory Active 7m50s kube-node-lease Active 10h kube-public Active 10h kube-system Active 10h payment Active 7m50s ``` Observations: * A `payment` namespace exists — this is our primary target based on the alert ("payment service down"). * Confirming namespace presence is a quick way to assert you have cluster visibility before deeper debugging. ## Step 2 — Inspect resources inside the payment namespace List all high-level resources (pods, services, deployments, replica sets) to get an immediate surface-level view. ```bash theme={null} # Show pods, services, deployments, etc. in the payment namespace kubectl get all -n payment ``` Example output: ```bash theme={null} controlplane ~ » kubectl get all -n payment NAME READY STATUS RESTARTS AGE pod/payment-gateway-5dbc999f64-mvmdn 1/1 Running 0 8m58s pod/payment-processor-7f45c86c6d-5csjm 1/1 Running 0 9m2s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/payment-gateway ClusterIP 10.43.242.173 80/TCP 8m58s service/payment-gateway-api ClusterIP 10.43.68.252 8080/TCP 8m58s service/payment-processor ClusterIP 10.43.123.248 8443/TCP 9m2s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/payment-gateway 1/1 1 1 8m58s deployment.apps/payment-gateway-api 0/0 0 0 8m58s deployment.apps/payment-processor 1/1 1 1 9m2s NAME DESIRED CURRENT READY AGE replicaset.apps/payment-gateway-5dbc999f64 1 1 1 8m58s replicaset.apps/payment-gateway-api-7dd7bd5999 0 0 0 8m58s replicaset.apps/payment-processor-7f45c86c6d 1 1 1 9m2s controlplane ~ » ``` Key takeaways: * Some pods are `Running`, but that does not guarantee the application is healthy or accepting traffic. * The `payment-gateway-api` deployment shows `0` replicas — this is notable and could indicate the API has been scaled to zero, failed to create pods, or is stuck on rollout. Important: Running containers can still fail readiness checks or return internal application errors. Always check readiness, events, and logs. ## Two primary next investigation paths 1. Check pod logs, events, and readiness/liveness probe results to surface runtime errors. 2. Verify service endpoints and traffic routing, and if needed, test the API from inside the cluster. ## Step 3 — List services to confirm how the app is exposed Get services in the `payment` namespace to confirm cluster-internal exposure and ports. ```bash theme={null} kubectl get svc -n payment ``` Example output: ```bash theme={null} controlplane ~ » kubectl get svc -n payment NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE payment-gateway ClusterIP 10.43.242.173 80/TCP 10m payment-gateway-api ClusterIP 10.43.68.252 8080/TCP 10m payment-processor ClusterIP 10.43.123.248 8443/TCP 10m controlplane ~ » ``` Notes: * These are `ClusterIP` services (internal cluster addresses). `EXTERNAL-IP` is ``, which is expected for ClusterIP. * To reach these services from outside the cluster, use port-forwarding, an ingress, or a debug pod. ## Useful commands and why to run them Below are the commands you'll commonly use during the next phase of debugging, with the reason to run each. | Command | Purpose | | - | - | | `kubectl get endpoints payment-gateway -n payment` | Verify which pod IPs are registered behind the service. | | `kubectl describe svc payment-gateway -n payment` | See endpoints, selectors, labels, and service metadata. | | `kubectl describe pod -n payment` | Inspect pod events, container states, readiness/liveness results, and recent errors. | | `kubectl logs -f -n payment` | Stream logs from the pod to watch runtime errors. | | `kubectl logs --previous -n payment` | Fetch logs from the previous container instance if the container restarted. | | `kubectl logs -f -c -n payment` | For multi-container pods, stream logs from a specific container. | | `kubectl port-forward svc/payment-gateway 8080:80 -n payment` | Expose the ClusterIP service locally for quick `curl` testing. | | `kubectl run -it --rm debug --image=radial/busyboxplus:curl --restart=Never -n payment -- sh` | Start a transient debug pod inside the cluster to `curl` services by their ClusterIP/DNS. | Examples (commands shown earlier in context): ```bash theme={null} # See endpoints backing the service kubectl get endpoints payment-gateway -n payment # Or describe the service to see endpoints and other metadata kubectl describe svc payment-gateway -n payment ``` ```bash theme={null} # Get detailed information about a pod (events, conditions, container states) kubectl describe pod payment-gateway-5dbc999f64-mvmdn -n payment ``` ```bash theme={null} # Stream logs from a pod kubectl logs -f payment-gateway-5dbc999f64-mvmdn -n payment # If multiple containers exist, specify the container name: kubectl logs -f payment-gateway-5dbc999f64-mvmdn -c -n payment # If the container restarted and you want the previous logs: kubectl logs --previous payment-gateway-5dbc999f64-mvmdn -n payment ``` ```bash theme={null} # Forward service port 80 to local port 8080 kubectl port-forward svc/payment-gateway 8080:80 -n payment # Then in another terminal: curl http://localhost:8080/health ``` ```bash theme={null} # Run a one-off debug pod inside the cluster and test the ClusterIP directly kubectl run -it --rm debug --image=radial/busyboxplus:curl --restart=Never -n payment -- sh # Inside the debug pod: curl http://payment-gateway:80/health ``` ## How to interpret different outcomes * Endpoints missing for a service: * Service cannot route traffic. Investigate pod labels/selectors and deployment scaling. Check if replicas were intentionally scaled to zero or if pods failed to start. * Pods `Running` but logs show errors or readiness probe failures: * Fix the application error, adjust configuration, or roll back to a known-good image/configuration. Look at probe definitions in the Deployment manifest and at recent changes in the image or config. * Service healthy internally but external users experiencing failures: * Investigate ingress controllers, external load balancers, network policies, and DNS. Ensure the path from the external load balancer to the ClusterIP is healthy. ## Quick checklist for narrowing root cause * [ ] Confirm cluster access and correct kube-context. * [ ] Verify the `payment` namespace and list resources with `kubectl get all -n payment`. * [ ] Check service endpoints (missing endpoints often indicate selector/label mismatches or zero replicas). * [ ] Describe pods for events and readiness/liveness probe failures. * [ ] Stream or inspect logs (including `--previous`) for runtime exceptions or stack traces. * [ ] Port-forward or run a debug pod to test cluster-internal connectivity and API responses. * [ ] If necessary, check ingress/load balancer and network policies for external reachability issues. Proceed through the above commands (describe, logs, endpoints, port-forward) to determine whether the incident stems from an application-level error, a readiness probe failure, a deployment/replica issue, or a networking/service-level problem. ## Links and references * [Kubernetes Concepts — Services](https://kubernetes.io/docs/concepts/services-networking/service/) * [kubectl reference](https://kubernetes.io/docs/reference/kubectl/) * [Debugging Pods](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-pod-replication-controller/) > **[Watch Video](https://learn.kodekloud.com/user/courses/building-kubernetes-finops-ai-agent/module/5d434b76-bd5d-4526-b9c6-4a994c195994/lesson/52b3db7f-ff0a-4b5b-bdf3-302a3d202101)** This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.