Skip to main content
Welcome back. This walkthrough reproduces the investigation Elena performed after being paged at 2 a.m. for a production incident involving the payment gateway. We follow the same sequence of checks you would use during a real production troubleshooting session, focusing on Kubernetes-level verification and service-level diagnostics. Before you run anything in production, confirm you have access to the production Kubernetes cluster and the appropriate approvals. Production kubeconfigs and contexts are typically audited.
Before you start running commands in a production cluster, ensure you have the appropriate approvals and that your kubeconfig/context points to the correct cluster.

​
Step 1 — Verify cluster access and find the namespace

Start by listing namespaces to confirm you can reach the cluster and to identify where the payment components live.
Example output:
Observations:
  • A payment namespace exists — this is our primary target based on the alert (“payment service down”).
  • Confirming namespace presence is a quick way to assert you have cluster visibility before deeper debugging.

​
Step 2 — Inspect resources inside the payment namespace

List all high-level resources (pods, services, deployments, replica sets) to get an immediate surface-level view.
Example output:
Key takeaways:
  • Some pods are Running, but that does not guarantee the application is healthy or accepting traffic.
  • The payment-gateway-api deployment shows 0 replicas — this is notable and could indicate the API has been scaled to zero, failed to create pods, or is stuck on rollout.
Important: Running containers can still fail readiness checks or return internal application errors. Always check readiness, events, and logs.

​
Two primary next investigation paths

  1. Check pod logs, events, and readiness/liveness probe results to surface runtime errors.
  2. Verify service endpoints and traffic routing, and if needed, test the API from inside the cluster.

​
Step 3 — List services to confirm how the app is exposed

Get services in the payment namespace to confirm cluster-internal exposure and ports.
Example output:
Notes:
  • These are ClusterIP services (internal cluster addresses). EXTERNAL-IP is <none>, which is expected for ClusterIP.
  • To reach these services from outside the cluster, use port-forwarding, an ingress, or a debug pod.

​
Useful commands and why to run them

Below are the commands you’ll commonly use during the next phase of debugging, with the reason to run each. Examples (commands shown earlier in context):

​
How to interpret different outcomes

  • Endpoints missing for a service:
    • Service cannot route traffic. Investigate pod labels/selectors and deployment scaling. Check if replicas were intentionally scaled to zero or if pods failed to start.
  • Pods Running but logs show errors or readiness probe failures:
    • Fix the application error, adjust configuration, or roll back to a known-good image/configuration. Look at probe definitions in the Deployment manifest and at recent changes in the image or config.
  • Service healthy internally but external users experiencing failures:
    • Investigate ingress controllers, external load balancers, network policies, and DNS. Ensure the path from the external load balancer to the ClusterIP is healthy.

​
Quick checklist for narrowing root cause

  • Confirm cluster access and correct kube-context.
  • Verify the payment namespace and list resources with kubectl get all -n payment.
  • Check service endpoints (missing endpoints often indicate selector/label mismatches or zero replicas).
  • Describe pods for events and readiness/liveness probe failures.
  • Stream or inspect logs (including --previous) for runtime exceptions or stack traces.
  • Port-forward or run a debug pod to test cluster-internal connectivity and API responses.
  • If necessary, check ingress/load balancer and network policies for external reachability issues.
Proceed through the above commands (describe, logs, endpoints, port-forward) to determine whether the incident stems from an application-level error, a readiness probe failure, a deployment/replica issue, or a networking/service-level problem.

Watch Video