> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# 2 AM Incident

> Guide for fast Kubernetes troubleshooting of a payment gateway outage, distinguishing cluster level issues from application level errors with practical checks and commands

Hello — in this lesson we'll reproduce the debugging approach Elena used when she was paged at 2 a.m. The incident points to a payment gateway outage. This guide walks through fast, practical checks you can run to determine whether the problem is Kubernetes-level (pods, services, probes) or application-level (runtime errors, misconfiguration).

1. Confirm cluster access

* Before doing anything, verify you have the correct access to the production Kubernetes cluster. Access to production clusters typically requires an approval process; do not bypass company policy.

Accessing production clusters usually requires explicit approvals and auditing. Ensure you have the right permissions and follow your organization's change and access control policies before running commands.

Run a quick command to confirm you can reach the cluster and list namespaces:

```bash theme={null}
kubectl get namespaces
```

Example output:

```bash theme={null}
controlplane ~ » kubectl get namespaces
NAME               STATUS    AGE
airline            Active    7m50s
booking-api        Active    7m51s
default            Active    10h
flight-search      Active    7m51s
inventory          Active    7m50s
kube-node-lease    Active    10h
kube-public        Active    10h
kube-system        Active    10h
payment            Active    7m50s

controlplane ~ »
```

Note: the `payment` namespace appears in the output — this aligns with the alert about the payment service/gateway being down. That makes `payment` the obvious place to continue your investigation.

2. Inspect resources in the payment namespace
   List all workloads, services, and replica sets in the `payment` namespace:

```bash theme={null}
kubectl get all -n payment
```

Example output:

```bash theme={null}
controlplane ~ » kubectl get all -n payment
NAME                                           READY   STATUS    RESTARTS   AGE
pod/payment-gateway-5dbc999f64-mvmdn           1/1     Running   0          8m58s
pod/payment-processor-7f45c86c6d-5csjm         1/1     Running   0          9m2s

NAME                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)      AGE
service/payment-gateway           ClusterIP   10.43.242.173    <none>        80/TCP       8m58s
service/payment-gateway-api       ClusterIP   10.43.68.252     <none>        8080/TCP     8m58s
service/payment-processor         ClusterIP   10.43.123.248    <none>        8443/TCP     9m2s

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/payment-gateway           1/1     1            1           8m58s
deployment.apps/payment-gateway-api       0/0     0            0           8m58s
deployment.apps/payment-processor         1/1     1            1           9m2s

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/payment-gateway-5dbc999f64             1         1         1       8m58s
replicaset.apps/payment-gateway-api-7dd7bd5999         0         0         0       8m58s
replicaset.apps/payment-processor-7f45c86c6d           1         1         1       9m2s

controlplane ~ »
```

Observations from the output:

* Several resources exist for payment: pods, services, deployments, replica sets.
* All pods shown are in `Running` state, but `Running` does not guarantee the application is healthy — further checks are required.
* Note a deployment `payment-gateway-api` with `0/0` ready pods and a corresponding replica set with `0/0` — this likely indicates a scaling or scheduling issue for that component.

Quick reference: what each resource type tells you

| Resource | What to check | Example command |
| - | - | - |
| Pod | Check container runtime logs, readiness & liveness probe failures, recent restarts | `kubectl describe pod -n payment <pod-name>` |
| Deployment | Desired vs available replicas, rollout history | `kubectl get deployment -n payment` |
| Service | How the app is exposed (ClusterIP vs LoadBalancer), ports | `kubectl get svc -n payment` |
| ReplicaSet | Which pods match the deployment template | `kubectl get rs -n payment` |

3. Inspect how the application is exposed
   To confirm whether the service is reachable within the cluster, list services in `payment`:

```bash theme={null}
kubectl get svc -n payment
```

Example output:

```bash theme={null}
controlplane ~ » kubectl get svc -n payment
NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE
payment-gateway        ClusterIP   10.43.242.173   <none>        80/TCP      10m
payment-gateway-api    ClusterIP   10.43.68.252    <none>        8080/TCP    10m
payment-processor      ClusterIP   10.43.123.248   <none>        8443/TCP    10m

controlplane ~ »
```

Notes:

* These are `ClusterIP` services, meaning they are reachable only from within the cluster network (unless there is an Ingress or LoadBalancer in front).
* A `ClusterIP` exists and points to a cluster-internal IP, but that does not guarantee the backend pods respond to requests on their ports.

4. Recommended next debugging steps
   Choose one or more of the following, depending on whether you can access logs or need to test endpoints locally.

* Tail pod logs to look for runtime errors or stack traces:
  ```bash theme={null}
  # Example: tail logs for the gateway pod
  kubectl logs -n payment pod/payment-gateway-5dbc999f64-mvmdn
  ```

* Inspect pod events and probe statuses to detect readiness/liveness probe failures or scheduling problems:
  ```bash theme={null}
  kubectl describe pod -n payment pod/payment-gateway-5dbc999f64-mvmdn
  ```

* Port-forward a service to your local machine to test HTTP endpoints if the service is ClusterIP:
  ```bash theme={null}
  # Forward local port 8080 to service port 80
  kubectl port-forward -n payment svc/payment-gateway 8080:80

  # Then test locally:
  curl http://localhost:8080/health
  ```

* If the `payment-gateway-api` deployment shows 0 desired/available replicas, check its deployment spec and scheduler events:
  ```bash theme={null}
  kubectl get deployment -n payment payment-gateway-api -o yaml
  kubectl describe deployment -n payment payment-gateway-api
  ```

Fast triage pattern: confirm cluster access → identify the relevant namespace → combine pod-level checks (logs, `describe`) with service-level checks (port-forward or internal probes). This helps you determine whether the issue is network/configuration-related (service, probes, scaling) or application-related (exceptions, crashes).

Links and references

* Kubernetes concepts: [https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/)
* kubectl reference: [https://kubernetes.io/docs/reference/kubectl/](https://kubernetes.io/docs/reference/kubectl/)
* Port forwarding: [https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/)
* Troubleshooting pods: [https://kubernetes.io/docs/tasks/debug-application-cluster/debug-pod-replication-controller/](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-pod-replication-controller/)

> **[Watch Video](https://learn.kodekloud.com/user/courses/building-kubernetes-finops-ai-agent/module/5d434b76-bd5d-4526-b9c6-4a994c195994/lesson/24101440-51e3-48eb-b121-6b45d72d881b)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.