Skip to main content
In this lesson we continue debugging a production incident. Our immediate questions are: who owns the payment namespace, and specifically who owns the failing deployment? Ownership metadata is commonly stored as labels (and sometimes annotations) on Kubernetes objects. We inspect deployment labels in the payment namespace to identify ownership and on-call information.

​
1) Inspect deployments and labels

Run:
Example output:
Observations:
  • payment-processor includes owner=payments-team.
  • payment-gateway-api is scaled to zero and has no owner label.
  • Labels are inconsistent across deployments (app, environment, cost-center, owner, etc.).
When triaging incidents, start by identifying metadata that points to an owning team or escalation channel — look for labels like owner, team, oncall, or structured app labels such as app.kubernetes.io/managed-by. Consistent labeling accelerates incident response and reduces mean time to resolution (MTTR).

​
2) Why labels (and ownership) matter

Inconsistent or missing ownership metadata makes it difficult to know who to contact during an incident. This is an organizational and governance problem that impacts escalation, incident retrospectives, and cost allocation (FinOps). Addressing it involves both process and automation:
  • Enforce labeling via admission controllers or CI checks.
  • Record owner and contact channel in a single, discoverable label (for example owner=payments-team or oncall=#payments).
  • Ensure labels are included in the source-of-truth repository so GitOps or CI/CD pipelines reconcile them automatically.

​
3) Immediate remediation — scale the deployment

To restore service quickly, scale payment-gateway-api to 1 replica:
You should see:
Confirm the pods are running:
Example output after scaling:

​
4) Validate the application behavior

In this lab the payment gateway is exposed on port 89. Open the gateway endpoint on that port and submit a test payment (use test card details appropriate for your environment). After the API pod becomes available the payment flow should complete successfully.

​
Important: make the fix durable

Directly changing resources in a production cluster (for example, scaling a deployment manually) fixes the symptom quickly but does not persist through a later GitOps or CI/CD reconciliation. Always follow your organization’s change process: make the fix in the source repository and pipeline so the change is durable and auditable.

​
5) Root cause and next steps

Although scaling resolved the incident, the root cause remains: inconsistent labeling and missing ownership metadata. Recommended next steps:
  • Add ownership labels in the source repository (e.g., owner, oncall, cost-center).
  • Implement automated checks to block deployments missing required metadata.
  • Consider a reconciliation agent or FinOps tool that surfaces missing owners and cost allocation.
  • Update runbooks to include how to discover ownership metadata quickly during incidents.

​
References

Watch Video

Practice Lab