> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Demo Ownership Wall Fix and Validate

> Guide to using Kubernetes labels to find ownership, scale a failing payment deployment, validate service functionality, and make ownership fixes durable with repository and automation.

In this lesson we continue debugging a production incident. Our immediate questions are: who owns the `payment` namespace, and specifically who owns the failing deployment? Ownership metadata is commonly stored as labels (and sometimes annotations) on Kubernetes objects. We inspect deployment labels in the `payment` namespace to identify ownership and on-call information.

## 1) Inspect deployments and labels

Run:

```bash theme={null}
kubectl get deployments -n payment --show-labels
```

Example output:

```bash theme={null}
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE    LABELS
payment-gateway         1/1     1            1           7m35s app.kubernetes.io/name=payment-gateway,app.kubernetes.io/tier=frontend,app=payment-gateway,environment=prod
payment-gateway-api     0/0     0            0           7m35s app.kubernetes.io/name=payment-gateway-api,app.kubernetes.io/tier=backend,app=payment-gateway-api,environment=prod
payment-processor       1/1     1            1           7m42s app.kubernetes.io/tier=backend,cost-center=payment,environment=prod,owner=payments-team
```

Observations:

* `payment-processor` includes `owner=payments-team`.
* `payment-gateway-api` is scaled to zero and has no `owner` label.
* Labels are inconsistent across deployments (`app`, `environment`, `cost-center`, `owner`, etc.).

When triaging incidents, start by identifying metadata that points to an owning team or escalation channel — look for labels like `owner`, `team`, `oncall`, or structured app labels such as `app.kubernetes.io/managed-by`. Consistent labeling accelerates incident response and reduces mean time to resolution (MTTR).

## 2) Why labels (and ownership) matter

Inconsistent or missing ownership metadata makes it difficult to know who to contact during an incident. This is an organizational and governance problem that impacts escalation, incident retrospectives, and cost allocation (FinOps). Addressing it involves both process and automation:

* Enforce labeling via admission controllers or CI checks.
* Record owner and contact channel in a single, discoverable label (for example `owner=payments-team` or `oncall=#payments`).
* Ensure labels are included in the source-of-truth repository so GitOps or CI/CD pipelines reconcile them automatically.

## 3) Immediate remediation — scale the deployment

To restore service quickly, scale `payment-gateway-api` to 1 replica:

```bash theme={null}
kubectl scale deployment payment-gateway-api -n payment --replicas=1
```

You should see:

```bash theme={null}
deployment.apps/payment-gateway-api scaled
```

Confirm the pods are running:

```bash theme={null}
kubectl get pods -n payment
```

Example output after scaling:

```bash theme={null}
NAME                                   READY   STATUS    RESTARTS   AGE
payment-gateway-5dbc000f64-vn2hk       1/1     Running   0          19m
payment-gateway-api-7dd7bd5999-qqdw2   1/1     Running   0          13s
payment-processor-7f45c86c6d-5dnzg     1/1     Running   0          10m
```

## 4) Validate the application behavior

In this lab the payment gateway is exposed on port `89`. Open the gateway endpoint on that port and submit a test payment (use test card details appropriate for your environment). After the API pod becomes available the payment flow should complete successfully.

## Important: make the fix durable

Directly changing resources in a production cluster (for example, scaling a deployment manually) fixes the symptom quickly but does not persist through a later GitOps or CI/CD reconciliation. Always follow your organization’s change process: make the fix in the source repository and pipeline so the change is durable and auditable.

## 5) Root cause and next steps

Although scaling resolved the incident, the root cause remains: inconsistent labeling and missing ownership metadata. Recommended next steps:

* Add ownership labels in the source repository (e.g., `owner`, `oncall`, `cost-center`).
* Implement automated checks to block deployments missing required metadata.
* Consider a reconciliation agent or FinOps tool that surfaces missing owners and cost allocation.
* Update runbooks to include how to discover ownership metadata quickly during incidents.

## References

* Kubernetes Labels and Selectors — [https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)
* GitOps best practices — [https://www.gitops.tech/](https://www.gitops.tech/)
* FinOps Foundation — [https://www.finops.org/](https://www.finops.org/)

> **[Watch Video](https://learn.kodekloud.com/user/courses/building-kubernetes-finops-ai-agent/module/5d434b76-bd5d-4526-b9c6-4a994c195994/lesson/66f4f79e-b4fc-4228-ae89-2d034d10b30e)**

> **[Practice Lab](https://learn.kodekloud.com/user/courses/building-kubernetes-finops-ai-agent/module/5d434b76-bd5d-4526-b9c6-4a994c195994/lesson/25e5b577-c469-419e-9cff-2e0a5c04efe0)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.