> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Demo Exploring the Labels in Our Cluster

> Demonstrates Kubernetes label inconsistencies across namespaces, explains impacts on operations and billing, and proposes a minimal labeling policy plus enforcement and remediation options.

Hello and welcome back.

Now that our deployment is complete, let's go straight to the core problem: inconsistent labeling. When Alina was troubleshooting an incident, inconsistent label naming prevented her from quickly finding the right team and routing the incident. With the SkyLine Air cluster fully deployed, we'll inspect how labeling varies across namespaces and deployments and then outline how to enforce and remediate a consistent labeling standard.

## What we'll do

1. List namespaces in the cluster.
2. Inspect deployments and their labels in two namespaces (inventory and payment).
3. Observe inconsistencies and summarize the gaps.
4. Propose a minimal labeling standard and enforcement options.

## 1) List namespaces

Run this command to see the namespaces in the cluster:

```bash theme={null}
kubectl get namespaces
```

Example output:

```text theme={null}
NAME                   STATUS    AGE
airline                Active    4d
booking-api            Active    4d
default                Active    4d
flight-search          Active    4d
inventory              Active    4d
kube-node-lease        Active    4d
kube-public            Active    4d
kube-system            Active    4d
local-path-storage     Active    4d
payment                Active    4d
```

## 2) Inspect deployments and labels

First, check deployments and the labels they expose in the `inventory` namespace:

```bash theme={null}
kubectl get deployments -n inventory --show-labels
```

Example output:

```text theme={null}
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE   LABELS
inventory-service    1/1     1            1           4d    app.kubernetes.io/name=inventory-service,app.kubernetes.io/tier=backend,cost-center=inventory,criticality=P2,environment=prod,owner=inventory-team
```

Next, inspect the `payment` namespace:

```bash theme={null}
kubectl get deployments -n payment --show-labels
```

Example output:

```text theme={null}
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE   LABELS
payment-gateway         1/1     1            1           4d    app.kubernetes.io/name=payment-gateway,app.kubernetes.io/tier=frontend,app=payment-gateway,environment=prod
payment-gateway-api     0/0     0            0           4d    app.kubernetes.io/name=payment-gateway-api,app.kubernetes.io/tier=backend,app=payment-gateway-api,environment=prod
payment-processor       1/1     1            1           4d    app.kubernetes.io/tier=backend,cost-center=payment,environment=prod,owner=payments-team
```

## 3) Observations — label inconsistency

From the outputs above:

* inventory namespace
  * `inventory-service` includes: `app.kubernetes.io/name`, `app.kubernetes.io/tier`, `cost-center`, `criticality`, `environment`, and `owner`.
  * This mix contains both operational metadata (owner, environment) and business metadata (cost-center, criticality).

* payment namespace
  * `payment-gateway` has `app.kubernetes.io/name`, `app.kubernetes.io/tier`, `app`, and `environment`, but lacks `owner`, `criticality`, `cost-center`.
  * `payment-gateway-api` similarly lacks `owner`, `criticality`, `cost-center`.
  * `payment-processor` includes `cost-center` and `owner` but does not include `app.kubernetes.io/name` in the same format as the others.

This inconsistency occurs both between namespaces and between deployments in the same namespace.

### Quick summary table

| Deployment | Namespace | Labels present (examples) | Missing / inconsistent keys |
| -: | - | - | - |
| `inventory-service` | `inventory` | `app.kubernetes.io/name`, `app.kubernetes.io/tier`, `cost-center`, `criticality`, `environment`, `owner` | — |
| `payment-gateway` | `payment` | `app.kubernetes.io/name`, `app.kubernetes.io/tier`, `app`, `environment` | `owner`, `cost-center`, `criticality` |
| `payment-gateway-api` | `payment` | `app.kubernetes.io/name`, `app.kubernetes.io/tier`, `app`, `environment` | `owner`, `cost-center`, `criticality` |
| `payment-processor` | `payment` | `app.kubernetes.io/tier`, `cost-center`, `environment`, `owner` | `app.kubernetes.io/name` format inconsistent |

## 4) Why this matters

In a large organization (many clusters and many teams), inconsistent labels cause problems with:

* Auditing and inventory reporting
* Cost allocation and chargeback (FinOps)
* Incident routing and ownership (SRE / on-call)
* Automated tooling (monitoring, dashboards, autoscaling or policy enforcement)

Who applies labels? Engineering teams typically add labels as they deploy resources. Who defines the standard? Platform teams, SRE, and FinOps should collaborate to define required keys, allowed values, and naming conventions. Enforcement mechanisms (described next) are needed so teams don’t drift from the standard.

## 5) Suggested minimal label policy

Consider this minimal set of required labels and conventions:

* Required keys:
  * `owner` — team or person responsible (e.g., `payments-team`)
  * `environment` — `prod`, `staging`, `dev`
  * `cost-center` — cost or billing code
* Kubernetes recommended labels:
  * `app.kubernetes.io/name` — application name
  * `app.kubernetes.io/tier` — `frontend`, `backend`, `middleware`
* Optional but useful:
  * `criticality` — e.g., `P0`, `P1`, `P2`
  * `release` — release/tag identifier
* Naming conventions:
  * Use lowercase, hyphen-separated team and app names (e.g., `payments-team`, `payment-gateway`)
  * `environment` should be one of a finite list (`prod`, `preprod`, `staging`, `dev`)

Define a label policy (required keys, allowed values, and naming rules) and enforce it via admission controllers (e.g., [Open Policy Agent](https://www.openpolicyagent.org/) / [Gatekeeper](https://open-policy-agent.github.io/gatekeeper/)) or CI checks so resources are labeled consistently at creation time.

## 6) Enforcement and remediation options

* Admission controllers
  * OPA/Gatekeeper: enforce required labels and allowed values during resource creation.
  * Kyverno: another policy engine focused on Kubernetes resource mutation and validation.
* CI / GitOps checks
  * Lint manifests in CI pipelines to ensure required labels exist before merging.
* Automated remediation
  * Controllers that add missing labels (with audit logs), or assign default values where safe.
* Reporting and alerts
  * Continuous audits that report unlabeled resources and alert owners.
* Documentation and templates
  * Provide manifest templates and starter charts that include required labels.

## 7) How to audit labels across the cluster

* Find all pods/deployments missing a required label (example using `kubectl` + `jq` or `grep`):

```bash theme={null}
# Deployments missing the 'owner' label in all namespaces
kubectl get deployments --all-namespaces -o json \
  | jq -r '.items[] | select(.metadata.labels.owner | not) | "\(.metadata.namespace)/\(.metadata.name)"'
```

* Search cluster resources for specific labels:

```bash theme={null}
# All resources with cost-center=payment
kubectl get all --all-namespaces -l cost-center=payment
```

* Use label-based reporting in dashboards and cost tools (e.g., Prometheus, Grafana, cloud billing exports).

## 8) Next steps / recommended rollout

1. Define the label policy (required keys, allowed values, naming rules).
2. Publish policy and templates to engineering teams.
3. Implement enforcement (Gatekeeper / Kyverno) in a staged rollout (non-blocking → audit → deny).
4. Add CI checks to catch label issues at PR time.
5. Implement reporting and remediation workflows for existing resources.

## Links and references

* Kubernetes labels and selectors: [https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)
* Open Policy Agent (OPA): [https://www.openpolicyagent.org/](https://www.openpolicyagent.org/)
* Gatekeeper: [https://open-policy-agent.github.io/gatekeeper/](https://open-policy-agent.github.io/gatekeeper/)
* Kyverno: [https://kyverno.io/](https://kyverno.io/)

This walkthrough highlights the real operational impact of inconsistent labels and provides practical steps to standardize, enforce, and remediate labeling across SkyLine Air (or any organization).

> **[Watch Video](https://learn.kodekloud.com/user/courses/building-kubernetes-finops-ai-agent/module/bd335eac-9e8a-4dc2-963b-1db50f6971f9/lesson/4135c5dd-dacd-48f7-9b6f-8d8f54c4db99)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.