Demo Audit Kubernetes Labels via MCP Agent

Welcome back. In this lesson we’ll run an automated label-audit against a Kubernetes cluster using the MCP (Multi-Cluster Proxy) agent. The audit is performed by a small Python script, label_auditor.py, which calls an LLM-backed ReAct-style agent via the MCP client. The agent enumerates namespaces and resources, inspects labels, flags missing governance labels (owner and cost-center), and returns a human-friendly summary. Below is the script used for this audit (complete and ready to run):

"""Ask a LangChain agent to audit cluster labels via MCP and print its final answer."""
import asyncio

from mcp_client import run_agent

PROMPT = """
You are auditing Kubernetes resource labels for FinOps governance through MCP tools.

Use the available Kubernetes MCP tools to:
1. List all namespaces.
2. For each application namespace (skip kube-* and local-path-storage), fetch deployments, configmaps, and persistentvolumeclaims.
3. Report each resource's name, namespace, kind, and labels exactly as returned by the tools.
4. Flag resources missing `owner` or `cost-center` labels.
5. Summarize the label audit in plain English.

Use only data returned by the tools. Do not invent labels or resources.
"""

async def main() -> None:
    print(await run_agent(PROMPT, max_tokens=2048))


if __name__ == "__main__":
    asyncio.run(main())

Key points about the script:

The prompt directs the agent to use only data returned by the MCP/Kubernetes tools (no hallucination).
max_tokens=2048 is set because the agent will both receive rich tool outputs and must return a detailed audit.
The script uses asyncio.run(...), so Python 3.7+ is required.

Before running the script, ensure your MCP server is running and reachable (for example at http://localhost:8000/mcp), and that your virtual environment includes the required packages. The script expects mcp_client.run_agent to be available.

Running the script in your terminal:

(.venv) ➜ k8-finops-agent git:(main) ✗ python3 sections/06-mcp-data-agent/code/label_auditor.py

Representative logs (sanitized and consolidated) that show the interaction with MCP and the LLM:

[INFO] MCP server "k8s": initializing with: {'url': 'http://localhost:8000/mcp', 'transport': 'streamable_http'}
[INFO] MCP server "k8s": Pre-validating authentication
[INFO] HTTP Request: POST http://localhost:8000/mcp "HTTP/1.1 200 OK"
[INFO] MCP server "k8s": Authentication validation passed: 200
[INFO] MCP server "k8s": connecting via Streamable HTTP (explicit) to http://localhost:8000/mcp
[INFO] Negotiated protocol version: 2025-11-25
[INFO] MCP server "k8s": connected
[INFO] MCP server "k8s": 8 tool(s) available:
- kubectl_get
- kubectl_describe
- kubectl_logs
- kubectl_context
- kubectl_reconnect
- explain_resource
- list_api_resources
- ping
[INFO] MCP servers initialized: 8 tool(s) available in total
[INFO] MCP tool "k8s"/"kubectl_get" received input: {'resourceType': 'namespaces', 'name': None, 'namespace': 'default', 'output': 'json', 'allNamespaces': False, 'labelSelector': None, 'fieldSelector': None, 'sortBy': None, 'context': ''}
[INFO] MCP tool "k8s"/"kubectl_get" received result (size: 1618)
[INFO] MCP tool "k8s"/"kubectl_get" received input: {'resourceType': 'deployments', 'name': None, 'namespace': 'airline', 'output': 'json', 'allNamespaces': False, 'labelSelector': None, 'fieldSelector': None, 'sortBy': None, 'context': ''}
[INFO] MCP tool "k8s"/"kubectl_get" received result (size: 535)
...
[INFO] MCP tool "k8s"/"kubectl_describe" received input: {'resourceType': 'configmap', 'name': 'kube-root-ca.crt', 'namespace': 'airline', 'context': '', 'allNamespaces': False}
[INFO] MCP tool "k8s"/"kubectl_describe" received input: {'resourceType': 'persistentvolumeclaim', 'name': 'orphaned-data-pvc', 'namespace': 'airline', 'context': '', 'allNamespaces': False}
[INFO] MCP tool "k8s"/"kubectl_describe" received input: {'resourceType': 'deployment', 'name': 'booking-api', 'namespace': 'booking-api', 'context': '', 'allNamespaces': False}
[INFO] MCP tool "k8s"/"kubectl_describe" received result (size: 1221)
[INFO] MCP tool "k8s"/"kubectl_describe" received result (size: 40632)
...
[INFO] HTTP Request: POST https://api.ai.kodekloud.com/v1/chat/completions "HTTP/1.1 200 OK"
[INFO] MCP server "k8s": session closed

How the audit flow works (step-by-step):

The script connects to the configured MCP server and discovers available tools (e.g., kubectl_get, kubectl_describe).
It lists all namespaces, filters application namespaces (skips kube-* and local-path-storage), and for each application namespace runs:
- kubectl get for deployments, configmaps, and persistentvolumeclaims (JSON output).
- kubectl describe for individual resources where more label metadata is required.
The collected tool outputs are sent to the LLM via chat/completions; the LLM analyzes the data and produces:
- A per-resource registry of labels as returned by the tools.
- Flags for resources missing owner or cost-center.
- A plain-English summary of findings.

Below is a representative audit report produced by the agent. This output is derived only from the tool responses returned by MCP; the agent formats and summarizes those results.

Kubernetes Resource Labels — FinOps Governance Audit Report

Executive Summary

This audit inspected 30 Kubernetes resources across 6 application namespaces (airline, booking-api, default, flight-search, inventory, payment) to check for required FinOps labels. The focus was on two required labels used for cost allocation and ownership: owner and cost-center.

Detailed Resource Inventory

NAMESPACE: airline

Resource Name	Kind	Labels	Owner Label	Cost-Center Label	Status
(Resources and rows for this namespace would appear here — the agent enumerated and described each resource using `kubectl_get` and `kubectl_describe`.)

NAMESPACE: booking-api

Resource Name	Kind	Labels	Owner Label	Cost-Center Label	Status
booking-api	Deployment	`app.kubernetes.io/name=booking-api`, `app.kubernetes.io/tier=backend`, `cost-center=booking-engine`, `environment=prod`	❌ Missing	✅ Present (`booking-engine`)	FLAGGED
booking-api-config	ConfigMap	`app.kubernetes.io/name=booking-api`, `cost-center=booking-engine`	❌ Missing	✅ Present (`booking-engine`)	FLAGGED
kube-root-ca.crt	ConfigMap	`<none>`	❌ Missing	❌ Missing	FLAGGED
booking-api-data	PersistentVolumeClaim	`app.kubernetes.io/name=booking-api`, `cost-center=booking-engine`	❌ Missing	✅ Present (`booking-engine`)	FLAGGED

Namespace Summary: 4 resources, 4 flagged (100% missing owner label)

NAMESPACE: default

Resource Name	Kind	Labels	Owner Label	Cost-Center Label	Status
analytics-collector	Deployment	`app=analytics`	❌ Missing	❌ Missing	FLAGGED
analytics-config	ConfigMap	`<none>`	❌ Missing	❌ Missing	FLAGGED
kube-root-ca.crt	ConfigMap	`<none>`	❌ Missing	❌ Missing	FLAGGED
untracked-config	ConfigMap	`<none>`	❌ Missing	❌ Missing	FLAGGED
analytics-data	PersistentVolumeClaim	`<none>`	❌ Missing	❌ Missing	FLAGGED

Namespace Summary: 5 resources, 5 flagged (100% non-compliant)

NAMESPACE: flight-search

Resource Name	Kind	Labels	Owner Label	Cost-Center Label	Status
flight-search-service	Deployment	`app=flight-search`, `app.kubernetes.io/tier=backend`, `environment=prod`	❌ Missing	❌ Missing	FLAGGED
flight-search-config	ConfigMap	`app=flight-search`	❌ Missing	❌ Missing	FLAGGED
kube-root-ca.crt	ConfigMap	`<none>`	❌ Missing	❌ Missing	FLAGGED
flight-search-data	PersistentVolumeClaim	`app=flight-search`	❌ Missing	❌ Missing	FLAGGED

Namespace Summary: 4 resources, 4 flagged (100% non-compliant)

NAMESPACE: inventory

Resource Name	Kind	Labels	Owner Label	Cost-Center Label	Status
inventory-service	Deployment	`app.kubernetes.io/name=inventory-service`, `app.kubernetes.io/tier=backend`, `cost-center=inventory`, `criticality=P2`, `environment=prod`, `owner=inventory-team`	✅ Present (`inventory-team`)	✅ Present (`inventory`)	COMPLIANT
inventory-config	ConfigMap	`app.kubernetes.io/name=inventory-service`, `app.kubernetes.io/tier=backend`, `compliance-level=standard`, `cost-center=inventory`, `environment=prod`, `owner=inventory-team`	✅ Present (`inventory-team`)	✅ Present (`inventory`)	COMPLIANT
kube-root-ca.crt	ConfigMap	`<none>`	❌ Missing	❌ Missing	FLAGGED
inventory-data	PersistentVolumeClaim	`app.kubernetes.io/name=inventory-service`, `app.kubernetes.io/tier=backend`, `compliance-level=standard`, `cost-center=inventory`, `environment=prod`, `owner=inventory-team`	✅ Present (`inventory-team`)	✅ Present (`inventory`)	COMPLIANT

Namespace Summary: 4 resources, 1 flagged (75% compliant)

NAMESPACE: payment

Resource Name	Kind	Labels	Owner Label	Cost-Center Label	Status
payment-gateway	Deployment	`app=payment-gateway`, `app.kubernetes.io/name=payment-gateway`, `app.kubernetes.io/tier=frontend`, `environment=prod`	❌ Missing	❌ Missing	FLAGGED
payment-gateway-api	Deployment	`app=payment-gateway-api`, `app.kubernetes.io/name=payment-gateway-api`, `app.kubernetes.io/tier=backend`, `environment=prod`	❌ Missing	❌ Missing	FLAGGED
payment-processor	Deployment	`app.kubernetes.io/tier=backend`, `cost-center=payment`, `environment=prod`, `owner=payments-team`	✅ Present (`payments-team`)	✅ Present (`payment`)	COMPLIANT
kube-root-ca.crt	ConfigMap	`<none>`	❌ Missing	❌ Missing	FLAGGED
payment-config	ConfigMap	`environment=prod`, `owner=payments-team`	✅ Present (`payments-team`)	❌ Missing	FLAGGED
payment-gateway-api-config	ConfigMap	`app=payment-gateway-api`	❌ Missing	❌ Missing	FLAGGED

Observations and recommended next steps:

The agent-produced report provides a quick first-pass inventory and highlights missing labels. Because the agent is restricted to using only returned tool data, results are limited to what Kubernetes APIs returned.
For consistent, audit-ready checks, pair the agent output with a machine-readable tagging policy (a ruleset). That enables deterministic checks and easy enforcement across teams.
Next, we will show how to merge agent-collected label data with an explicit tagging policy so you can produce deterministic, actionable reports (and drive automated remediation or alerts).

That’s it for this lesson — you now have a compact script and workflow to audit Kubernetes labels via an MCP-connected ReAct agent.

Links and references

Kubernetes Basics
Kubernetes Labels and Selectors
Python asyncio documentation
MCP / multi-cluster tooling (refer to your organization’s MCP server docs or the MCP project docs)

Introduction

The Problem Statement

Introduction to FinOps

Setting Up SkyLine Air Kubernetes Services

MCP Gateway for Kubernetes

LangChain MCP Agents

LLM Policy-Aware Label Audit

From Findings to Actionable Tickets

Demo Audit Kubernetes Labels via MCP Agent

Kubernetes Resource Labels — FinOps Governance Audit Report

Executive Summary

Detailed Resource Inventory

NAMESPACE: airline

NAMESPACE: booking-api

NAMESPACE: default

NAMESPACE: flight-search

NAMESPACE: inventory

NAMESPACE: payment

Links and references

Watch Video

Practice Lab

​Kubernetes Resource Labels — FinOps Governance Audit Report

​Executive Summary

​Detailed Resource Inventory

​NAMESPACE: airline

​NAMESPACE: booking-api

​NAMESPACE: default

​NAMESPACE: flight-search

​NAMESPACE: inventory

​NAMESPACE: payment

​Links and references

Watch Video

Practice Lab

Kubernetes Resource Labels — FinOps Governance Audit Report

Executive Summary

Detailed Resource Inventory

NAMESPACE: airline

NAMESPACE: booking-api

NAMESPACE: default

NAMESPACE: flight-search

NAMESPACE: inventory

NAMESPACE: payment

Links and references