> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Demo Run the Policy Auditor Agent

> Guide to running and comparing baseline and structured Kubernetes label auditing agents using MCP, demonstrating policy-aware LLM prompts for consistent, machine-friendly compliance reports.

Welcome back. In this lesson you'll run the auditor agent and compare its output to the earlier baseline auditor run. The improved "structured" agent includes the tagging rules and a system prompt when it queries the LLM, producing a consistent, policy-aware report.

Prerequisites

* A running kind cluster used by the lab (example name: `finops-cluster`).
* Docker available to run the MCP service.
* A Python virtual environment with the auditor scripts installed.
* (Optional) Familiarity with Kubernetes labeling/tagging best practices: [Kubernetes Basics](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/)

Quick command reference

| Purpose | Command |
| - | - |
| Generate kubeconfig for kind (internal) | `kind get kubeconfig --name finops-cluster --internal > /tmp/kubeconfig-docker.yaml` |
| Start MCP service (exposes MCP on port 8000) | See the full `docker run` block below |
| Run baseline Python auditor | `python3 sections/06-mcp-data-agent/code/label_auditor.py` |
| Run structured Python auditor | `python3 sections/07-llm-structured-agent/code/structured_auditor.py` |

Tip: use three terminals while experimenting — one for MCP, one for the auditor script, and one to interactively inspect cluster resources or logs.

When practicing in the lab, you will typically use three terminals so you can run the MCP server, the auditor scripts, and inspect outputs interactively.

1. Start the MCP service
   Run MCP in a Docker container and mount the kubeconfig produced for the kind cluster. The command below sets environment variables required for the MCP image used in the lab.

```bash theme={null}
docker run --rm -p 8000:8000 --network kind \
  -v /tmp/kubeconfig-docker.yaml:/home/appuser/.kube/config:ro \
  -e ENABLE_UNSAFE_STREAMABLE_HTTP_TRANSPORT=1 \
  -e PORT=8000 \
  -e HOST=0.0.0.0 \
  -e ALLOW_ONLY_READONLY_TOOLS=true \
  mcp/kubernetes:latest
```

Expected MCP startup output (trimmed):

```text theme={null}
Streamable HTTP server started
mcp-kubernetes-server is listening on port 8000
Use the following url to connect to the server:
http://0.0.0.0:8000/mcp
```

If you need to regenerate the kubeconfig and restart MCP, run:

```bash theme={null}
# generate kubeconfig for the kind cluster (internal)
kind get kubeconfig --name finops-cluster --internal > /tmp/kubeconfig-docker.yaml

# run MCP server (same as above)
docker run --rm -p 8000:8000 --network kind \
  -v /tmp/kubeconfig-docker.yaml:/home/appuser/.kube/config:ro \
  -e ENABLE_UNSAFE_STREAMABLE_HTTP_TRANSPORT=1 \
  -e PORT=8000 \
  -e HOST=0.0.0.0 \
  -e ALLOW_ONLY_READONLY_TOOLS=true \
  mcp/kubernetes:latest
```

2. Run the baseline Python auditor
   This "auditor" script connects to MCP, enumerates available tools, and queries namespaces/resources. It prints the raw LLM output as plain text without enforcing tagging rules or a strict output schema.

Run:

```bash theme={null}
(.venv) ➜ k8-finops-agent git:(main) python3 sections/06-mcp-data-agent/code/label_auditor.py
```

Sample trimmed logs showing MCP interactions:

```text theme={null}
[INFO] MCP tool "k8s"/"kubectl_get" received input: {'resourceType': 'deployments', 'name': None, 'namespace': 'payment', 'output': 'json', 'allNamespaces': False, 'labelSelector': None, 'fieldSelector': None, 'sortBy': None, 'context': ''}
[INFO] HTTP Request: POST http://localhost:8000/mcp "HTTP/1.1 200 OK"
[INFO] MCP tool "k8s"/"kubectl_get" received result (size: 191)
[INFO] MCP tool "k8s"/"kubectl_get" received result (size: 205)
[INFO] MCP tool "k8s"/"kubectl_get" received result (size: 535)
...
[INFO] HTTP Request: POST http://localhost:8000/mcp "HTTP/1.1 200 OK"
[INFO] MCP tool "k8s"/"kubectl_get" received result (size: 206)
```

Wait for the script to finish. The baseline auditor output is useful for a quick pass but is often verbose and inconsistently formatted because it doesn't pass tagging rules to the LLM. Example excerpts from the earlier baseline auditor:

## AIRLINE Namespace

| Resource Name | Kind | Labels | Missing Labels |
| - | - | - | - |
| kube-root-ca.crt | ConfigMap | `<none>` | ⚠️**owner, cost-center** |
| orphaned-data-pvc | PersistentVolumeClaim | `app=old-service` | ⚠️**owner, cost-center** |

## BOOKING-API Namespace

| Resource Name | Kind | Labels | Missing Labels |
| - | - | - | - |
| booking-api | Deployment | `app.kubernetes.io/name=booking-api`, `app.kubernetes.io/tier=backend`, `cost-center=booking-engine`, `environment=prod` | ⚠️**owner** |
| booking-api-config | ConfigMap | `app.kubernetes.io/name=booking-api`, `cost-center=booking-engine` | ⚠️**owner** |
| kube-root-ca.crt | ConfigMap | `<none>` | ⚠️**owner, cost-center** |
| booking-api-data | PersistentVolumeClaim | `app.kubernetes.io/name=booking-api`, `cost-center=booking-engine` | ⚠️**owner** |

## DEFAULT Namespace

| Resource Name | Kind | Labels | Missing Labels |
| - | - | - | - |
| analytics-collector | Deployment | `app=analytics` | ⚠️**owner, cost-center** |
| analytics-config | ConfigMap | `<none>` | ⚠️**owner, cost-center** |
| kube-root-ca.crt | ConfigMap | `<none>` | ⚠️**owner, cost-center** |
| untracked-config | ConfigMap | `<none>` | ⚠️**owner, cost-center** |
| analytics-data | PersistentVolumeClaim | `<none>` | ⚠️**owner, cost-center** |

(The baseline auditor continues similarly for other namespaces.)

Why this matters: the baseline output lacks explicit tagging rules and a strict schema, so it can be inconsistent and harder to use in automated remediation workflows.

3. Run the structured auditor (policy-aware)
   The structured auditor passes the tagging rules and a system prompt to the LLM before sending resource data. This enforces a consistent format and makes the output machine-friendly.

Run:

```bash theme={null}
(.venv) ➜ k8-finops-agent git:(main) python3 sections/07-llm-structured-agent/code/structured_auditor.py
```

Trimmed initialization and MCP connection logs for the structured auditor:

```text theme={null}
[INFO] MCP server "k8s": initializing with: {'url': 'http://localhost:8000/mcp', 'transport': 'streamable_http'}
[INFO] MCP server "k8s": Pre-validating authentication
[INFO] HTTP Request: POST http://localhost:8000/mcp "HTTP/1.1 200 OK"
[INFO] MCP server "k8s": Authentication validation passed: 200
[INFO] MCP server "k8s": connecting via Streamable HTTP (explicit) to http://localhost:8000/mcp
[INFO] Negotiated protocol version: 2025-11-25
[INFO] MCP server "k8s": connected
[INFO] HTTP Request: POST http://localhost:8000/mcp "HTTP/1.1 202 Accepted"
[INFO] MCP server "k8s": 8 tool(s) available:
[INFO] - kubectl_get
[INFO] - kubectl_describe
[INFO] - kubectl_logs
[INFO] - kubectl_context
[INFO] - kubectl_reconnect
[INFO] - explain_resource
[INFO] - list_api_resources
[INFO] - ping
[INFO] MCP servers initialized: 8 tool(s) available in total
```

Sample MCP tool traces while building the structured report:

```text theme={null}
[INFO] MCP tool "k8s"/"kubectl_get" received result (size: 205)
[INFO] MCP tool "k8s"/"kubectl_get" received result (size: 194)
[INFO] MCP tool "k8s"/"kubectl_get" received result (size: 535)
[INFO] MCP tool "k8s"/"kubectl_get" received result (size: 724)
[INFO] MCP tool "k8s"/"kubectl_describe" received result (size: 33244)
[INFO] MCP tool "k8s"/"kubectl_describe" received result (size: 652)
[INFO] HTTP Request: POST https://api.ai.kodekloud.com/v1/chat/completions "HTTP/1.1 200 OK"
[INFO] MCP server "k8s": session closed
Now I have all the data I need. Let me compile the comprehensive audit report.
```

Because the tagging rules are provided to the LLM via the system prompt, the LLM produces structured entries that follow policy. Example structured outputs (trimmed):

### Example structured outputs (trimmed):

* **Violations:** Missing application, slack-channel
* **Status:** ❌ FAIL (4 of 6 required tags present)

#### Resource: ConfigMap / kube-root-ca.crt

* **Labels:** None
* **Violations:** Missing all 6 required tags
* **Status:** ❌ FAIL

#### Resource: ConfigMap / payment-config

* **Labels:**
  * environment=prod
  * owner=payments-team
* **Violations:** Missing application, cost-center, tier, slack-channel
* **Status:** ❌ FAIL (2 of 6 required tags present)

#### Resource: ConfigMap / payment-gateway-api-config

* **Labels:** app=payment-gateway-api
* **Violations:** Missing owner, environment, cost-center, tier, slack-channel
* **Status:** ❌ FAIL (1 of 6 required tags present)

#### Resource: ConfigMap / payment-gateway-ui-config

* **Labels:** app=payment-gateway
* **Violations:** Missing owner, environment, cost-center, tier, slack-channel
* **Status:** ❌ FAIL (1 of 6 required tags present)

#### Resource: PersistentVolumeClaim / payment-logs

* **Labels:**
  * environment=prod
  * owner=payments-team
* **Violations:** Missing application, cost-center, tier, slack-channel
* **Status:** ❌ FAIL (2 of 6 required tags present)

## Compiled audit summary (example produced by the structured auditor)

## KUBERNETES RESOURCE LABEL AUDIT REPORT

### FinOps Governance Assessment

**Audit Date:** 2026-07-18\
**Namespaces Audited:** 5 application namespaces\
**Total Resources Evaluated:** 26\
**Excluded Namespaces:** kube-system, kube-public, kube-node-lease, local-path-storage

***

### NAMESPACE: airline

#### Resource: ConfigMap / kube-root-ca.crt

* **Labels:** None
* **Violations:** Missing all 6 required tags (owner, environment, cost-center, application, tier, slack-channel)
* **Status:** ❌ FAIL

#### Resource: PersistentVolumeClaim / orphaned-data-pvc

* **Labels:** app=old-service
* **Violations:**
  * Missing: owner, environment, cost-center, tier, slack-channel
  * Present: application (via `app` label)
* **Status:** ❌ FAIL (5 of 6 required tags missing)

***

### NAMESPACE: booking-api

#### Resource: Deployment / booking-api

* **Labels:**
  * app.kubernetes.io/name=booking-api
  * app.kubernetes.io/tier=backend
  * cost-center=booking-engine
  * environment=prod
* **Violations:** Missing owner and slack-channel
* **Status:** ❌ FAIL (4 of 6 required tags present)

***

### NAMESPACE: flight-search

#### Resource: Deployment / flight-search-service

* **Labels:**
  * app=flight-search
  * app.kubernetes.io/tier=backend
  * environment=prod
* **Violations:** Missing owner, cost-center, slack-channel
* **Status:** ❌ FAIL (3 of 6 required tags present)

#### Resource: ConfigMap / flight-search-config

* **Labels:** app=flight-search
* **Violations:** Missing owner, environment, cost-center, tier, slack-channel
* **Status:** ❌ FAIL (1 of 6 required tags present)

***

### NAMESPACE: inventory

#### Resource: Deployment / inventory-service

* **Labels:**
  * app.kubernetes.io/name=inventory-service
  * app.kubernetes.io/tier=backend
  * cost-center=inventory
  * criticality=P2
  * environment=prod
  * owner=inventory-team
* **Violations:** Missing slack-channel
* **Status:** ⚠️ PARTIAL (5 of 6 required tags present)

#### Resource: ConfigMap / inventory-config

* **Labels:**
  * app.kubernetes.io/name=inventory-service
  * app.kubernetes.io/tier=backend
  * compliance-level=standard
  * cost-center=inventory
  * environment=prod
  * owner=inventory-team
* **Violations:** Missing slack-channel
* **Status:** ⚠️ PARTIAL (5 of 6 required tags present)

***

### NAMESPACE: payment

#### Resource: Deployment / payment-gateway

* **Labels:**
  * app=payment-gateway
  * environment=prod
  * owner=payments-team
* **Violations:** Missing cost-center, tier, slack-channel, application (if not derivable)
* **Status:** ❌ FAIL / PARTIAL (depending on application derivation)

Key insights

* Feeding tagging rules and a clear system prompt to the LLM yields consistent, machine-friendly audit output.
* The structured approach ignores unrelated tags and evaluates resources strictly according to governance rules, improving remediation workflows.
* The structured auditor is better suited for automation and reporting (JSON/CSV export is simpler with a consistent schema).

If your structured auditor output appears truncated or stops mid-report with no error, check the LLM configuration for the max tokens (or response length) parameter. A too-small max tokens setting can cause the LLM to truncate long audit reports. Increase the token limit to ensure the LLM can return the complete report.

Why did earlier runs stop abruptly?

* The agent logic completed successfully, but the LLM response was capped by the configured `max_tokens` (or equivalent) setting. The truncated output shows the audit logic worked, but the LLM could not return the entire report. Increase the token limit in your LLM client or script to get full reports.

Next steps

* Increase the LLM token/response-size settings and re-run the structured auditor to retrieve full reports.
* Export the structured audit results to JSON or CSV for integration with ticketing/automation systems.
* Extend tagging rules to include environment-specific or team-specific policies and re-run to validate compliance.

References and further reading

* Kubernetes Documentation: [https://kubernetes.io/docs/](https://kubernetes.io/docs/)
* kind: [https://kind.sigs.k8s.io/](https://kind.sigs.k8s.io/)

> **[Watch Video](https://learn.kodekloud.com/user/courses/building-kubernetes-finops-ai-agent/module/5045eac8-8b03-4ecd-9213-d50eda3353c5/lesson/cae41bd2-ebb4-4577-8c62-996535aa0cc5)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.