Skip to main content
Welcome back. In this lesson you’ll run the auditor agent and compare its output to the earlier baseline auditor run. The improved “structured” agent includes the tagging rules and a system prompt when it queries the LLM, producing a consistent, policy-aware report. Prerequisites
  • A running kind cluster used by the lab (example name: finops-cluster).
  • Docker available to run the MCP service.
  • A Python virtual environment with the auditor scripts installed.
  • (Optional) Familiarity with Kubernetes labeling/tagging best practices: Kubernetes Basics
Quick command reference Tip: use three terminals while experimenting — one for MCP, one for the auditor script, and one to interactively inspect cluster resources or logs.
When practicing in the lab, you will typically use three terminals so you can run the MCP server, the auditor scripts, and inspect outputs interactively.
  1. Start the MCP service Run MCP in a Docker container and mount the kubeconfig produced for the kind cluster. The command below sets environment variables required for the MCP image used in the lab.
Expected MCP startup output (trimmed):
If you need to regenerate the kubeconfig and restart MCP, run:
  1. Run the baseline Python auditor This “auditor” script connects to MCP, enumerates available tools, and queries namespaces/resources. It prints the raw LLM output as plain text without enforcing tagging rules or a strict output schema.
Run:
Sample trimmed logs showing MCP interactions:
Wait for the script to finish. The baseline auditor output is useful for a quick pass but is often verbose and inconsistently formatted because it doesn’t pass tagging rules to the LLM. Example excerpts from the earlier baseline auditor:

AIRLINE Namespace

BOOKING-API Namespace

DEFAULT Namespace

(The baseline auditor continues similarly for other namespaces.) Why this matters: the baseline output lacks explicit tagging rules and a strict schema, so it can be inconsistent and harder to use in automated remediation workflows.
  1. Run the structured auditor (policy-aware) The structured auditor passes the tagging rules and a system prompt to the LLM before sending resource data. This enforces a consistent format and makes the output machine-friendly.
Run:
Trimmed initialization and MCP connection logs for the structured auditor:
Sample MCP tool traces while building the structured report:
Because the tagging rules are provided to the LLM via the system prompt, the LLM produces structured entries that follow policy. Example structured outputs (trimmed):

Example structured outputs (trimmed):

  • Violations: Missing application, slack-channel
  • Status: ❌ FAIL (4 of 6 required tags present)

Resource: ConfigMap / kube-root-ca.crt

  • Labels: None
  • Violations: Missing all 6 required tags
  • Status: ❌ FAIL

Resource: ConfigMap / payment-config

  • Labels:
    • environment=prod
    • owner=payments-team
  • Violations: Missing application, cost-center, tier, slack-channel
  • Status: ❌ FAIL (2 of 6 required tags present)

Resource: ConfigMap / payment-gateway-api-config

  • Labels: app=payment-gateway-api
  • Violations: Missing owner, environment, cost-center, tier, slack-channel
  • Status: ❌ FAIL (1 of 6 required tags present)

Resource: ConfigMap / payment-gateway-ui-config

  • Labels: app=payment-gateway
  • Violations: Missing owner, environment, cost-center, tier, slack-channel
  • Status: ❌ FAIL (1 of 6 required tags present)

Resource: PersistentVolumeClaim / payment-logs

  • Labels:
    • environment=prod
    • owner=payments-team
  • Violations: Missing application, cost-center, tier, slack-channel
  • Status: ❌ FAIL (2 of 6 required tags present)

Compiled audit summary (example produced by the structured auditor)

KUBERNETES RESOURCE LABEL AUDIT REPORT

FinOps Governance Assessment

Audit Date: 2026-07-18
Namespaces Audited: 5 application namespaces
Total Resources Evaluated: 26
Excluded Namespaces: kube-system, kube-public, kube-node-lease, local-path-storage

NAMESPACE: airline

Resource: ConfigMap / kube-root-ca.crt

  • Labels: None
  • Violations: Missing all 6 required tags (owner, environment, cost-center, application, tier, slack-channel)
  • Status: ❌ FAIL

Resource: PersistentVolumeClaim / orphaned-data-pvc

  • Labels: app=old-service
  • Violations:
    • Missing: owner, environment, cost-center, tier, slack-channel
    • Present: application (via app label)
  • Status: ❌ FAIL (5 of 6 required tags missing)

NAMESPACE: booking-api

Resource: Deployment / booking-api

  • Labels:
    • app.kubernetes.io/name=booking-api
    • app.kubernetes.io/tier=backend
    • cost-center=booking-engine
    • environment=prod
  • Violations: Missing owner and slack-channel
  • Status: ❌ FAIL (4 of 6 required tags present)

Resource: Deployment / flight-search-service

  • Labels:
    • app=flight-search
    • app.kubernetes.io/tier=backend
    • environment=prod
  • Violations: Missing owner, cost-center, slack-channel
  • Status: ❌ FAIL (3 of 6 required tags present)

Resource: ConfigMap / flight-search-config

  • Labels: app=flight-search
  • Violations: Missing owner, environment, cost-center, tier, slack-channel
  • Status: ❌ FAIL (1 of 6 required tags present)

NAMESPACE: inventory

Resource: Deployment / inventory-service

  • Labels:
    • app.kubernetes.io/name=inventory-service
    • app.kubernetes.io/tier=backend
    • cost-center=inventory
    • criticality=P2
    • environment=prod
    • owner=inventory-team
  • Violations: Missing slack-channel
  • Status: ⚠️ PARTIAL (5 of 6 required tags present)

Resource: ConfigMap / inventory-config

  • Labels:
    • app.kubernetes.io/name=inventory-service
    • app.kubernetes.io/tier=backend
    • compliance-level=standard
    • cost-center=inventory
    • environment=prod
    • owner=inventory-team
  • Violations: Missing slack-channel
  • Status: ⚠️ PARTIAL (5 of 6 required tags present)

NAMESPACE: payment

Resource: Deployment / payment-gateway

  • Labels:
    • app=payment-gateway
    • environment=prod
    • owner=payments-team
  • Violations: Missing cost-center, tier, slack-channel, application (if not derivable)
  • Status: ❌ FAIL / PARTIAL (depending on application derivation)
Key insights
  • Feeding tagging rules and a clear system prompt to the LLM yields consistent, machine-friendly audit output.
  • The structured approach ignores unrelated tags and evaluates resources strictly according to governance rules, improving remediation workflows.
  • The structured auditor is better suited for automation and reporting (JSON/CSV export is simpler with a consistent schema).
If your structured auditor output appears truncated or stops mid-report with no error, check the LLM configuration for the max tokens (or response length) parameter. A too-small max tokens setting can cause the LLM to truncate long audit reports. Increase the token limit to ensure the LLM can return the complete report.
Why did earlier runs stop abruptly?
  • The agent logic completed successfully, but the LLM response was capped by the configured max_tokens (or equivalent) setting. The truncated output shows the audit logic worked, but the LLM could not return the entire report. Increase the token limit in your LLM client or script to get full reports.
Next steps
  • Increase the LLM token/response-size settings and re-run the structured auditor to retrieve full reports.
  • Export the structured audit results to JSON or CSV for integration with ticketing/automation systems.
  • Extend tagging rules to include environment-specific or team-specific policies and re-run to validate compliance.
References and further reading

Watch Video