> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Demo Running the Audit Tracker Agent and Review the Tickets

> Demonstrates running a FinOps audit agent that collects Kubernetes Deployments, evaluates tagging compliance, and creates issue tracker tickets for non compliant resources across namespaces

Welcome back. This lesson walks through running the FinOps audit agent to:

* collect Deployments from a Kubernetes cluster,
* evaluate resources against FinOps tagging rules,
* and create issues in the issue tracker for non-compliant resources.

Prerequisites

* Ensure your Kubernetes MCP(s) and the issue tracker service are running and reachable (Uvicorn endpoints for the tracker).
* Confirm the tracker currently has no tickets before starting.

Quick check (tracker empty):

```bash theme={null}
(.venv) ➜  k8-finops-agent git:(main) ls
# (UI shows no tickets at this time)
```

Ensure your MCP servers and issue tracker container are running (Uvicorn / MCP endpoints reachable) before running the audit agent. If the tracker or MCP is down, the agent cannot create tickets.

Running the auditor
Execute the tracker auditor script from the project root:

```bash theme={null}
(.venv) ➜  k8-finops-agent git:(main) python3 sections/08-from-findings-to-tickets/agent/tracker_auditor.py
```

Representative runtime output
You should see the tracker and MCP services start, followed by the agent discovering MCP tools and connecting to MCPs. Example (condensed):

```bash theme={null}
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO:     Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)
INFO:     172.17.0.1:61612 - "GET / HTTP/1.1" 200 OK
INFO:     172.17.0.1:61612 - "GET /api/issues HTTP/1.1" 200 OK
INFO:     172.17.0.1:56934 - "GET /sse HTTP/1.1" 200 OK
INFO:     172.17.0.1:56944 - "POST /messages/?session_id=05bdea03d0254494b096592b72325cf3 HTTP/1.1" 202 Accepted
Processing request of type ListToolsRequest
ERRO[2070] error waiting for container: invalid character 'u' looking for beginning of value
```

Note: You might see a deprecation warning that the tracker MCP uses SSE transport. SSE is deprecated for MCPs as of 2025-03-26 — consider migrating the tracker MCP to `streamable_http`.

MCP server "tracker" is using SSE transport (deprecated). Consider migrating to `streamable_http` to avoid future incompatibilities.

MCP discovery logs
The agent will log its connections to MCP servers and list the tools available from each MCP.

Kubernetes MCP example:

```bash theme={null}
[INFO] MCP server "k8s": initializing with: {'url': 'http://localhost:8000/mcp', 'transport': 'streamable_http'}
[INFO] MCP server "k8s": Pre-validating authentication
[INFO] HTTP Request: POST http://localhost:8000/mcp "HTTP/1.1 200 OK"
[INFO] MCP server "k8s": Authentication validation passed: 200
[INFO] MCP server "k8s": connecting via Streamable HTTP (explicit) to http://localhost:8000/mcp
[INFO] Negotiated protocol version: 2025-11-25
[INFO] MCP server "k8s": connected
[INFO] MCP server "k8s": 8 tool(s) available:
[INFO] - kubectl_get
[INFO] - kubectl_describe
[INFO] - kubectl_logs
[INFO] - kubectl_context
[INFO] - kubectl_reconnect
[INFO] - explain_resource
[INFO] - list_api_resources
[INFO] - ping
```

Tracker MCP example:

```bash theme={null}
[INFO] MCP server "tracker": connected
[INFO] MCP server "tracker": 4 tool(s) available:
[INFO] - create_issue
[INFO] - list_issues
[INFO] - get_issue
[INFO] - update_issue
```

Agent execution flow
The auditor performs a small, deterministic set of steps defined by its prompt. In this lesson the prompt limited the agent to a single `kubectl_get` call scoped to the `payment` namespace, then evaluated labeling compliance and created tickets.

Example agent actions (condensed):

* Request Deployments (single kubectl\_get):

```bash theme={null}
[INFO] MCP tool "k8s"/"kubectl_get" received input: {'resourceType': 'deployments', 'name': None, 'namespace': 'payment', 'output': 'json', 'allNamespaces': False, 'labelSelector': None, 'fieldSelector': None, 'sortBy': None, 'context': ''}
[INFO] MCP tool "k8s"/"kubectl_get" received result (size: 547)
```

* Create issues on the tracker MCP (example payload):

```bash theme={null}
[INFO] MCP tool "tracker"/"create_issue" received input: {
  'title': '[FinOps] payment/payment-gateway - MISSING-LABELS',
  'summary': 'Deployment payment-gateway in namespace payment has no labels present, resulting in all required tags (owner, environment, cost-center, application, tier, slack-channel) being missing.',
  'body': "The Deployment 'payment-gateway' in the 'payment' namespace was evaluated against the FinOps tagging policy. No labels matching any of the required label_mappings keys were found for owner, environment, cost-center, application, tier, or slack-channel. This makes it impossible to attribute cost, identify the responsible team, or determine deployment environment/tier for this resource. Given this resource resides in the payment namespace (likely handling sensitive transaction data), lack of ownership and cost-center tagging poses both governance and financial tracking risk.",
  'namespace': 'payment',
  'resource_name': 'payment-gateway',
  'resource_kind': 'Deployment',
  'category': 'missing-labels',
  'priority': 'high',
  'cost_impact': 0.0,
  'assignee': '',
  'suggested_owner': 'payment-team',
  'suggested_cost_center': 'payment',
  'reasoning': 'No labels were present on the resource at all, so none of the accepted keys for any required tag could be matched. Namespace name suggests cost-center "payment" and an owning team responsible for payment services.',
  'source': 'mcp-llm-agent'
}
[INFO] MCP tool "tracker"/"create_issue" received result (size: 1662)
```

Run summary — first (payment-only) run
The following summarized what happened during the first run (when the agent was restricted to the `payment` namespace):

* Step 1 – Collected Deployments in `payment`: 3 found
  * `payment-gateway`
  * `payment-gateway-api`
  * `payment-processor`
* Step 2 – Evaluated each Deployment against the FinOps tagging policy. None of the three contained a `labels` field, so required tags (owner, environment, cost-center, application, tier, slack-channel) could not be matched — all three were non-compliant.
* Step 3 – Created 3 tickets (category: `missing-labels`, priority: `high`), one per Deployment:
  * FINOPS-0001 – payment/payment-gateway
  * FINOPS-0002 – payment/payment-gateway-api
  * FINOPS-0003 – payment/payment-processor
    Each ticket suggested `payment-team` as owner and `payment` as cost-center based on namespace context.
* Step 4 – Confirmed via `list_issues`: 3 total issues, all in `backlog`, 0 estimated monthly savings recorded.

Summary table (first run)

| Step | Action | Result |
| - | - | - |
| 1 | Collected Deployments | 3 (namespace: `payment`) |
| 2 | Tagging evaluation | 3 non-compliant (no labels) |
| 3 | Issues created | 3 — FINOPS-0001..0003 (category: `missing-labels`) |
| 4 | List issues | 3 in backlog, 0 estimated savings |

Why the agent only evaluated the `payment` namespace
Open the Python auditor file and inspect the prompt and allowed tools. The PROMPT explicitly constrained the collection to a single `kubectl_get` against `namespace=payment`. The agent follows the prompt literally and therefore limited its scope accordingly.

Relevant excerpt from the Python auditor (kept concise):

```python theme={null}
"""Unified FinOps agent: audit cluster labels via K8s MCP and post tickets via tracker MCP."""
import asyncio
from pathlib import Path

from mcp_client import run_agent

# Keep the tool surface small so the agent spends steps on create_issue, not discovery.
ALLOWED_TOOLS = {"kubectl_get", "create_issue", "list_issues"}

PROMPT = """
You are a FinOps agent with Kubernetes kubectl_get and issue tracker tools (create_issue, list_issues).

Execute this exact plan – do not invent extra discovery steps:

Step 1 – Collect (1 kubectl_get call only):
- kubectl_get resourceType=deployments namespace=payment output=json

Step 2 – Evaluate labels on each Deployment against the tagging rules.
A required tag is missing when NONE of its label_mappings keys appear on the resource.

Step 3 – For EVERY non-compliant Deployment, call create_issue (one call per resource).
Required fields:
- title: "[FinOps] {namespace}/{resource_name} – {CATEGORY}"
- summary, body, namespace, resource_name, resource_kind
- category: missing-owner | unallocated | orphaned | missing-labels
- priority: critical | high | medium | low
- suggested_owner, suggested_cost_center, reasoning
- source: "mcp-llm-agent"

Step 4 – Call list_issues once and stop.

Hard rules:
- Follow Step 1's scope exactly – do not invent extra kubectl_get calls.
- Do not fetch configmaps, PVCs, or individual resources by name.
- Do not write a plain-English audit report instead of creating tickets.
"""
```

How to run the auditor across all namespaces
To audit Deployments across all namespaces with a single `kubectl_get`, update Step 1 in the `PROMPT` to request all namespaces (e.g., set `allNamespaces=True` or remove the `namespace` constraint so the tool uses `allNamespaces=True`).

Example conceptual change:

```python theme={null}
# Replace or modify Step 1 in PROMPT:
# - kubectl_get resourceType=deployments namespace=payment output=json
# with:
# - kubectl_get resourceType=deployments allNamespaces=true output=json
```

Re-run the auditor after saving the change. The agent will then collect Deployments cluster-wide in a single call and create issues for non-compliant resources across namespaces.

Representative summary — all-namespaces run
After changing to collect all namespaces in Step 1, a follow-up run produced the following condensed results:

* Step 1 – Collected: 9 Deployments across all namespaces via one `kubectl_get` call.
* Step 2 – Evaluated against tagging rules: 2 Deployments were excluded by `excluded_namespaces` (e.g., `kube-system` and `local-path-storage`); the remaining 7 had zero labels and were non-compliant.
* Step 3 – Created 7 new tickets (in addition to the 3 pre-existing payment tickets from the prior run):
  * FINOPS-0004 — booking-api/booking-api — missing-labels — high
  * FINOPS-0005 — default/analytics-collector — missing-labels — low
  * FINOPS-0006 — flight-search/flight-search-service — missing-labels — medium
  * FINOPS-0007 — inventory/inventory-service — missing-labels — medium
  * FINOPS-0008 — payment/payment-gateway — missing-labels — critical
  * FINOPS-0009 — payment/payment-gateway-api — missing-labels — critical
  * FINOPS-0010 — payment/payment-processor — missing-labels — critical
* Step 4 – `list_issues` confirmed: 10 total open tickets in backlog (0 in progress/done), all `missing-labels`.

Note: Some resources (e.g., `payment-gateway-api`) may appear duplicated across runs with slightly different metadata. Add deduplication logic in the tracker integration or agent to avoid duplicate tickets for the same resource.

Next steps and operational notes

* Triage: Engineering teams can triage issues, assign owners, estimate savings, and progress tickets (backlog → in-progress → done).
* Remediation: Applying the required labels to Deployments resolves FinOps issues and enables cost attribution.
* Automation: Consider deduplication, severity heuristics, and automatic reassignment rules in the tracker integration to reduce noise.
* Architecture: Update your documentation to reflect the actual flow used here — collector agent → Kubernetes MCP(s) → LLM-based evaluation using tagging rules → issue tracker MCP → ticket board.

Links and references

* [Kubernetes Basics](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/)
* [Uvicorn ASGI server](https://www.uvicorn.org/)
* MCP / mcp\_client: refer to your project's MCP documentation or the repository that provides `mcp_client`.

Congratulations — you completed the end-to-end flow: the audit agent collected cluster state, evaluated resources against FinOps tagging rules, and created actionable tickets in the issue tracker. See the tracker UI at [http://localhost:8085](http://localhost:8085) to review and triage created issues.

> **[Watch Video](https://learn.kodekloud.com/user/courses/building-kubernetes-finops-ai-agent/module/bbcc27a3-b731-4d76-9ba0-b3df1a8b3afb/lesson/7d1e2e19-fb1f-41d9-bb63-26404aa3fe9a)**

> **[Practice Lab](https://learn.kodekloud.com/user/courses/building-kubernetes-finops-ai-agent/module/bbcc27a3-b731-4d76-9ba0-b3df1a8b3afb/lesson/6bc5673e-2379-4f50-b4dd-9fcb265d6aa0)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.