> ## Documentation Index > Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt > Use this file to discover all available pages before exploring further. # Production is down > AI-driven multi-agent incident response automating triage and remediation for Kubernetes CrashLoopBackOff and Terraform drift to reduce production outage time. It's Friday, 2 a.m. After a long week everything is quiet — until the incident management system, [Planhat](https://planhat.com), lights up with new tickets. A P1 incident appears: a cascade failure in production. Kubernetes pods are crashing and monitoring shows a sudden spike in errors. Alex, the junior on-call engineer, jolts awake. His old instinct is to escalate and wake the senior expert. But this time he trusts the team's tooling: an AI incident commander built to take first-response actions. He types a single instruction: ```text theme={null} Initiate P1 incident response. Analyze all open tickets in the 'capstone-project' via the Planhat MCP. Delegate resolution tasks to the appropriate specialist agents and provide a final report. ``` The general-purpose sub-agent activates immediately and begins querying the Planhat MCP to enumerate projects, modules, and open issues. ```text theme={null} > Task: general-purpose subagent "P1 incident response" — Running Task Detail: - Get project ID for 'capstone-project' using get_projects. - Analyze all open tickets and identify P1-critical issues. - For each P1 issue, inspect details, comments, and worklogs. - Delegate remediation to specialist agents: - kubernetes-specialist for container/orchestration issues - cloud-architect for Terraform / cloud infrastructure issues - general-purpose for complex cross-domain tasks - Provide a final report summarizing analysis, delegated tasks, and status. Available tools: - list_modules - list_module_issues - get_projects (Calls in progress...) ``` In seconds, the AI reports its analysis: it has scanned every ticket and identified two distinct critical problems — a Terraform infrastructure drift and multiple Kubernetes pods in CrashLoopBackOff. Issue analysis (summary): ```text theme={null} All identified issues were treated as P1-critical based on their open status and problem descriptions. Comments and worklogs were absent (comments lists empty; worklog requests returned 404). Issues of interest: 1) Issue ID: 1680e319-08bf-48f3-907d-e9fe85868495 Name: Resolve Terraform Drift State: Backlog (open) Description: Minimal details provided Analysis: Terraform-managed resources are out of sync with state. Critical to correct to prevent infrastructure inconsistencies. 2) Issue ID: 9e7b9f03-c30c-40d6-9235-5b9c3cb27102 Name: Pod kk-pod1 reporting CrashLoopBackOff State: Backlog (open) Description: Minimal details provided Analysis: Kubernetes pod failure (CrashLoopBackOff) causing service unavailability. Requires logs, command, and manifest checks. ``` Issue summary table | Issue ID | Name | State | Root Area | | - | - | - | - | | 1680e319-08bf-48f3-907d-e9fe85868495 | Resolve Terraform Drift | Backlog (open) | Terraform/Cloud | | 9e7b9f03-c30c-40d6-9235-5b9c3cb27102 | Pod kk-pod1 CrashLoopBackOff | Backlog (open) | Kubernetes | The orchestrator assigns work to specialist agents: Cloud-Architect handles the Terraform drift; Kubernetes-Specialist handles the CrashLoopBackOff pods. Both specialists run diagnostics (logs, manifests, restart policies) and apply fixes in parallel. A nighttime bedroom/desk scene showing a worried person in pajamas standing next to a bed while multiple monitors and pop-up windows display a critical PagerDuty incident and work items about crashing Kubernetes pods. A coffee mug and energy drink sit on the desk beside the computer. Kubernetes specialist execution (excerpt) ```text theme={null} / Task: kubernetes-specialist subagent: "Resolve Pod CrashLoopBackOff" — Completed Task Detail: - Target pods (capstone-project): kk-pod1 (9e7b9f03...), kk-pod2 (9fbffc22...), kk-pod3 (1bc2018a...) - Actions: - kubectl logs kk-pod1 (inspect restart errors) - kubectl get pods -n capstone-project (verify pod states) - kubectl apply -f /root/fix-incident/pod3.yml - kubectl apply -f /root/fix-incident/pod2.yml - Update manifests / commands / restartPolicy as required - Diagnostics found invalid container commands and unsuitable restart policies. Execution Summary: - Duration: 1m 56s - Tool calls: 23 (22 success, 1 failed) - Rounds: 24 - Result: Pods stabilized after command and manifest fixes. ``` Concurrent to the Kubernetes work, the Cloud-Architect agent corrected Terraform drift by updating a conflicting S3 bucket resource and reapplying Terraform, ensuring the state matched the deployed infrastructure. When specialist fixes complete, the incident commander consolidates the results, updates todos, and closes P1 work items. A screenshot of a technical incident report assigning tasks to Cloud-Architect, Kubernetes-Specialist, and General-Purpose agents with details on responsibilities. It lists status and recommendations (next steps, potential risks, completion criteria) and mentions Terraform drift and pod CrashLoopBackOff issues. Final status and tasks (concise report) | Agent | Task (Issue ID) | Status | Summary of Fixes | | - | - | - | - | | Cloud-Architect | Resolve Terraform Drift (1680e319-08bf-48f3-907d-...) | Completed | Updated S3 bucket name; redeployed static site; reconciled Terraform state | | Kubernetes-Specialist | Resolve Pod kk-pod1 CrashLoopBackOff (9e7b9f03-...) | Completed | Replaced invalid command with a valid sleep command; pod stable | | Kubernetes-Specialist | Resolve Pod kk-pod2 CrashLoopBackOff (9fbffc22-...) | Completed | Adjusted restart policy & command so pod completes normally | | Kubernetes-Specialist | Resolve Pod kk-pod3 CrashLoopBackOff (1bc2018a-...) | Completed | Fixed command to use proper shell interpreter; pod completes successfully | Todos updated: * Resolve Terraform Drift — Completed (Cloud-Architect) * Resolve Pod kk-pod1 CrashLoopBackOff — Completed (Kubernetes-Specialist) * Resolve Pod kk-pod2 CrashLoopBackOff — Completed (Kubernetes-Specialist) * Resolve Pod kk-pod3 CrashLoopBackOff — Completed (Kubernetes-Specialist) Impact and outcome * Outage time reduced from a typical 4-hour manual response to a near-instant AI-assisted diagnosis and remediation workflow (minutes). * Reproducible runbooks and RAG-powered documentation replaced manual search and fragmented notes. (Learn more about RAG techniques: [RAG fundamentals](https://learn.kodekloud.com/user/courses/fundamentals-of-rag).) * Security and compliance scans were automated for fast, consistent audits. * The team achieved horizontal scalability of expertise by composing a virtual team of specialist sub-agents under an AI incident commander. Automated orchestration excels at fast triage and repeatable remediation steps. Always pair AI-driven actions with human oversight for high-impact production changes and post-incident reviews. The transformation is complete — from chaotic manual firefighting to an AI-coordinated, multi-agent incident response system that detects, diagnoses, and remediates critical production issues. Now it's your turn: step into the labs, apply these patterns, and lead the next evolution of AI-assisted operations. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.