Skip to main content
This capstone ties together the observability and incident-response concepts covered earlier into a single, repeatable flow. Use this five-phase playbook to go from “something’s broken” to “it’s fixed and confirmed” and to map each phase to the right tools and actions. A logistics company had all the right tools — Prometheus, Grafana, Jaeger, Loki, and Alertmanager — but no playbook. At 2:00 a.m. their shipping API went down. The on-call engineer opened Grafana, saw red everywhere, panicked, and started restarting pods at random. Forty-five minutes later they discovered the root cause: an HPA had scaled the API to zero because a custom metric was broken. After implementing a simple playbook, the next similar incident took four minutes instead of forty-five. The tools were the same; the outcome changed because they had a process. Use this playbook for every incident, every time.
  • Alert: Something triggered (Alertmanager, PagerDuty, Slack, or a custom report). It tells you which metric crossed a threshold and when.
The image outlines a "Five-Phase Incident Playbook" consisting of steps: Alert, Assess, Investigate, Fix, and Validate. A description under "Alert" mentions triggers like PagerDuty and Slack, detailing what and when a metric threshold was crossed.
  • Assess: Before you touch anything, spend a short, focused time (aim ~60 seconds) to understand the scope: how bad it is, who is affected, and whether it’s partial or total.
The image depicts a five-phase incident playbook with steps: Alert, Assess, Investigate, Fix, and Validate, highlighting "Assess" with a note on evaluating scope, impact, and urgency.
Spend a minute to assess before you act. Skipping assess is the most common cause of wasted time and harmful changes.
  • Investigate: Dig into the root cause in a focused, methodical order: metrics first, then traces, then logs. Follow a triage workflow to narrow scope gradually.
The image illustrates "The Five-Phase Incident Playbook," detailing phases: Alert, Assess, Investigate, Fix, and Validate. There is additional text guiding the investigation process with steps like searching and correlating.
  • Fix: Apply the minimum-change necessary to restore service: rollback, scale up, adjust configuration, or restart. Prefer a reversible, conservative change that buys time to investigate further.
The image presents a "Five-Phase Incident Playbook" consisting of five steps: Alert, Assess, Investigate, Fix, and Validate, with detailed instructions under the "Fix" phase.
  • Validate: Don’t close the incident until you confirm the fix worked. Watch dashboards and verify stability for a few minutes.
Tools map naturally to the phases:
  • Alertmanager, PagerDuty, Slack — alerts and notification (Alert, Validate)
  • Grafana — dashboards and runbooks (Assess, Validate)
  • Prometheus — metrics and PromQL (Investigate)
  • Jaeger — distributed traces (Investigate)
  • Loki / kubectl logs — logs and pod-level details (Investigate)
  • kubectl, GitOps, Helm — apply safe, reversible fixes (Fix)
Now let’s dig into each phase, starting with Assess — the step most engineers skip or rush. Assess: four quick questions
  • What’s broken? Read the alert and identify the affected service, endpoint, or pod.
  • How bad is it? Is the outage partial (a single endpoint or region) or total?
  • Who’s affected? One customer, a region, or everyone?
  • What changed recently? A deployment or config change within the alert window is a prime suspect.
Investigation: use the observability funnel (metrics → traces → logs)
  • Metrics first: Inspect the alert’s PromQL expression and open Grafana. Confirm the alert timestamps, pattern (sudden vs gradual), and compare to baseline (same time yesterday, same weekday).
  • Traces next: Search Jaeger for slow or error traces. Use minimal-duration filters, inspect the waterfall, and find the longest-duration span or failing service.
  • Logs last: Query logs by service and timestamp, or by trace ID to pinpoint the exact error message.
Each pillar narrows the search: metrics detect and scope, traces locate the problematic service or interaction, and logs explain the root cause.
The image outlines "Phase 3: Investigate With ALL Three Pillars" in observability, featuring metrics, traces, and logs using tools like Prometheus, Grafana, Jaeger, Loki, and Kubectl logs.
Measure platform performance over time with DORA metrics. These give you objective targets to reduce MTTR and improve reliability.
The image lists key platform efficiency metrics for measuring platform health, featuring four DORA metrics: Deployment Frequency, Lead Time for Changes, Change Failure Rate, and MTTR. Each metric includes a brief explanation and a target goal.
Phase 4 — Fix: four safe strategies
  • Rollback
    • When to use: a recent deployment caused the incident and assessment/investigation confirm it.
    • Why: fastest, lowest-risk path to restore service.
    • Example:
  • Scale up
    • When to use: load exceeds capacity (CPU/memory/requests throttled) and you need breathing room.
    • Why: buys time to investigate whether load is legitimate or anomalous.
    • Example:
  • Config fixes
    • When to use: misconfigured DB URL, expired secret, or bad environment variable.
    • How: update ConfigMap/Secret and trigger a restart or rolling update.
    • Example:
  • Resource adjustments (scheduling failures)
    • When to use: pods Pending because nodes lack allocatable resources.
    • How: free resources, tune requests/limits, or adjust quotas for targeted workloads.
    • Note: avoid broad, cluster-wide changes during an active incident; prefer targeted, minimal changes.
During an incident prefer reversible, minimal changes. Large, unreviewed edits or cluster-wide operations increase risk and can create cascading failures.
Quick phase-to-tool reference Key takeaways
  • Follow the playbook every time: Alert → Assess → Investigate → Fix → Validate.
  • Never skip the Assess phase; the 60-second check prevents wasted effort.
  • Use the three observability pillars as a funnel: metrics detect, traces locate, logs explain.
  • Prefer minimal, reversible fixes that restore service quickly and enable deeper post-incident investigation.
  • Track DORA metrics to measure platform health and quantify improvements to MTTR.
Links and references Use this playbook as your default incident response flow. With consistent practice and post-incident learning, the same tools will yield much faster, safer outcomes.

Watch Video

Practice Lab