Skip to main content
This capstone lesson brings together observability, incident response, and platform operations into a single, repeatable incident playbook. Follow this five‑phase flow to go from “something’s broken” to “it’s fixed and confirmed,” and learn which tools map to each phase. A short real‑world story: a logistics company had Prometheus, Grafana, Jaeger, Loki, and Alertmanager but no playbook. At 2 a.m. their shipping API failed. The on‑call engineer opened Grafana, saw red everywhere, panicked, and restarted pods at random. Forty‑five minutes later they discovered the root cause: an HPA scaled the API to zero because a custom metric was broken. After implementing a simple playbook, the next similar incident took four minutes instead of forty‑five. Same tooling, different outcome — the difference was process. Overview — the five phases we follow on every incident
  1. Alert
    An alert fires via Alertmanager, PagerDuty, Slack, or a custom notifier. The alert message includes the metric that crossed a threshold and the timestamp.
  2. Assess
    Spend a short, focused period (60–120 seconds) to determine scope and urgency: what’s affected, how severe, and what changed recently.
  3. Investigate
    Use the three pillars of observability — metrics, traces, logs — in a funneled approach: metrics → traces → logs to find the root cause.
  4. Fix
    Apply the minimum change required to restore service (rollback, scale, config change, resource tweak, or restart). Prefer the safest effective action.
  5. Validate
    Confirm the remediation actually fixed the issue. Monitor metrics, dashboards, traces, and logs before closing the incident.
Map phases to tools
  • Alerting & notification: Alertmanager, PagerDuty, Slack (Alert + Validate)
  • Dashboards & metrics: Grafana, Prometheus (Assess + Validate)
  • Tracing: Jaeger (Investigate)
  • Logs: Loki or other log stores (Investigate)
  • Control plane / deployment tooling: kubectl, GitOps, Helm (Fix)
Always do a short, structured assessment before running commands. A focused 60–120 second review prevents many mistakes and wasted effort.
Phase 1 — Alert
  • Confirm the alert details: metric name, threshold crossed, alert start time, and any annotations or runbook links included by the rule.
  • If the alert includes a runbook or playbook link, open it first — it often contains the recommended triage commands and safe mitigations.
Phase 2 — Assess (first 60–120 seconds) Answer these questions quickly:
  • What’s broken? Identify the service and the metric that triggered the alert.
  • How bad is it? Is the outage partial (some endpoints or regions) or total?
  • Who’s affected? Single endpoint, single service, or global user impact?
  • What changed recently? Check for deployments, config, or infra changes in the minutes before the alert.
If a deployment occurred shortly before the alert, that deployment is the top suspect. Rolling back to the previous revision is often the fastest safe remediation. Investigation funnel (metrics → traces → logs)
  • Metrics (the 10,000‑foot view)
    Inspect the PromQL expression behind the alert and open relevant Grafana dashboards. Determine timeline and scope: sudden crash vs. gradual degradation; compare to baseline (same time yesterday or pre‑incident). Metrics tell you who’s affected and when the problem began.
  • Traces (the call path)
    Search Jaeger for slow or failed traces (filter by duration, error tags, or service). Open the trace waterfall and find the longest or most error‑prone spans to localize the failing service or operation.
  • Logs (the exact error)
    Search logs by service name and timestamp, or use the trace ID from Jaeger to jump directly to correlated log lines. Logs typically reveal the concrete error message and stack trace.
Each pillar narrows the next: metrics point you to the service and time window, traces localize the failing span or dependency, and logs explain the underlying error. DORA metrics — measure platform health over time
  • Deployment frequency: How often do you ship? Target: at least once per day.
  • Lead time for changes: Time from commit to production. Target: under 1 hour.
  • Change failure rate: Percentage of deployments that cause incidents. Target: <5%.
  • Mean time to recovery (MTTR): Time to restore service. Target: <1 hour.
An infographic titled "Deployment Metrics: Measuring Platform Health" showing the four DORA metrics: Deployment Frequency, Lead Time for Changes, Change Failure Rate, and MTTR. Each card gives a short definition and target (e.g., >1x/day, <1 hour, <5%, <1 hour).
DORA metrics let you track whether platform and process improvements reduce MTTR and improve reliability. A robust playbook directly reduces MTTR. Phase 4 — common fix strategies (apply the smallest effective change) When you fix, prefer the least invasive action that restores service quickly and leaves an auditable trail. Rollback
  • When a recent deployment is implicated, rollback immediately and debug later.
Scale up / buy time
  • If the issue is capacity-related, scale replicas temporarily to restore availability:
Configuration fixes
  • For configuration errors (bad DB URL, expired secret), update the ConfigMap/Secret and restart the deployment to pick up changes:
Resource adjustments / scheduling fixes
  • If pods are Pending due to resource shortages, options include reducing requests on noncritical workloads, adjusting LimitRanges/ResourceQuotas, or temporarily increasing node capacity. When using GitOps or Helm, prefer applying minimal changes through your normal pipeline to preserve history and auditability.
Avoid broad destructive actions (e.g., mass pod deletion or cluster restarts) unless you understand the impact. Always prefer targeted, auditable remediations that can be rolled back.
Phase 5 — validate the fix
  • Confirm the original alert clears and does not immediately reappear.
  • Watch Grafana dashboards and Prometheus metrics for a few minutes (or longer if the metric requires a longer sampling window).
  • Re-check traces and logs to ensure the previous errors no longer occur.
  • If the remediation was temporary (e.g., scaling), schedule a follow‑up to implement a durable fix.
Tools-to-phase quick reference table Key takeaways
  • Follow the playbook consistently: Alert → Assess → Investigate → Fix → Validate.
  • Never skip the Assess phase — a brief structured assessment prevents wasted effort and dangerous fixes.
  • Use the three pillars as a funnel: metrics detect, traces locate, logs explain.
  • Know which tools to use at each stage and keep remediation minimal and auditable.
  • Track DORA metrics to measure platform health and process effectiveness over time.
Links and references

Watch Video

Practice Lab